How to Investigate: Failed to Bind to Port on Linux

From the server log file (i.e., CRMCommonServer_1.log) of WebLogic, I have found the following messages:

####<Aug 12, 2013 10:40:43 AM PDT> <Emergency> <Security> <myserver> <CRMCommonServer_1> <[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1376329243268> <BEA-090087> <Server failed to bind to the configured Admin port. The port may already be used by another process.>
####<Aug 12, 2013 10:40:43 AM PDT> <Error> <Server> <myserver> <CRMCommonServer_1> <DynamicListenThread[Default]> <<WLS Kernel>> <> <> <1376329243268> <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 10.241.88.31 might be incorrect or another process is using port 9004: java.net.BindException: Address already in use.>

In this article, I will show you how to investigate: 

• Which process is using port 9004?

Netstat Command on Linux

To investigate failed-to-bind-to -port issue, netstat comes in handy on Linux systems.  netstat command can be used to:

• Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

In this detective work, we have used the following options:

   -a, --all

       Show both listening and non-listening sockets.  With the --interfaces  option,  show  inter-

       faces that are not marked

   -p, --program

       Show the PID and name of the program to which each socket belongs.

The results are shown below:

$ netstat -ap | grep 9004 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 myserver.us.ora:interserver myserver.oracle.com:9004 ESTABLISHED 12550/oidldapd tcp 0 0 myserver.us.oracle.com:9004 myserver.ora:interserver ESTABLISHED 22328/java

From the output, we know a Java application (i.e., process 22328) is using port 9004. When the first socket is bound to that port, then no other socket could be bound on port 9004 as long as the first socket remains open.  To know which application it is, we check out that process' command line:

• $ vi /proc/22328/cmdline 

On the command line, we have found the following information:

• -Dweblogic.Name=AdminServer

Also, BIDomain was mentioned there. So, that process is the AdminServer of BIDomain.

Port 7020

Similarly, we have seen port 7020 was used in another server's log file:

• <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 10.241.88.31 might be incorrect or another process is using port 7020: java.net.BindException: Address already in use.>

When you tried:

# netstat -ap  |grep 7020

No entries have been returned.   However, if you use:

# netstat -an  |grep 7020

You could find one entry:

tcp        0      0 ::ffff:10.241.88.31:7020    :::*                        LISTEN

In this case, we need to use the following command line:

# netstat -ap --numeric-ports |grep 7020

tcp        0      0 slcag044.us.oracle.com:7020 *:*                         LISTEN      21696/java      

So, we know process 21696 is using port 7020.  To investigate further, we typed:

# netstat -ap  |grep  21696

tcp        0      0 slcag044.us.oracle.:dpserve *:*                         LISTEN      21696/java

It shows dpserve in the place of 7020.  So, that's why our first search ended up with no entries. Now we know port 7020 was used by the dpserve protocol for service type dpserve^[2,3].

Our Solution

In our case, we need to re-order our start-up steps (see [4] for another approach). Instead of starting BIDomain first, we need to start it last. To fix our issue, we have done:

• Shut down BIDomain 
• Start up CRMDomain 
• Start up BIDomain

References

1. Using netstat
2. Port Number 7020/tcp | What is Port 7020/tcp?
3. Port 7020 Details 
4. Oracle Fusion Apps: The port may already be used by another process

UnicastUdpSocket failed to set receive buffer size to 1428 packets

The following messages have been found from the WebLogic Server output:

<Jan 24, 2013 2:32:04 PM PST> <Warning> <Coherence> <BEA-000000> <2013-01-24 14:32:04.325/37.402 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): UnicastUdpSocket failed to set receive buffer size to 1428 packets (2096304 bytes); actual size is 357 packets (524288 bytes). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.>

As warned: if you proceed with the actual receive buffer value, it may cause sub-optimal performance for your application. In this case, it's our OAM server.

What's OAM Server?

Oracle Access Manager Server^[2] is a runtime engine used to provide shared services for access such as Authentication/Authorization service, session management, token processing, and single sign-on. OAM Server runs on one of WebLogic's Managed servers, such as oam_server1.

How to Fix the Issue?

To fix the issue, you can increase TCP max buffer size.  For example, our original values were:

net.core.rmem_max=524288
net.core.wmem_max=524288

After increasing both read and write buffer to bigger size (note that they are in bytes) as follows:

net.core.rmem_max = 4194304

net.core.wmem_max = 4194304

We don't see the above warning anymore.  You can find the above TCP buffer sizes in:

• /etc/sysctl.config

After login as a root, you can modify them and issue

• sysctl -p 

to enable sysctl data immediately.

Similarly you can issue the following Linux commands to achieve the same effects:

• sysctl -w net.core.rmem_max=4194304
• sysctl -w net.core.wmem_max=4194304

As suggest in [3], you may want to tune other TCP parameters for better networking performance.

References

1. 'UdpSocket failed to set receive buffer size to 1428 packets' Error Due to Coherence Tuning
2. Oracle Identity and Access Manager 11g for Administrators
3. Linux Tuning