How to Investigate: Failed to Bind to Port on Linux

From the server log file (i.e., CRMCommonServer_1.log) of WebLogic, I have found the following messages:

####<Aug 12, 2013 10:40:43 AM PDT> <Emergency> <Security> <myserver> <CRMCommonServer_1> <[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1376329243268> <BEA-090087> <Server failed to bind to the configured Admin port. The port may already be used by another process.>
####<Aug 12, 2013 10:40:43 AM PDT> <Error> <Server> <myserver> <CRMCommonServer_1> <DynamicListenThread[Default]> <<WLS Kernel>> <> <> <1376329243268> <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 10.241.88.31 might be incorrect or another process is using port 9004: java.net.BindException: Address already in use.>

In this article, I will show you how to investigate: 

• Which process is using port 9004?

Netstat Command on Linux

To investigate failed-to-bind-to -port issue, netstat comes in handy on Linux systems.  netstat command can be used to:

• Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

In this detective work, we have used the following options:

   -a, --all

       Show both listening and non-listening sockets.  With the --interfaces  option,  show  inter-

       faces that are not marked

   -p, --program

       Show the PID and name of the program to which each socket belongs.

The results are shown below:

$ netstat -ap | grep 9004 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 myserver.us.ora:interserver myserver.oracle.com:9004 ESTABLISHED 12550/oidldapd tcp 0 0 myserver.us.oracle.com:9004 myserver.ora:interserver ESTABLISHED 22328/java

From the output, we know a Java application (i.e., process 22328) is using port 9004. When the first socket is bound to that port, then no other socket could be bound on port 9004 as long as the first socket remains open.  To know which application it is, we check out that process' command line:

• $ vi /proc/22328/cmdline 

On the command line, we have found the following information:

• -Dweblogic.Name=AdminServer

Also, BIDomain was mentioned there. So, that process is the AdminServer of BIDomain.

Port 7020

Similarly, we have seen port 7020 was used in another server's log file:

• <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 10.241.88.31 might be incorrect or another process is using port 7020: java.net.BindException: Address already in use.>

When you tried:

# netstat -ap  |grep 7020

No entries have been returned.   However, if you use:

# netstat -an  |grep 7020

You could find one entry:

tcp        0      0 ::ffff:10.241.88.31:7020    :::*                        LISTEN

In this case, we need to use the following command line:

# netstat -ap --numeric-ports |grep 7020

tcp        0      0 slcag044.us.oracle.com:7020 *:*                         LISTEN      21696/java      

So, we know process 21696 is using port 7020.  To investigate further, we typed:

# netstat -ap  |grep  21696

tcp        0      0 slcag044.us.oracle.:dpserve *:*                         LISTEN      21696/java

It shows dpserve in the place of 7020.  So, that's why our first search ended up with no entries. Now we know port 7020 was used by the dpserve protocol for service type dpserve^[2,3].

Our Solution

In our case, we need to re-order our start-up steps (see [4] for another approach). Instead of starting BIDomain first, we need to start it last. To fix our issue, we have done:

• Shut down BIDomain 
• Start up CRMDomain 
• Start up BIDomain

References

1. Using netstat
2. Port Number 7020/tcp | What is Port 7020/tcp?
3. Port 7020 Details 
4. Oracle Fusion Apps: The port may already be used by another process