Monday, August 12, 2013

How to Investigate: Failed to Bind to Port on Linux

From the server log file (i.e., CRMCommonServer_1.log) of WebLogic, I have found the following messages:

####<Aug 12, 2013 10:40:43 AM PDT> <Emergency> <Security> <myserver> <CRMCommonServer_1> <[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1376329243268> <BEA-090087> <Server failed to bind to the configured Admin port. The port may already be used by another process.>
####<Aug 12, 2013 10:40:43 AM PDT> <Error> <Server> <myserver> <CRMCommonServer_1> <DynamicListenThread[Default]> <<WLS Kernel>> <> <> <1376329243268> <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 10.241.88.31 might be incorrect or another process is using port 9004: java.net.BindException: Address already in use.>

In this article, I will show you how to investigate: 
  • Which process is using port 9004?

Netstat Command on Linux


To investigate failed-to-bind-to -port issue, netstat comes in handy on Linux systems.  netstat command can be used to:
  • Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

In this detective work, we have used the following options:

   -a, --all
       Show both listening and non-listening sockets.  With the --interfaces  option,  show  inter-
       faces that are not marked
   -p, --program
       Show the PID and name of the program to which each socket belongs.

The results are shown below:

$ netstat -ap | grep 9004 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 myserver.us.ora:interserver myserver.oracle.com:9004 ESTABLISHED 12550/oidldapd tcp 0 0 myserver.us.oracle.com:9004 myserver.ora:interserver ESTABLISHED 22328/java


From the output, we know a Java application (i.e., process 22328) is using port 9004. When the first socket is bound to that port, then no other socket could be bound on port 9004 as long as the first socket remains open.  To know which application it is, we check out that process' command line:
  • $ vi /proc/22328/cmdline 
On the command line, we have found the following information:
  • -Dweblogic.Name=AdminServer
Also, BIDomain was mentioned there. So, that process is the AdminServer of BIDomain.

Port 7020


Similarly, we have seen port 7020 was used in another server's log file:
  • <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 10.241.88.31 might be incorrect or another process is using port 7020: java.net.BindException: Address already in use.>
When you tried:
    # netstat -ap  |grep 7020

    No entries have been returned.   However, if you use:

    # netstat -an  |grep 7020

    You could find one entry:

    tcp        0      0 ::ffff:10.241.88.31:7020    :::*                        LISTEN

    In this case, we need to use the following command line:

    # netstat -ap --numeric-ports |grep 7020
    tcp        0      0 slcag044.us.oracle.com:7020 *:*                         LISTEN      21696/java      

    So, we know process 21696 is using port 7020.  To investigate further, we typed:
    # netstat -ap  |grep  21696
    tcp        0      0 slcag044.us.oracle.:dpserve *:*                         LISTEN      21696/java

    It shows dpserve in the place of 7020.  So, that's why our first search ended up with no entries. Now we know port 7020 was used by the dpserve protocol for service type dpserve[2,3].

    Our Solution


    In our case, we need to re-order our start-up steps (see [4] for another approach). Instead of starting BIDomain first, we need to start it last. To fix our issue, we have done:
    • Shut down BIDomain 
    • Start up CRMDomain 
    • Start up BIDomain

    References

    No comments: