Cloning Issue—IPv4 vs. IPv6

With two similarly cloned environments, one env works and another env fails. Both of them use /etc/hosts to redirect:

• idm-db.us.oracle.com

to its own IP address^[1]. However, when one of the WLS managed server tried to connect to its Admin Server using the following URL:

• t3://[2606:a800:2010:4048:221:28ff:fefb:7756]:17001,

it failed.

In this article, we discuss how to trouble shoot and resolve this issue.

Log Messages^[2]

WebLogic Server provides handlers for sending log messages to standard out, the server log file, broadcasting messages to the domain log, remote clients, and a memory buffer for tail viewing log events in the WebLogic Server Administration Console. You can achieve volume control for each type of handler by filtering log messages based on severity level and other criteria. The LogMBean, described in Oracle WebLogic Server MBean Reference^[3], defines attributes for setting the severity level and specifying filter criteria for WebLogic Server handlers.

For example,  a Stdout Handler is provided in WLS—it sends stdout of the JVM in which a WebLogic Server instance runs to server terminal console.  WLS also provides a Stdout Filter which can be configured to filter log events being sent to the standard out.  By default, the Stdout Handler has a NOTICE threshold severity level. Therefore, INFO and DEBUG level messages are not sent to standard out.  There is also an attribute named RedirectStdoutToServerLogEnabled in LogMBean.  When enabled, this redirects the stdout of the JVM in which a WebLogic Server instance runs, to the WebLogic logging system.

For our case, the above-mentioned error message was found in the standard out which was redirected to a file when the WebLogic Server instance was started.

So, we need to figure out why the connection failed and why

• [2606:a800:2010:4048:221:28ff:fefb:7756]

was referenced in the URL.  After some digging, we have found that [2606...] is the inet6 addr as shown below:

# /sbin/ifconfig
eth0      Link encap:Ethernet  HWaddr 00:31:28:FB:88:56
          inet addr:xx.xxx.xx.xxx  Bcast:xx.xxx.xx.255  Mask:255.255.248.0
          inet6 addr: 2606:a800:2010:4048:221:28ff:fefb:7756/64 Scope:Global
          inet6 addr: fe80::521:25ff:fefb:7756/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:251183 errors:0 dropped:0 overruns:0 frame:0
          TX packets:150153 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:61626169 (58.7 MiB)  TX bytes:84744375 (80.8 MiB)
          Memory:df460000-df480000

Cloning Advantage

When you clone a system, you usually have a working system to clone from.  So, if your cloned system failed to start, you can compare the working and non-working systems side-by-side.  For our case, we have found that the system that fails have ipv6 enabled and the one works didn't.

So, the next step for us to take is to disable IPv6 protocol.  First, we have tried it from the JVM level.

Disabling the IPv6 Protocol at JVM Level

Two JVM options are provided in JRockit to enable and disable ipv6 addresses as below:

• -Djava.net.preferIPv6Addresses=true (or false)
• -DuseIPv6Address=true (or false)

However, when we tried the above tricks from the command line that started the WLS server instance, it didn't work.  Since we don't own the application (note that we use the application as a benchmark only), we cannot change coding to make it work with IPv6.  So, the next step is for us to disable IPv6 Protocol at kernel level.

Disabling the IPv6 Protocol at Kernel Level

We have followed the instructions described in [4-6] to disable IPv6 protocol at Linux kernel level.  There are two files we (as root) have edited by adding lines as shown below:

# vi /etc/modprobe.conf
alias net-pf-10 off
alias ipv6 off
options ipv6 disable=1


#vi /etc/sysconfig/network
NETWORKING_IPV6=no

After saved and closed files, we rebooted the system. Finally, the above changes help us resolve the issue.  Be warned that your Linux platform (note that our OS is Redhat Linux) may need different configuration from the one described here.

References

1. Simplify Cloning by Using Hosts File
2. Configuring WebLogic Logging Services
3. LogMBean
   • Configures the threshold severity level and filter settings for logging output.
4. Networking IPv6 User Guide for J2SDK/JRE 1.4
5. Linux: How To Disable The IPv6 Protocol
6. RedHat / Centos Disable IPv6 Networking
7. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
• This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
• Oracle Fusion Middleware provides a series of scripts for this task.
8. Oracle Products: What Patching, Migration, and Upgrade Mean?
9. DNS and BIND, 5th edition, by Cricket Liu and Paul Albitz
• What’s this fe80:: address?
• These are link-local scoped addresses, derived automatically from the interfaces’ hardware addresses. The link-local scope is significant—you can’t access these addresses from anywhere but the local subnet, so don’t use them in delegation, masters substatements, and the like. 
10. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
• For your Oracle production systems, follow official recommendations as shown in this article.