Monday, July 23, 2012

Troubleshooting System Crashes in WebLogic Server

In this article, we will describe how to troubleshoot system crashes in WebLogic Server step by step:
  1. Enabling core dump if it's not done yet
  2. Determining where the crash occurred

Enabling Core Dump

In my test, a fatal error has been detected by the Java Runtime Environment and it tried to write core dump.  However, it cannot:

# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

You can find above message in the server log file (in <domain loc>/logs). In most Linux Distributions core file creation is disabled by default for a normal user. To enable writing core files you use the ulimit command, it controls the resources available to a process started by the shell. For example, I've added the following line at the top of my start script:

ulimit -c unlimited

Determining Where the Crash Occurred

Basically a hs_err files is what you get when something goes wrong during a local Java session. Local as in using a Java Virtual Machine (JVM) locally. The HotSpot VM is the default JVM for the Java SE plattform. Hence the name of hs_err files: HotSpot Error log. For example, I have found this file in the WLS domain folder:
  • hs_err_pid18181.log
The error log is a text file consisting of the following sections:
  • A header that provides a brief description of the crash
  • A section with thread information
  • A section with process information
  • A section with system information

It contains information obtained at the time of the fatal error.  For detailed descriptions of the above sections, read [4].

A Case Study

From the error log header, you can tell what the problematic frame is.  For example, this is what I have seen in the error log:

# A fatal error has been detected by the Java Runtime Environment:
#  Internal Error (/HXXXXX/workspace/jdk7u4-2-build-linux-amd64-product/jdk7u4/hotspot/src/share/vm/oops/methodOop.cpp:498), pid=19941, tid=1098848576
#  assert(Klass::cast(k)->is_subclass_of(SystemDictionary::Throwable_klass())) failed: invalid exception class
# JRE version: 7.0_04-bXX
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.0-b21-fastdebug mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /data/ATG/MXXXXX/user_projects/domains/atgdomain/core or core.19941
# If you would like to submit a bug report, please visit:

---------------  T H R E A D  ---------------

Current thread (0x00002aaac0871000):  JavaThread "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_vm, id=19995, stack(0x00000000416f1000,0x00000000417f2000)]

Stack: [0x00000000416f1000,0x00000000417f2000],  sp=0x00000000417ed4d0,  free space=1009k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  []  VMError::report_and_die()+0x2f2
V  []  report_vm_error(char const*, int, char const*, char const*)+0x84
V  []  methodOopDesc::resolved_checked_exceptions_impl(methodOopDesc*, Thread*)+0x1c8
V  []  Reflection::get_exception_types(methodHandle, Thread*)+0x24
V  []  Reflection::new_constructor(methodHandle, Thread*)+0x161
V  []  JVM_GetClassDeclaredConstructors+0x626
J  java.lang.Class.getDeclaredConstructors0(Z)[Ljava/lang/reflect/Constructor;

In this case, an Internal Error occurred with a thread executing in the library From the Thread section , we know a JavaThread fails while in the _thread_in_vm state (meaning that it is executing in Java VM code).

Thread Information[2]

The first part of the thread section shows the thread that provoked the fatal error, as follows.
Current thread (0x00002aaac0871000):  JavaThread "[AC...]" [_thread_in_native, id=21139]
                    |                   |         |            |                +-- ID
                    |                   |         |            +---------------- state
                    |                   |         +------------------------------ name
                    |                   +---------------------------------------- type
                    +--------------------------------------------------------- pointer

The thread pointer is the pointer to the Java VM internal thread structure. It is generally of no interest unless you are debugging a live Java VM or core file. The following list shows possible thread types.
  • JavaThread
  • VMThread
  • CompilerThread
  • GCTaskThread
  • WatcherThread
  • ConcurrentMarkSweepThread
The following table shows the important thread states.

Thread State

Thread is not created. This occurs only in the case of memory corruption.
Thread has been created but it has not yet started.
Thread is running native code. The error is probably a bug in native code.
Thread is running VM code.
Thread is running either interpreted or compiled Java code.
Thread is blocked.
If any of the above states is followed by the string _trans, that means that the thread is changing to a different state.

The thread ID in the output is the native thread identifier. If a Java thread is a daemon thread, then the string daemon is printed before the thread state.


  1. Troubleshooting System Crashes
  2. Thread Information
  3. Description of Java hs_err error log files
  4. Description of Fatal Error Log
  5. Monitoring WebLogic Server Thread Pool at Runtime
  6. Understanding JVM Thread States
  7. Understanding WebLogic Incident and the Diagnostic Framework behind It

No comments: