In this article, we will describe how to troubleshoot system crashes in WebLogic Server step by step:
You can find above message in the server log file (in <domain loc>/logs). In most Linux Distributions core file creation is disabled by default for a normal user. To enable writing core files you use the ulimit command, it controls the resources available to a process started by the shell. For example, I've added the following line at the top of my start script:
It contains information obtained at the time of the fatal error. For detailed descriptions of the above sections, read [4].
In this case, an Internal Error occurred with a thread executing in the library libjvm.so. From the Thread section , we know a JavaThread fails while in the _thread_in_vm state (meaning that it is executing in Java VM code).
The thread pointer is the pointer to the Java VM internal thread structure. It is generally of no interest unless you are debugging a live Java VM or core file. The following list shows possible thread types.
The thread ID in the output is the native thread identifier. If a Java thread is a daemon thread, then the string daemon is printed before the thread state.
- Enabling core dump if it's not done yet
- Determining where the crash occurred
Enabling Core Dump
In my test, a fatal error has been detected by the Java Runtime Environment and it tried to write core dump. However, it cannot:
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
You can find above message in the server log file (in <domain loc>/logs). In most Linux Distributions core file creation is disabled by default for a normal user. To enable writing core files you use the ulimit command, it controls the resources available to a process started by the shell. For example, I've added the following line at the top of my start script:
#!/bin/ksh
ulimit -c unlimited
Determining Where the Crash Occurred
Basically a hs_err files is what you get when something goes wrong during a local Java session. Local as in using a Java Virtual Machine (JVM) locally. The HotSpot VM is the default JVM for the Java SE plattform. Hence the name of hs_err files: HotSpot Error log. For example, I have found this file in the WLS domain folder:- hs_err_pid18181.log
- A header that provides a brief description of the crash
- A section with thread information
- A section with process information
- A section with system information
It contains information obtained at the time of the fatal error. For detailed descriptions of the above sections, read [4].
A Case Study
From the error log header, you can tell what the problematic frame is. For example, this is what I have seen in the error log:# A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/HXXXXX/workspace/jdk7u4-2-build-linux-amd64-product/jdk7u4/hotspot/src/share/vm/oops/methodOop.cpp:498), pid=19941, tid=1098848576 # assert(Klass::cast(k)->is_subclass_of(SystemDictionary::Throwable_klass())) failed: invalid exception class # # JRE version: 7.0_04-bXX # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.0-b21-fastdebug mixed mode linux-amd64 compressed oops) # Core dump written. Default location: /data/ATG/MXXXXX/user_projects/domains/atgdomain/core or core.19941 # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x00002aaac0871000): JavaThread "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_vm, id=19995, stack(0x00000000416f1000,0x00000000417f2000)] Stack: [0x00000000416f1000,0x00000000417f2000], sp=0x00000000417ed4d0, free space=1009k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xc6f6d2] VMError::report_and_die()+0x2f2 V [libjvm.so+0x5dfa44] report_vm_error(char const*, int, char const*, char const*)+0x84 V [libjvm.so+0x9fd8c8] methodOopDesc::resolved_checked_exceptions_impl(methodOopDesc*, Thread*)+0x1c8 V [libjvm.so+0xb14eb4] Reflection::get_exception_types(methodHandle, Thread*)+0x24 V [libjvm.so+0xb19741] Reflection::new_constructor(methodHandle, Thread*)+0x161 V [libjvm.so+0x8524f6] JVM_GetClassDeclaredConstructors+0x626 J java.lang.Class.getDeclaredConstructors0(Z)[Ljava/lang/reflect/Constructor;
In this case, an Internal Error occurred with a thread executing in the library libjvm.so. From the Thread section , we know a JavaThread fails while in the _thread_in_vm state (meaning that it is executing in Java VM code).
Thread Information[2]
The first part of the thread section shows the thread that provoked the fatal error, as follows.Current thread (0x00002aaac0871000): JavaThread "[AC...]" [_thread_in_native, id=21139]
| | | | +-- ID
| | | +---------------- state
| | +------------------------------ name
| +---------------------------------------- type
+--------------------------------------------------------- pointer
The thread pointer is the pointer to the Java VM internal thread structure. It is generally of no interest unless you are debugging a live Java VM or core file. The following list shows possible thread types.
- JavaThread
- VMThread
- CompilerThread
- GCTaskThread
- WatcherThread
- ConcurrentMarkSweepThread
|
The thread ID in the output is the native thread identifier. If a Java thread is a daemon thread, then the string daemon is printed before the thread state.
No comments:
Post a Comment