Diagnosing Java.lang.OutOfMemoryError

The heap is one of the foremost components that should be monitored to trace performance issues^{[18, 19,22]}. Heap pressure is created when the heap usage approaches the maximum heap size permitted. This leads to frequent full garbage collection events. This steals CPU cycles available for processing and the overall response times degrade. Extreme cases can lead to OutOfMemory conditions, which are not recoverable without a JVM restart.

OutOfMemoryError

When I ran my application, the following exceptions have been thrown in sequence:

• java.lang.OutOfMemoryError: GC overhead limit exceeded^[7,9,15]
• java.lang.OutOfMemoryError: Java heap space

The first message means that, for some reason, the garbage collector is taking an excessive amount of time and recovers very little memory in each run. After I removed the following statement:

• System.gc();

The 1st message was gone. However, the system threw the 2nd message. So, obviously my heap space issue remains. Here are the steps that I took to investigate it:

1. Add the following Java Options
   • -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
     • System will generate a gc.log file
   • -XX:+HeapDumpOnOutOfMemoryError
     • System will generate a heap dump file
     • There is an additional HotSpot VM command line option that allows a user to specify a path where the heap dump will be placed
     • -XX:HeapDumpPath=
2. Analyze the log files:
   • Use regular text editor to examine gc.log file
   • Use Eclipse Memory Analyzer to examine heap dump file (i.e., java_xxx.hprof)

Sometimes your Java process could have run into  "GC overhead limit exceeded" and remain alive.  In this case, you may gather extra information using jstat:

jstat -gcutil
jstat -gccapacity

without restarting the Java process.  Note that all command options discussed in this article applied to Hotspot VM.

JVM Options

The command line options specify:

• -XX:+PrintGCDetails
  • prints more details at garbage collection.
• -XX:+PrintGCTimeStamps
  • prints a time stamp representing the number of seconds since the HotSpot VM was launched until the garbage collection occurred.
• -Xloggc:gc.log
  • causes information about the heap and garbage collection to be printed at each collection.

To set Java Options in JDeveloper, do the following:

1. Right select your project (i.e., ViewController) and bring up the context menu
2. Select Project Properties...
3. Select Run/Debug/Profile
4. Select your Run Configuration (i.e., Default)
5. Click Edit button
6. Specify -Xloggc:gc.log -XX:-PrintGCDetails in the Java Options field

Run your application and reproduce the out-of-memory exception. A log file named gc.log will be generated. I've found mine in the following default location:

• .../system11.1.1.5.37.60.13/DefaultDomain

because my web application was deployed to the Integrated WLS^[4] and run from DefaultDomain. To understand the format of gc.log, read [5,15] for details.

However, gc.log file was not really helpful because it simply pointed out there was a heap issue. But, it didn't say where.

The next step I have taken is running my server with the following flag:

-XX:+HeapDumpOnOutOfMemoryError

it generated a java_pid30835.hprof file when my server encountered a heap error.

Eclipse Memory Analyzer

The heap dump file (i.e., java_pid30835.hprof) is generated by HPROF—a heap and cpu profiling tool. My heap dump file was generated in binary format. Therefore I need to use Eclpse Memory Analyzer to examine it.

You can install Eclipse MAT via the Eclipse Update manager . Select "General Purpose Tools " and install "Memory Analyser (Incubation)" and "Memory Analyser (Charts)".
After installation, double-click your heap dump file and select "Leak Suspects Report".
Eclipse MAT will show a diagram:
and problem suspects:
You can click on the "Details" link to investigate.

Heap Size Adjustment

If you observe many full GCs, try to determine if your old generation is sized too small to hold all the live objects collected from the Survivor and Eden spaces. Alternatively, there may be too many live objects that do not fit into the configured heap size. If it is the latter, increase the overall heap size.

Based on whether the old generation space or the permanent generation space is running out of memory, you may adjust the sizes of heap and meta spaces in this way^[10]:

• For old generation space OutOfMemoryErrors
• Increase -Xms and -Xmx
• For permanent generation OutOfMemoryErrors
• Increase -XX:PermSize and -XX:MaxPermSize

References

1. Eclipse Update Manager
2. Eclipse Memory Analyzer
3. Java Hotspot VM Options
4. Integrated WebLogic Server (WLS)
5. Diagnosing a Garbage Collection problem
6. Frequently Asked Questions about Garbage Collection
7. GC Overhead Limit Exceeded
8. HPROF: A Heap/CPU Profiling Tool in J2SE 5.0
9. Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning
10. Java Performance by Charlie Hunt and Binu John
11. Understanding Garbage Collection
12. Java HotSpot VM Options
13. GCViewer (a free open source tool)
14. Understanding Garbage Collector Output of Hotspot VM
15. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded
16. Memory Analyzer Downloads
• The stand-alone Memory Analyzer is based on Eclipse RCP.
• Can find the update site here too.
17. Shallow vs. Retained Heap (MAT)
18. Diagnosing Heap Stress in HotSpot (XML and More)
19. Diagnosing OutOfMemoryError or Memory Leaks in JRockit (XML and More)
20. MAT Documentation
21. Out of Memory Error while Running the Memory Analyzer
• I need to modify -Xmx512m to -Xmx4g in the file eclipse.ini to analyze a 2GB hprof file.
22. Eclipse MAT: Querying Heap Objects Using OQL (Xml and More)
23. Eclipse MAT: Understand Incoming and Outgoing References  (Xml and More)
24. Memory Mapped File and IO in Java
• Memory used to load Memory mapped file is outside of Java heap Space.  If OOM is caused by memory-mapped files, you may want to reduce your Java heap allocation (i.e., reducing -Xmx).
• java.io.IOException: Map failed
25. jstat - Java Virtual Machine Statistics Monitoring Tool (XML and More)
26. Eclipse MAT — Incoming, Outgoing References (good)