Sunday, May 15, 2011

Diagnosing Java.lang.OutOfMemoryError

The heap is one of the foremost components that should be monitored to trace performance issues[18, 19,22]. Heap pressure is created when the heap usage approaches the maximum heap size permitted. This leads to frequent full garbage collection events. This steals CPU cycles available for processing and the overall response times degrade. Extreme cases can lead to OutOfMemory conditions, which are not recoverable without a JVM restart.

OutOfMemoryError

When I ran my application, the following exceptions have been thrown in sequence:
  • java.lang.OutOfMemoryError: GC overhead limit exceeded[7,9,15]
  • java.lang.OutOfMemoryError: Java heap space
The first message means that, for some reason, the garbage collector is taking an excessive amount of time and recovers very little memory in each run. After I removed the following statement:
  • System.gc();
The 1st message was gone. However, the system threw the 2nd message. So, obviously my heap space issue remains. Here are the steps that I took to investigate it:
  1. Add the following Java Options
    • -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
      • System will generate a gc.log file
    • -XX:+HeapDumpOnOutOfMemoryError
      • System will generate a heap dump file
      • There is an additional HotSpot VM command line option that allows a user to specify a path where the heap dump will be placed
        • -XX:HeapDumpPath=
  2. Analyze the log files:
    • Use regular text editor to examine gc.log file
    • Use Eclipse Memory Analyzer to examine heap dump file (i.e., java_xxx.hprof)
    Sometimes your Java process could have run into  "GC overhead limit exceeded" and remain alive.  In this case, you may gather extra information using jstat:
    jstat -gcutil
    jstat -gccapacity
    without restarting the Java process.  Note that all command options discussed in this article applied to Hotspot VM.

    JVM Options

    The command line options specify:
    • -XX:+PrintGCDetails
      • prints more details at garbage collection.
    • -XX:+PrintGCTimeStamps
      • prints a time stamp representing the number of seconds since the HotSpot VM was launched until the garbage collection occurred.
    • -Xloggc:gc.log
      • causes information about the heap and garbage collection to be printed at each collection.

    To set Java Options in JDeveloper, do the following:
    1. Right select your project (i.e., ViewController) and bring up the context menu
    2. Select Project Properties...
    3. Select Run/Debug/Profile
    4. Select your Run Configuration (i.e., Default)
    5. Click Edit button
    6. Specify -Xloggc:gc.log -XX:-PrintGCDetails in the Java Options field
    Run your application and reproduce the out-of-memory exception. A log file named gc.log will be generated. I've found mine in the following default location:
    • .../system11.1.1.5.37.60.13/DefaultDomain
    because my web application was deployed to the Integrated WLS[4] and run from DefaultDomain. To understand the format of gc.log, read [5,15] for details.

    However, gc.log file was not really helpful because it simply pointed out there was a heap issue. But, it didn't say where.

    The next step I have taken is running my server with the following flag:

    -XX:+HeapDumpOnOutOfMemoryError

    it generated a java_pid30835.hprof file when my server encountered a heap error.

    Eclipse Memory Analyzer

    The heap dump file (i.e., java_pid30835.hprof) is generated by HPROF—a heap and cpu profiling tool. My heap dump file was generated in binary format. Therefore I need to use Eclpse Memory Analyzer to examine it.

    You can install Eclipse MAT via the Eclipse Update manager . Select "General Purpose Tools " and install "Memory Analyser (Incubation)" and "Memory Analyser (Charts)".
    After installation, double-click your heap dump file and select "Leak Suspects Report".
    Eclipse MAT will show a diagram:
    and problem suspects:
    You can click on the "Details" link to investigate.

    Heap Size Adjustment

    If you observe many full GCs, try to determine if your old generation is sized too small to hold all the live objects collected from the Survivor and Eden spaces. Alternatively, there may be too many live objects that do not fit into the configured heap size. If it is the latter, increase the overall heap size.

    Based on whether the old generation space or the permanent generation space is running out of memory, you may adjust the sizes of heap and meta spaces in this way[10]:
    • For old generation space OutOfMemoryErrors
      • Increase -Xms and -Xmx
    • For permanent generation OutOfMemoryErrors
      • Increase -XX:PermSize and -XX:MaxPermSize

    References
    1. Eclipse Update Manager
    2. Eclipse Memory Analyzer
    3. Java Hotspot VM Options
    4. Integrated WebLogic Server (WLS)
    5. Diagnosing a Garbage Collection problem
    6. Frequently Asked Questions about Garbage Collection
    7. GC Overhead Limit Exceeded
    8. HPROF: A Heap/CPU Profiling Tool in J2SE 5.0
    9. Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning
    10. Java Performance by Charlie Hunt and Binu John
    11. Understanding Garbage Collection
    12. Java HotSpot VM Options
    13. GCViewer (a free open source tool)
    14. Understanding Garbage Collector Output of Hotspot VM
    15. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded
    16. Memory Analyzer Downloads
      • The stand-alone Memory Analyzer is based on Eclipse RCP.
      • Can find the update site here too.
    17. Shallow vs. Retained Heap (MAT)
    18. Diagnosing Heap Stress in HotSpot (XML and More)
    19. Diagnosing OutOfMemoryError or Memory Leaks in JRockit (XML and More)
    20. MAT Documentation
    21. Out of Memory Error while Running the Memory Analyzer
      • I need to modify -Xmx512m to -Xmx4g in the file eclipse.ini to analyze a 2GB hprof file.
    22. Eclipse MAT: Querying Heap Objects Using OQL (Xml and More)
    23. Eclipse MAT: Understand Incoming and Outgoing References  (Xml and More)
    24. Memory Mapped File and IO in Java
      • Memory used to load Memory mapped file is outside of Java heap Space.  If OOM is caused by memory-mapped files, you may want to reduce your Java heap allocation (i.e., reducing -Xmx).
        • java.io.IOException: Map failed
    25. jstat - Java Virtual Machine Statistics Monitoring Tool (XML and More)
    26. Eclipse MAT — Incoming, Outgoing References (good)

    3 comments:

    Spyros Doulgeridis said...

    Excellent post as usual!!! Thorough, analytic but not boring :)

    Are there any advantages in using
    Eclipse Memory Analyzer instead of Java VisualVM (http://sdoulger.blogspot.com/2011/03/jvisualvm-aka-java-visualvm.html)?

    Travel for Life said...

    >Eclipse Memory Analyzer vs. Java VisualVM
    I would say there is not any advantages in using MAT instead of Java VisualVM. It's just personal preference (i.e., familiarity with Eclipse).

    Unknown said...

    I tend to disagree (ok I'm biased ;-) ), but the Eclipse Memory Analyzer is still (after VisualVM "copied" some features) much more feature rich and also faster for large heap dumps.

    Regards,
    Markus