JRockit: What's the Total Memory Footprint of a Java Process?

In [1], we have shown a case of performance tuning by sizing JRockit's Thread Local Area (TLA). For both test runs—TLA default and TLA tuned, they have been given the same heap size (i.e, 2g). However, in the conclusion, I have said:

Better performance is achieved by reducing pause time % in GC andTotal CPU % at the expense of total memory footprint (-1.6%).

In this article, we will clarify what the following phrase:

at the expense of total memory footprint (-1.6%)

means.

Java Heap vs Native Memory

As described in [2], total memory footprint of a Java process includes not only Java Heap, but also Native Memory. For example, you can use print_memusage to analyze the memory (including native memory) allocated by a Java process:

> ./jrcmd 26413 print_memusage >JR_print_memusage.txt

>cat JR_print_memusage.txt

26413:
Total mapped                  4897980KB           (reserved=1384484KB)
-              Java heap      2097152KB           (reserved=0KB)
-              GC tables        70156KB
-          Thread stacks        48324KB           (#threads=141)
-          Compiled code      1048576KB           (used=41160KB)
-               Internal         1480KB
-                     OS       401856KB
-                  Other       760932KB
-            Classblocks        27136KB           (malloced=26821KB #58548)
-        Java class data       441344KB           (malloced=439083KB #271063 in 58548 classes)
- Native memory tracking         1024KB           (malloced=242KB #10)

Native Memory

Internal JVM memory management is, to a large extent, kept off the Java heap and allocated natively in the operating system, through system calls like malloc. This non-heap system memory allocated by the JVM is referred to as native memory.

For JRockit, you can constraint the amount of memory allocated to the Java heap, but not the native memory, in a Java process.^[3] As shown above, process 26413 has been allocated 4897980KB (i.e., VSZ) in the virtual address space. However, this VSZ value is not very useful. What counts is the one reported by Resident Set Size (RSS; or physically resident memory). To find out the RSS of a Java process, you can do:

>ps -o pid,uid,state,rss,vsz,minflt,majflt,args -p 26413 >ps_26413.tmp
>cat ps_26413.tmp

PID   UID S   RSS    VSZ MINFLT MAJFLT COMMAND
26413 60000 S 3295216 4889792 15194852 1 /scratch/user1/JVMs/jdk-jr/bin/java ...

For the process 26413, its total memory footprint (or RSS) is 3295216 KB and this is the KPI we have quoted in our benchmark comparison.

References

1. JRockit: A Case Study of Thread Local Area (TLA) Tuning
2. Why is my JVM process larger than max heap size?
3. How to Debug Native OutOfMemory in JRockit (Xml and More)

JRockit: Thread Local Area Size and Large Objects

Large objects can be a problem for an application. If your Java application allocates a lot of objects, especially large objects, it helps to play around with various Thread Local Area (TLA) settings.^[1] For a case study, read [11].

If you try to find the default settings of TLA by using:

• -XX (an alias for -Xprintflags)

it prints out the following information in JRockit R28.2.5:

        TlaWasteLimit = 0 (default)
                (Alias: -XXlargeObjectLimit)
                - Internal. Use -XXtlaSize:wasteLimit instead.
        TlaMinSize = 0 (default)
                (Alias: -XXminBlockSize)
                - Internal. Use -XXtlaSize:min instead.
        TlaPreferredSize = 0 (default)
                - Internal. Use -XXtlaSize:preferred instead.

Hmm. That is not very helpful. In this article, we will discuss:

• Thread local area size and large objects
• How to find out the default TLA settings?

Thread Local Area Size and Large Objects^[1-4]

The thread local area (TLA) is a chunk of free space reserved on the heap or the nursery and given to a thread for its exclusive use. A thread can allocate small objects in its own TLA without synchronizing with other threads. When the TLA gets full the thread simply requests a new TLA. The objects allocated in a TLA are accessible to all Java threads and are not considered “thread local” in any way after they have been allocated.

Increasing the TLA size is beneficial for multi-threaded applications where each thread allocates a lot of objects. Increasing the TLA size is also beneficial when the average size of the allocated objects is large, as this allows larger objects to be allocated in the TLAs. Increasing the TLA size too much may however cause more fragmentation and more frequent garbage collections. Before any JRockit performance tuning, you need to assess the sizes of the objects allocated by your application. One way to access live data size is to view object allocation statistics. There are multiple ways of achieving that:

• You can create a JRockit Flight Recorder recording and view object allocation statistics in the JRockit Flight Recorder^[7]
• You can use the following JVM option:
• -Xverbose:memdbg Xverbose:gc -Xverbosedecorations=level,module,timestamp,millis,pid

Large Object: Pre-R28 vs. Post-R28^[1]

In JRockit versions earlier than R28, large objects were allocated immediately on the heap and never in a TLA. A flag called –XXlargeObjectLimit was provided to tell JRockit the minimum number of bytes an object should be of in order to be treated as "large". The default was 2 KB.

JRockit post R28 uses a waste limit for TLA space instead. This constrains the amount of TLA space that can be thrown away for each TLA when large objects are allocated and is a more flexible solution.

The R28 allocation algorithm now works like this—JRockit tries to allocate every object regardless of its size in the current TLA. If it doesn't fit and the waste limit is less than the space left in the TLA, the object
goes directly on the heap. Otherwise, JRockit will "waste" the rest of this TLA and try to allocate the object in a new TLA or directly on the heap, depending on the size of the object.

The TLA sizes are set using the following option:

• -XXtlaSize:min=size,preferred=size,wasteLimit=size^[8]
• min
• Sets the minimum TLA size. 
• preferred
• Sets a preferred TLA size.  This means that TLAs will be the preferred size whenever possible, but can be as small as the min size.
• wasteLimit
• Sets the waste limit for TLAs. This is the maximum amount of free memory that a TLA is allowed to have when a thread requires a new TLA.

The following relation is true for the TLA size parameters:

• -XXtlaSize:wasteLimit <= -XXtlaSize:min <= -XXtlaSize:preferred

Read [9] for the tuning advice.

What are the default TLA settings?

To find out the default TLA settings, you can use the following JVM option:^[12]

• -Xverbose:memdbg

For example, here is the output:

[DEBUG][memory ][19709] Minimum TLA size is 2048 bytes.
[DEBUG][memory ][19709] Preferred TLA size is 16384 bytes.
[DEBUG][memory ][1388121278654][19709] TLA waste limit is 2048 bytes.

from the following JVM options:

• -Xms2g -Xmx2g -Xverbose:memdbg -Xgc:pausetime -Xverbose:gc -Xverbosedecorations=level,module,timestamp,millis,pid

References

1. Oracle JRockit- The Definitive Guide
2. Tuning Java Virtual Machines (JVMs)
3. On Nursery Sizing (Migrated)
• The goal of nursery sizing to get as much memory as possible freed by young collections rather than old collections.
• The rule of thumb is that a nursery size of approximately half of the free memory on the heap is nearly optimal.
• If your application is sensitive to latencies you may want to decrease the nursery size to shorten the young collection pause times.
• If you're using the garbage collection mode optimizing for short pauses (-Xgcprio:pausetime) or the static generational concurrent garbage collector (-Xgc:gencon) you will most likely want to tune the nursery size manually.
4. First Steps for Tuning the Oracle JRockit JVM
5. Oracle® JRockit Diagnostics and Troubleshooting Guide (Release R28)
6. Oracle® JRockit Performance Tuning Guide (Release R28)
7. Oracle® JRockit Flight Recorder Run Time Guide (Release R28)
8. -XXtlaSize Parameters
9. Optimizing Memory Allocation Performance (Section 4.4 of this pdf)
10. Oracle® JRockit Command-Line Reference (Release R28)
11. JRockit: A Case Study of Thread Local Area (TLA) Tuning (Xml and More)
12. JRockit: Analyzing GC With JRockit Verbose Output (-Xverbose:memdbg)  (Xml and More)
13. Where did all of these ConstPoolWrapper objects come from?!

Default Values of JRockit's VM Options

Updated (09/19/2014): You can also add -XX:+UnlockInternalVMOptions to the command line to access JVM internal flags.

To find out what the JRockit's default VM options are, you type:

•  jdk-jr/bin/java -XX >JR_defaults.txt

Note that -XX is an alias for -Xprintflags in JRockit.

Default Values

Here are the output from this JRockit version:
  java version "1.6.0_37"
  Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
  Oracle JRockit(R) (build R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64, compiled mode)

Global:
        UnlockDiagnosticVMOptions = false (default, writeable)
                - Enable processing of flags relating to field diagnostics
        UnlockInternalVMOptions = false (default)
                - Enable processing of internal, unsupported flags
Class:
        FailOverToOldVerifier = true (default, writeable)
                - Fail over to old verifier when split verifier fails
        UseVerifierClassCache = true (default)
                - Try to cache java.lang.Class lookups for old verifier.
        UseClassGC = true (default)
                (Alias: -Xnoclassgc)
                - Allow GC of Java classes
Threads:
        UseThreadPriorities = false (default)
                - Use native thread priorities
        DeferThrSuspendLoopCount = 4000 (default, writeable)
                - Number of iterations in safepoint loop until we try blocking
        SafepointSpinBeforeYield = 2000 (default, writeable)
                - Number of iterations in safepoint loop until we yield instead
                  of pause (MP only)
        UseCompilerSafepoints = true (default)
                - Insert safepoint polls in compiled code
        DeferPollingPageLoopCount = -1 (default)
                - Number of iterations in safepoint loop before arming
                  safepoint poll page
        UseMembarForTransitions = false (default)
                - Use membar to serialize thread states.
        UseNativeLockProfiling = false (default)
                - Profile use of internal JVM monitors
        TrustPThreadStackInfo = false (default)
                - Trust information from pthreads about stack start and size
JNI:
        CheckJNICalls = false (default)
                - Verify all arguments to JNI calls
        AbortOnFailedJNICheck = true (default)
                - Used with CheckJNICalls. If true, abort the JVM upon first
                  JNI parameter error.
        ErrorOnFailedJNICheck = false (default)
                - Used with CheckJNICalls. If true, any errors will be
                  signalled through a java.lang.Error.
JDK:
        UseNewHashFunction = false (default)
                - Use HashMaps new hash function on jdks that does not do so by
                  default
        TreeMapNodeSize = 64 (default)
                - Size of entry array in each java.util.TreeMap node
        MaxDirectMemorySize = 0 (default)
                - Maximum total size of NIO direct-buffer allocations
        UseLazyStackTraces = true (default)
                - Generate stacktraces lazily for thrown Exceptions
        ShowInternalMethodsInStackTrace = false (default, writeable)
                - Show JVM internal code in java stacktraces.
        ExceptionTraceFilter = (null) (default)
                - Pattern that limits what exceptions are logged.
OS:
        ReduceSignalUsage = false (default)
                (Alias: -Xrs)
                - Reduce the use of OS signals in Java and/or the VM
        MaxFDLimit = true (default)
                - Maximize the available number of filedescriptors.
        MaxLargePageSize = 256M (default)
                - Use value as maximum size for large pages (if possible).
GC:
        UseLowAddressForHeap = true (default)
                - Use low 4Gb address space for Java heap if possible.
        UseLargePagesForHeap = false (default)
                - Attempt to use large page translation for the Java heap.
        ForceLargePagesForHeap = false (default)
                - Force the use of large page translation for the Java heap.
        CompressedRefs = false (default)
                (Alias: -XXcompressedRefs)
                - Use 32-bit java references on 64-bit OS - implies a heap
                  maximum of 4Gb (probably less)
        InitialHeapSize = 0 (default)
                (Alias: -Xms)
                - Initial size of Java Object heap
        MaxHeapSize = 0 (default)
                (Alias: -Xmx)
                - Maximum size of Java Object heap
        GCTimeRatio = 19 (default)
                - The ratio of time spent in garbage collection compared to
                  outside of garbage collection.
        GCTimePercentage = 0.000000 (default)
                - The percentage of time spent in garbage collection of total
                  run time.
        GCTrigger = 0 (default)
                (Alias: -XXgcTrigger)
                - The threshold of free heap before a concurrent GC is started
        ForceEarlyOC = true (default)
                - Force an early OC before old space is empty to avoid
                  promotion failed.
        ForceEarlyOCMaxPercentage = 5.000000 (default)
                - Maximum percentage of heap that is allowed to have left
                  before doing an early OC.
        ForceYCOnLargeAllocationFailed = false (default)
                - Force YC on a large allocation failure.
        UseNurseryEvacuation = false (default)
                - Try to evacuate the nursery when a promotion failed has
                  occured.
        DisableEvacuationToNursery = false (default)
                - Disallows evacuation to move objects to where the nursery is.
        NurseryPartsLimit = 10000 (default)
                - The maximum number of nursery parts we will allow before
                  forcing an early OC to fight fragmentation, or 0 for no
                  limit.
        SemiRefPostponedPacketSize = 492 (default)
                - The number of references in a postponed semiref packet.
        SemiRefPrefetchDistance = 0 (default)
                - The number of reference packet indexes to prefetch, or 0 for
                  no prefetch.
        FinalHandleParallelThreshold = -1 (default)
                - The minimum number of final handles needed to process them in
                  parallel.
        FinalHandlePacketSize = 200 (default)
                - The number of handles in a final handle packet.
        MaximumNurseryPercentage = 95 (default)
                - Sets the maximum size of the nursery relative to the amount
                  of free heap after the last old collection.
        AllowYCDuringOC = true (default)
                - Allow young collections during old collections.
        YcAlignAll = false (default)
                - Align all objects (with regards to YcAlignMaxSpill) during YC
        YcAlignMaxSpill = 40 (default)
                - Max spill allowed when aligning objects during YC.
        FullSystemGC = false (default)
                (Alias: -XXfullSystemGC)
                - Always run full GC (with full compaction) when System.gc() is
                  called
        AllowSystemGC = true (default)
                (Alias: -XXnoSystemGC)
                - Run a GC when System.gc() is called
        GcCardTableParts = 1024 (default)
                - Initial number of parts of the card table array
        GcBalancePrefetchDistance = 4 (default)
                - Prefetch distance in a GC balance system workpacket
        GcBalancePacketSize = 493 (default)
                - Packet size of GC balance system workpackets
        NumGenConPrecleaningIterations = 3 (default)
                - Number of precleaning iterations for gencon.
        AllowEmergencyParSweep = true (default)
                - Allow the OC to temporarily change concurrent sweep to
                  parallel if needed.
        UseCfsAdaptedYield = false (default)
                - Use a version of yield adapted for the CFS scheduler. Only
                  for use on CFS.
        TlaWasteLimit = 0 (default)
                (Alias: -XXlargeObjectLimit)
                - Internal. Use -XXtlaSize:wasteLimit instead.
        TlaMinSize = 0 (default)
                (Alias: -XXminBlockSize)
                - Internal. Use -XXtlaSize:min instead.
        TlaPreferredSize = 0 (default)
                - Internal. Use -XXtlaSize:preferred instead.
GC::Compaction:
        UseFullCompaction = false (default)
                - All compactions will be full compactions. Internal. Use
                  -XXcompaction:full instead.
        InternalCompactionPercentage = -1.000000 (default)
                - The percentage of the heap to compact for internal
                  compaction. Internal. Use -XXcompaction:internalPercentage
                  instead.
        ExternalCompactionPercentage = -1.000000 (default)
                - The percentage of the heap to compact for external
                  compaction. Internal. Use -XXcompaction:externalPercentage
                  instead.
        InitialCompactionPercentage = -1.000000 (default)
                - The initial percentage of the heap to compact, for both
                  internal and external compaction. Internal. Use
                  -XXcompaction:initialPercentage instead.
        UseCompaction = true (default)
                - Use compaction to reduce fragmentation. Internal. Use
                  -XXcompaction:enable instead.
        UseAbortableCompaction = false (default)
                - The compactions should be possible to abort. Internal. Use
                  -XXcompaction:abortable instead.
        NumCompactionHeapParts = 4096 (default)
                (Alias: -XXheapParts)
                - The number of heap parts in compaction heuristics. Internal.
                  Use -XXcompaction:heapParts instead.
        InitialExternalReservedHeap = 4M (default)
                - The initial size of the memory reserved by external
                  compaction. Internal. Use
                  -XXcompaction:initialExternalReservedHeap instead.
        UseFixedExternalReservedHeap = false (default)
                - The size of the memory reserved by external compaction is
                  fixed. Internal. Use
                  -XXcompaction:externalReservedHeapIsFixed instead.
        MaxCompactionReferences = 0 (default)
                (Alias: -XXcompactSetLimit)
                - The maximum number of references to store in compaction
                  before skipping. Internal. Use -XXcompaction:maxReferences
                  instead.
        MaxCompactionReferencesPerObject = 0 (default)
                (Alias: -XXcompactSetLimitPerObject)
                - The maximum number of references to store per object in
                  compaction. Internal. Use
                  -XXcompaction:maxReferencesPerObject instead.
        InternalCompactionParts = -1 (default)
                (Alias: -XXinternalCompactRatio)
                - Deprecated. Use -XXcompaction:internalPercentage instead.
        ExternalCompactionParts = -1 (default)
                (Alias: -XXexternalCompactRatio)
                - Deprecated. Use -XXcompaction:externalPercentage instead.
Object allocation:
        UseAllocPrefetch = true (default)
                - Use prefetch on object allocation
        RedoAllocPrefetch = true (default)
                - Do prefetch on object allocation from start of the allocated
                  object
        AllocPrefetchLineLength = -1 (default)
                - Line length for allocation prefetch
        AllocPrefetchDistance = -1 (default)
                - Distance for allocation prefetch
        AllocChunkSize = -1 (default)
                - Size of chunks to clear/prefetch
Javalock:
        UseLockProfiling = false (default)
                - Enable Java lock profiling.
        ThinLockContendedSpinCount = -1 (default)
                - Number of spins between each poll when acquiring a thin lock
        ThinLockContendedPollCount = -1 (default)
                - Number of polls between each short nap when acquiring a thin
                  lock
        ThinLockConvertToFatThreshold = -1 (default)
                - Number of of short naps before converting thin lock to fat
        FatLockContendedSpinCount = -1 (default)
                - Number of spins between each poll when acquiring a fat lock
        FatLockContendedPollCount = -1 (default)
                - Number of polls between each short nap when acquiring a fat
                  lock
        MonitorContendedSpinCount = -1 (default)
                - Number of spins between each poll when acquiring a monitor
                  lock
        MonitorContendedPollCount = -1 (default)
                - Number of polls between each short nap when acquiring a
                  monitor lock
        UseFatLockDeflation = true (default)
                - Try to deflate fat locks to thin
        FatLockDeflationThreshold = 50 (default)
                - Number of uncontended entries on lock before deflation occurs
        UseLockQueueLength = true (default)
                - Make threads go to sleep if contention exceeds # cpus
        UseFatSpin = true (default)
                (Alias: -XXdisableFatSpin)
                - Should we spin-try then acquiring a fat lock
        UseAdaptiveFatSpin = false (default)
                - Should we use adaptive spinning acquiring a fat lock
        UseThreadContentionMonitoring = true (default)
                - Allow thread contention monitoring
JavaLock::LazyUnlocking:
        UseLazyUnlocking = true (default)
                (Alias: -XXlazyUnlocking)
                - Enable lazy unlocking
        UseLazyUnlockingInJIT = true (default)
                - Use lazy locks in JIT code
        UseLazyUnlockingClassBan = true (default)
                - Use class banning
        UseLazyUnlockingTransferClassBan = true (default)
                - Use transfer class banning
JFR:
        FlightRecorder = true (default)
                - Enable flightrecorder
        FlightRecorderOptions = (null) (default)
                - Flight recorder arguments
        StartFlightRecording = (null) (default)
                - Start a Flight recording with args. Equivalent to using
                  "start_flightrecording".
Code memory:
        CodeBlockAbsorbtionSize = 32 (default)
                - Maximum extra size allowed for fitting code memory chunks
        FreeEmptyCodeBlocks = true (default)
                - Free unused code memory
        UseLargePagesForCode = false (default)
                - Attempt to use large page translation compiled code.
        MaxCodeMemory = 0 (default)
                - Maximum amount of memory used for generated code
        ReserveCodeMemory = true (default)
                - Reserve all memory for code at startup
        UseCodeGC = true (default)
                - Allow GC of discarded compiled code
        CodeGCThreshold = 0 (default)
                - Released byte threshold for initiating a code GC
        CodeGCReclaimThreshold = 0 (default)
                - Released byte threshold before compiler attempts to reclaim
                  unused code space
        CodeGCUseReclaim = true (default)
                - Should Code GC attempt reclamation of unused code memory
        CodeGCTaskInterval = 5 (default)
                - Interval in secs between background scans for unused code
                  blocks
Compiler broker:
        MaxOptQueueLength = 0 (default)
                - Maximum allowed optimization queue length before JIT thread
                  helps generate code
        OptThreads = 1 (default)
                - Number of background optimization threads
        JITThreads = 1 (default)
                - Number of background JIT threads
        JITThreadPrio = 5 (default)
                - Priority of background JIT threads
        OptThreadPrio = 5 (default)
                - Priority of background optimization threads
        DisableOptsAfter = -1 (default)
                (Alias: -Xnoopt)
                - Disable optimizations after n seconds
Compiler:
        PreOpt = false (default)
                - Optimize all code on jit (first generation)
        UseCallProfiling = false (default)
                - Use call profiling on unoptimized code
        StrictFP = false (default)
                (Alias: -Xstrictfp)
                - Force strict FP for all methods
        CheckStacks = false (default)
                (Alias: -Xcheckedstacks)
                - Do explicit checks for stack overflow
        DevirtualizeAlways = false (default)
                - Forces devirtualization in jitted code
        UseStringCache = false (default)
                - Cache common arrays used in String constructor
        MethodCodeAlignment = 32 (default)
                - Byte alignment for start of method code
        UseInlineObjectAlloc = true (default)
                - Generate inlined object allocation code.
        UseSafeTimer = false (default)
                - Use fast, signal based timer for System.currentTimeMillis.
        UseOldLockMatching = false (default)
                - Compatibility mode lock matching
JVMTI:
        JavaDebug = false (default)
                (Alias: -Xdebug)
                - Enable java debugging
Management:
        DisableAttachMechanism = false (default)
                - Disable mechanism that allows tools to attach to this VM
        CrashOnOutOfMemoryError = false (default, writeable)
                - Crash JVM process on OutOfMemory
        ExitOnOutOfMemoryError = false (default, writeable)
                - Terminate JVM process on OutOfMemory
        ExitOnOutOfMemoryErrorExitCode = 51 (default, writeable)
                - Exit code for termination of  JVM process on OutOfMemory
        HeapDiagnosticsOnOutOfMemoryError = false (default, writeable)
                - Print Java heap diagnostics on OutOfMemory
        HeapDiagnosticsPath = (null) (default, writeable)
                - When HeapDiagnosticsOnOutOfMemoryError is on, the path
                  (filename or directory) of the dump file (defaults to
                  jrockit_.oomdiag in the working directory)
        HeapDumpOnOutOfMemoryError = false (default, writeable)
                - Dump Java heap to a hprof binary format file on OutOfMemory
        HeapDumpOnCtrlBreak = false (default)
                - Dump heap to file in Ctrl-Break handler
        HeapDumpPath = (null) (default, writeable)
                - When HeapDumpOnOutOfMemoryError is on, the path (filename or
                  directory) of the dump file (defaults to jrockit_.hprof
                  in the working directory)
        SegmentedHeapDumpThreshold = 2G (default, writeable)
                - Generate a segmented heap dump (JAVA PROFILE 1.0.2 format)
                  when the heap usage is larger than this
        HeapDumpSegmentSize = 1G (default)
                - Approximate segment size when generating a segmented heap
                  dump
        StartMemleakOnPort = 0 (default, writeable)
                (Alias: -XXmemleak)
                - Listen for memleak connections on this port (0 for default)
        FlightRecordingDumpOnUnhandledException = false (default, writeable)
                - Generate a Flight Recording dump when a thread is terminated
                  due to an unhandled exception
        FlightRecordingDumpPath = (null) (default, writeable)
                - When FlightRecordingDumpOnUnhandledException is on, the path
                  (filename or directory) of the dump file (defaults to
                  jrockit__.jfr in the working directory)
Runtime:
        AbortOnCrash = false (default, writeable)
                - Abort the JVM in case of an crash.
        DumpOnCrash = true (default, writeable)
                - Generate a dump of the JVM state in case of a crash.
        CoreOnCrash = true (default, writeable)
                - Generate a core dump file of the JVM state in case of a
                  crash.
        WaitOnCrash = false (default, writeable)
                - Wait for user debugger attach in case of a crash.
        AbortOnAssert = true (default, writeable)
                - Abort JVM on assertion
        CrashOnAssert = false (default, writeable)
                - Crash dump the JVM on assertion
        WaitOnAssert = false (default, writeable)
                - Spin and wait JVM on assertion
        NumaMemoryPolicy = (null) (default)
                - Numa memory policy (interleave, preferredlocal, strictlocal)
        BindToNumaNodes = (null) (default)
                - Bind process to Numa nodes
        BindToCPUs = (null) (default)
                - Bind process to CPUs
        UseFastTime = true (default)
                - Force/disable usage of hardware platform support for fast
                  time
        UseJNIPinning = true (default)
                - Use pinning for Objects in call to GetPrimitiveArrayCritical
                  etc

See Also

1. What Are the Default HotSpot JVM Values? (XML and More)

Diagnosing OutOfMemoryError or Memory Leaks in JRockit

When you run into OutOfMemoryError or other memory-leak issues, generating a heap histogram or heap dump can help you diagnose the memory-bloating issues.

In [1], it lists the following Java Heap related problems:

• Exceeding max heap 
• The heap is full and cannot fit a new object
• Large allocation
• The new object is too large for the contiguous free space
• Native exhaustion
• There is not enough native heap for the requested object^[10]
• GC Starvation
• The heap is almost full and causing frequent garbage collection^[9]
• Optimization
• Heap utilization is higher than expected for the current number of users

In this article, we will discuss the following topics:

1. Heap histogram vs. heap dump (see also [8])
2. How to generate heap histogram or heap dump in JRockit
3. JVM options that are useful for heap analysis

Heap Histogram vs. Heap Dump

A heap dump is a snapshot of all the objects in the Java Virtual Machine (JVM) heap at a certain point in time. The JVM software allocates memory for objects from the heap for all class instances and arrays. The garbage collector reclaims the heap memory when an object is no longer needed and there are no references to the object. By examining the heap you can locate where objects are created and find the references to those objects in the source.  However, dumping of Java heap is time-consuming and lengthy in size.

On the other hand, heap histogram gives a very good summary of heap objects used in the application without doing a full heap dump. It can help you quickly narrow down a memory leak. This information can be obtained in several way:

• Attach a running process using the command jrcmd.
• Generate from a core file or heap dump

Note that we refer to heap histogram, heap summary, or heap diagnostics interchangeably in this article.

Generating Heap Histogram

A heap histogram can be obtained from a running process using the command:

• jrcmd 20488 heap_diagnostics

--------- Detailed Heap Statistics: ---------
30.0% 65027k   672176     -1k [C
10.2% 22119k   943754     -2k java/lang/String
10.0% 21592k   183036     -3k [Ljava/lang/Object;
 4.7% 10185k   434587     +0k java/util/HashMap$Entry
 4.7% 10114k    27029  -1254k [B
 4.5% 9783k   111539     +0k [Ljava/util/HashMap$Entry;
 1.9% 4075k    34777     +0k java/lang/Class
 1.9% 4058k    86590     +0k java/util/HashMap
 1.6% 3448k   147156     +0k javax/management/ObjectName$Property
 1.2% 2593k    82994     +0k java/util/LinkedHashMap$Entry
 1.1% 2398k    76765     +0k java/util/concurrent/ConcurrentHashMap$Segment
 1.0% 2215k     9311     +0k [I
 0.9% 1975k    18469     +0k [J

In the output, there is a "Detailed Heap Statistics" section, which shows the total size and instance count for each class type in the heap:

• The first column corresponds to the Class object type contribution to the Java Heap footprint in %
• The second column correponds to the Class object type memory footprint in K
• The third column correponds to the # of Class instances of a particular type
• The fourth column correponds to the delta - / + memory footprint of a particular type

As you can see from the above snapshot, the biggest data type is [C (i.e., character array) and java.lang.String. In order to see which data types are leaking, you will probably need to generate several snapshots, which you might be able to observe a trend that can lead to further analysis.

Generating Heap Dump

Heap dump is a file containing all the memory contents of a Java application. It can be generated via 

• jrcmd 20488 hprofdump

• Wrote dump to /.../appmgr/APPTOP/instance/debug/jrockit_20488.hprof

Then you can use various tools to load that file and look at various things in the heap: how much each kind of object is using, what things are holding onto the most amount of memory, and so on.  The size of heap dump file is proportional to the size of Java Heap and can be large.

Three of the most common tools are:

• jhat
• This is the original heap analyzer tool, which reads the heap dump and runs a small HTTP server that lets you look at the dump through a series of web page links.
• VisualVM ^[3]
• MAT ^[4,5]

Heap-Related JVM Options

When your JVM runs into OutOfMemoryError, you can set:

• -XX:+HeapDumpOnOutOfMemoryError

to get a heap dump after the heap is big and bloated just before the JVM dies.  Also, you can provide the following flags:

• -XX:HeapDumpPath=<path to the destination>
• -XX:+ExitOnOutOfMemoryError

Similarly, you can get a heap histogram instead of a full heap dump using^[7]:

• -XX:+HeapDiagnosticsOnOutOfMemoryError 
• -XX:HeapDiagnosticsPath=<path to the destination>

References

1. Java™ Memory: Understanding and Analyzing the Java Heap
2. Getting histogram from .hprof heap dump file
3. VisualVM
4. Memory Analyzer (MAT)
5. Diagnosing Java.lang.OutOfMemoryError (XML and More)
6. Troubleshooting Memory Leaks
7. Default Values of JRockit's VM Options (XML and More)
8. Diagnosing Heap Stress in HotSpot (XML and More)
9. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded (XML and More)
10. Memory Mapped File and IO in Java
• Memory used to load Memory mapped file is outside of Java heap Space.  If OOM is caused by memory-mapped files, you may want to reduce your Java heap allocation (i.e., reducing -Xmx).

JRockit: Unable to open temporary file /mnt/hugepages/jrock8SadIG

After system maintenance, our JRockit failed to create the Java virtual machine because it could not acquire the large pages as shown below:

$ bin/java -Xms2560m -Xmx2560m -XlargePages -Xgc:genpar -XlargePages:exitOnFailure -version
[ERROR][osal   ] Unable to open temporary file /mnt/hugepages/jrock8SadIG
[ERROR][memory ] Could not acquire large pages for Java heap.
[ERROR][memory ] Could not setup java heap.
Could not create the Java virtual machine.

In this article, we will discuss how to investigate and resolve this issue.

What to Check?

When using JRockit we have to make a hugetblfs file system available in the directory /mnt/hugepages. Since the message says that it cannot open temporary file /mnt/hugepages/jrock8SadIG, the first thing to check is if that directory was mounted (note that it was mounted before system maintenance).

$ mount -l
nodev on /mnt/hugepages type hugetlbfs (rw)

As shown above, /mnt/hugepages directory was mounted.  The next thing to check is why JRockit cannot open temporary file in that directory.  Could it be privilege?

As described in [1], to make the /mnt/hugepages directory accessible for the oracle user, you need to do:

• chmod -R 777 /mnt/hugepages

It turns out that it was the culprit.  After issuing the above command, JRockit was able to create Java virtual machine.

References

1. Tune the JVM that runs Coherence
2. JRockit: Could not acquire large pages for 2Mbytes code
3. How to Test Large Page Support on Your Linux System
4. Understanding Application Memory Performance - Red Hat

JRockit: Could not acquire large pages for 2Mbytes code

When we tried to enable Large Pages for JRockit, we have seen the following message from the WebLogic Server console output:

• WARN codegc Could not acquire large pages for 2Mbytes code (at 0x2aaab0622000).

In this article, we will show you how to investigate and resolve this issue.

How to Test for Large Pages Support?

Similar to [1], here are the VM options for testing large-pages support for JRockit:

$  bin/java -Xms2560m -Xmx2560m -XlargePages -Xgc:genpar -XlargePages:exitOnFailure -version

When we ran the above command, we have seen the following warning:

[WARN ][codegc ] Could not acquire large pages for 2Mbytes code (at 0x2aaab0622000).
[WARN ][codegc ] Falling back to normal page size.
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64, compiled mode)

Enabling Large Pages Support in Linux Kernel

We have followed the procedure described in [2,5] to enable huge-pages support in the Linux kernel.  One of the requirements is to mount hugepages directory on the hugetlbfs type filesystem^[6] (note that this step is required for JRockit, but not HotSpot).  For example, we have hugepages directory mounted as follows:

$ mount -l
nodev on /mnt/hugepages type hugetlbfs (rw,noexec,nosuid,nodev,sync,uid=59951)

How to Investigate?

When we tried to explicitly disabled large pages for Java code (but not Java heap) , the following VM options ran fine:

$  bin/java -Xms2560m -Xmx2560m -XlargePages -Xgc:genpar -XlargePages:exitOnFailure -XX:+UseLargePagesForHeap -XX:-UseLargePagesForCode -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=false  -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64, compiled mode)


So, the issue is related to Java code reservation in Huge Pages.  After further investigation, we have found the real problem is that: when we mounted hugepages directory, we have chosen the following option:

• noexec
• Do not allow direct execution of any binaries on the mounted filesystem

After we have removed that constraint and rebooted the system, we have finally resolved the issue as shown below:

$  bin/java -Xms2560m -Xmx2560m -XlargePages -Xgc:genpar -XlargePages:exitOnFailure -XX:+UseLargePagesForHeap -XX:+UseLargePagesForCode -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=false -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64, compiled mode)

Here are the settings of newly mounted hugepages directory:

$ mount -l
nodev on /mnt/hugepages type hugetlbfs (rw)

Acknowledgement

This issue was resolved based on the feedbacks from Scott Oats.

References

1. How to Test Large Page Support on Your Linux System?
2. Java SE Tuning Tip: Large Pages on Windows and Linux 
3. Oracle® JRockit Command-Line Reference Release R28
4. Memlock limit too small (one of the requirements for large page support)
5. How to acquire large pages for Java heap
6. Linux / Unix Command: mount
7. Oracle® JRockit Performance Tuning Guide Release R28

Java Performance Tips

Every little things add up. The overall performance of an application depends on the individual performance of its components. Here we list some of the Java programming tips to help your application perform:

Programming Tips	Why and How
Avoid creating duplicate objects	Why: Less objects less GC. Application with smaller footprint performs better. How: Reuse a single object instead of creating a new functionally equivalent object each time it's needed. (Do) String s = "No longer silly"; (Don't) String s = new String("silly"); Use static factory methods in preference to constructors on immutable classes that provide both. Reuse mutable objects that are never modified once their values have been computed (within a static initializer).
Avoid circular references	Why: Groups of mutually referencing objects which are not directly referenced by other objects and are unreachable can thus become permanently resident. How: You can use strong references for "parent-to-child" references, and weak references for "child-to-parent" references, thus avoiding cycles.
Caching frequently-used data or objects	Why: One of the canonical performance-related use cases for Java is to supply a middle tier that caches data from back-end database resources. (Warning) Like all cases where objects are reused, there is a potential performance downside: if the cache consumes too much memory, it will cause GC pressure. How: Code should use a PreparedStatement rather than a Statement for its JDBC calls. Reuse JDBC connections because connections to a database are time-consuming to create.
Use the == operator instead of the equals(Object) method	Why: == operator performs better. For example, for String comparison, the equals( ) method compares the characters inside a String object. The == operator compares two object references to see whether they refer to the same instance. How: a.equals(b) if and only if a == b For example, use static factory method to return the same object from repeated invocations.
Eliminate obsolete object references	Why: Reduced performance with more garbage collector activity Reduced performance with increased memory footprint How: Null out references once they become obsolete (Do) public Object pop() { if (size == 0) throw new EmptyStackException(; Object result = elements[--size]; elements[size] == null; // Eliminate obsolete reference return result; } (Don't) public Object pop(){ if (size == 0) throw new EmptyStackException(; return elements[--size]; }
Avoid finalizers	Why: Objects waiting for fnalization have to be kept track of separately by the garbage collector.  There is also call overhead when the finalize method is invoked Finalizers are unsafe in that they can resurrect objects and interfere with the GC.
Avoid reference objects	Why: As with fnalizers, the garbage collector has to treat soft, weak, and phantom references specially.  Although all of these can provide great aid in, for example, simplifying a cache implementation, too many live Reference objects will make the garbage collector run slower.  A Reference object is usually a magnitude more expensive than a normal object (strong reference) to bookkeep. To find out the number of references and the time it takes to process them in HotSpot, add the following JVM option: -XX:+PrintReferenceGC
Avoid object pooling	Why: Object pooling contributes both to more live data and to longer object life spans. Note that large amounts of live data is a GC bottleneck and the GC is optimized to handle many objects with short life spans. Also, allocating fresh objects instead of keeping old objects alive, will most likely  be more beneficial to cache locality. However, performance in an environment with many large objects, for example large arrays, may occasionally benefit from object pooling.
Choose good algorithms and data structures	Why: Consider a queue implementation in the form of a linked list. Even if your program never iterates over the entire linked list, the garbage collector still has to. Bad cache locality can ensue because payloads or element wrappers aren't guaranteed to be stored next to each other in memory. This will cause long pause times as, if the object pointers are spread over a very large heap area, a garbage collector would repeatedly miss the cache while doing pointer chasing during its mark phase.
Avoid System.gc	Why: There is no guarantee from the Java language specifcation that calling System.gc will do anything at all. But if it does, it probably does more than you want or doesn't do the same thing every time you call it.
Avoid too many threads	Why: The number of context switches grows proportionally to the number of fairly scheduled threads, and there may also be hidden overhead here. For example, a native thread context on a system such as Intel IA-64 processor is on the order of several KB.
Avoid contented locks	Why: Contended locks are bottlenecks, as their presence means that several threads want to access the same resource or execute the same piece of code at the same time.
Avoid unnecessary exceptions	Why: Handling exceptions takes time and interrupts normal program flow.  Authors[1] have seen cases with customer applications throwing tens of thousands of unnecessary NullPointerExceptions every second, as part of normal control fow. Once this behavior was rectifed, performance gains of an order of magnitude were achieved.
Avoid large objects	Why: Large objects sometimes have to be allocated directly on the heap and not in thread local areas (TLA). Large objects on the heap are bad in that they contribute to fragmentation more quickly. Large object allocation in VMs (for example, in JRockit) also contributes to overhead because it may require taking a global heap lock on allocation. An overuse of large objects leads to full heap compaction being done too frequently, which is very disruptive and requires stopping the world for large amounts of time.

Tuning Guidelines

As stated in [6], here are the guidelines for tuning your applications:

You should take an overall system approach to ensure that you cover all facets of the environment in which the application runs. The overall system approach begins with the external environment and continues drilling down into all parts of the system and application. Taking a broad approach and tuning the overall environment ensures that the application will perform well and that system performance can meet all of your requirements.

References

1. Oracle JRockit
2. Effective Java by Joshua Bloch
3. hashCode() and equals() in Java
4. HotSpot VM Performance Tuning Tips
5. Java Performance by Charlie Hunt, Binu John, David Dagastine
6. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston
7. Sun Performance and Tuning: Java and the Internet by Adrian Cockroft and Richard Pettit
8. Concurrent Programming in Java: Design Principles and Patterns by Doug Lea
9. Capacity Planning for Web Performance: Metrics, Models, and Methods by Daniel A. Menascé and Virgilio A.F. Almeida
10. Big-O complexities of common algorithms (Cheat Sheet)

Understanding JVM Thread States

To investigate CPU issues in Java applications, one approach is to diagnose monitor locks and thread activities^[1].

In this article, we will show you:

• How to generate thread dumps on different VMs
• What to know about thread states

JVM Thread States

A thread can be in only one state at a given point in time. These states are virtual machine states which do not reflect any operating system thread states. As an example, here shows different threads in different states on a HotSpot VM:

$ cat thread.tmp | grep "java.lang.Thread.State" | sort | uniq -c
      3    java.lang.Thread.State: BLOCKED (on object monitor)
     18    java.lang.Thread.State: RUNNABLE
      6    java.lang.Thread.State: TIMED_WAITING (on object monitor)
      2    java.lang.Thread.State: TIMED_WAITING (sleeping)
     13    java.lang.Thread.State: WAITING (on object monitor)
      3    java.lang.Thread.State: WAITING (parking)

Below we describe some of the thread states that can be found in a thread dump:

• NEW - this state represents a new thread which is not yet started.
• RUNNABLE - this state represents a thread which is executing in the underlying JVM. Here executing in JVM doesn't mean that the thread is always executing in the OS as well - it may wait for a resource from the Operating system like the processor while being in this state.
• BLOCKED (on object monitor)- this state represents a thread which has been blocked and is waiting for a moniotor to enter/re-enter a synchronized block/method. A thread gets into this state after calling Object.wait method.
• WAITING - this state represnts a thread in the waiting state and this wait is over only when some other thread performs some appropriate action. A thread can get into this state either by calling - Object.wait (without timeout), Thread.join (without timeout), or LockSupport.park methods.
• TIMED_WAITING - this state represents a thread which is required to wait at max for a specified time limit. A thread can get into this state by calling either of these methods: Thread.sleep, Object.wait (with timeout specified), Thread.join (with timeout specified), LockSupport.parkNanos, LockSupport.parkUntil
• TERMINATED - this state reprents a thread which has completed its execution either by returning from the run() method after completing the execution OR by throwing an exception which propagated from the run() method and hence caused the termination of the thread.
• WAITING (parking)- it means a wait state after being parked. A thread can suspend its execution until permit is available (or thread is interrupted, or timeout expired, etc) by calling park(). You can give permit to a thread by calling unpark(). When permit is available, the parked thread consumes it and exits a park() method. Unlike Semaphore's permits, permits of LockSupport are associated with threads (i.e. permit is given to a particular thread) and doesn't accumulate (i.e. there can be only one permit per thread, when thread consumes the permit, it disappears).

Notes:

1. It's hard to investigate any CPU issue just from one thread dump. So, you need to prepare a series of thread dumps for investigation. For an example of analyzing hanging problems, see [9].
2. The above thread state information is for HotSpot.  To understand JRockit's thread dump contents, read [4].

Generating Thread Dumps in HotSpot

$jcmd 15679 Thread.print >thread.tmp

where 15679 is the process ID. For example, it will print the following messages:

15679:
2012-08-24 11:27:27
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.0-b21-internal mixed mode):

"Attach Listener" daemon prio=10 tid=0x000000000528e000 nid=0x4653 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Thread-29" daemon prio=10 tid=0x00002aaab9447000 nid=0x3fe3 runnable [0x000000004408b000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00000000f5ae4738> (a java.io.BufferedInputStream)
        at com.sun.jndi.ldap.Connection.run(Connection.java:849)
        at java.lang.Thread.run(Thread.java:722)

Generating Thread Dumps in JRockit^[4]

$jrcmd 1286 print_threads

where 1286 is the process ID. For example, it will print the following messages:

1286:

===== FULL THREAD DUMP ===============
Thu Aug 23 17:36:30 2012
Oracle JRockit(R) R28.1.3-11-141760-1.6.0_24-20110301-1432-linux-x86_64

"Main Thread" id=1 idx=0x4 tid=1287 prio=5 alive, waiting, native_blocked
    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xe48b4598[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)
    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xe48b4598[fat lock]
    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)
    at weblogic/Server.main(Server.java:71)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace

"(Signal Handler)" id=2 idx=0x8 tid=1288 prio=5 alive, native_blocked, daemon

"(OC Main Thread)" id=3 idx=0xc tid=1289 prio=5 alive, native_waiting, daemon
...

References

1. Understanding Threads and Locks
2. Thread States Diagram
3. Useful tool: jrcmd
4. Using Thread Dumps
5. Understanding a Java thread dump
6. Fun with JStack 
7. Java 2 Platform, Standard Edition 5.0 "Trouobingshooting and Diagnostic Guide"
8. HotSpot VM Performance Tuning Tips
9. Analyze Hanging Programs Using Java Thread Traces (XML and More)
10. Analyzing Thread Dumps in Middleware - Part 1
11. Analyzing Thread Dumps in Middleware - Part 2