Spark on YARN: Sizing Executors and Other Tuning Ideas

Spark on YARN leverages YARN services for resource allocation, runs Spark executors in YARN containers, and supports workload management and Kerberos security features. It supports two modes:^[2]

• YARN-cluster mode
• Optimized for long-running production jobs
• YARN-client mode
• Best for interactive use such as prototyping, testing, and debugging
• Spark shell and the Spark Thrift server run in YARN-client mode. 

In this article, we will use YARN-cluster mode for illustration (see Figure 1).  Before jump in the tuning part, you should read [1] first.

Figure 1.  YARN-clsuter mode

Spark Application

When tuning Spark applications, it is important to understand how Spark works and what types of resources your application requires. For example, machine learning tasks are usually CPU intensive, whereas extract-transform-load (ETL) operations are I/O intensive.

Using PageRank benchmark in HiBench as an example, a spark application can be submitted with following command line:^[6]

$ /usr/hdp/current/spark-client/bin/spark-submit  

--properties-file ./BDCSCE-HiBench/report/pagerank/spark/scala/conf/sparkbench/spark.conf --class org.apache.spark.examples.SparkPageRank --master yarn-cluster --num-executors 17 --executor-cores 5 --executor-memory 19G --driver-memory 2G ./BDCSCE-HiBench/src/sparkbench/target/sparkbench-5.0-SNAPSHOT-MR2-spark1.6-jar-with-dependencies.jar 

"properties-file" option  allow you to provide a path to a file from which to load extra properties.  However, the most important tuning parts (shown on the command line) are to size up your executors right:

• --num-executors
• Number of Spark executors to launch (default: 2)
• --executor-cores
• Number of cores per executor. (Default: 1 in YARN mode, or all available cores on the worker in standalone mode)
• --executor-memory
• Memory per executor (e.g. 1000M, 2G) (Default: 1G).

Figure 2.  spark.shuffle.manager = sort^[5]

Tuning Guidelines

To do our tuning exercise, we use the following YARN cluster as an example:

• Cluster Size: Total 6 Nodes
• 16 Cores each
• 64 GB or RAM each
• Total Cores
• 96 
• Total Memory
• 384 GB

Using [3] as our reference, here are the tuning guidelines:

• Want to run multiple tasks in the same JVM (see Figure 2)
• Need to leave some memory overhead for OS/Hadoop daemons
• Need some overhead for off heap memory
• Configured by spark.yarn.executor.memory.overhead
• Default: max (384MB, .07 * spark.executor.memory)
• YARN Application Master needs a core in both  YARN modes:
• YARN-client mode
• Spark Driver is run in the Application client process
• YARN-cluster mode
• Spark Driver in run in the Spark ApplicationMaster process
• Optimal HDFS I/O Throughput
• Best is to keep 5 cores per executor
• No Spark shuffle block can be greater than 2 GB
• Read [3] for explanation
• Spark SQL
• Especially problematic for Spark SQL
• Default number of partitions to use when doing shuffle is 200
• Low number of partitions leads to
• high shuffle block size and could exceed the 2GB limit
• not making good use of parallelism
• Rule of thumb is around 128 MB per partition

Calculations

Based on the guidelines, here are the steps to size executors:

• 5 cores per executor
• For max HDFS throughput
• Cluster has 6 x 15 = 90 cores in total
• After taking out Hadoop/YARN daemon cores
• 90 cores / 5 cores per executor = 18 executors
• 1 executor for ApplicationMaster => 17 executors
• Each node has 3 executors
• 63 GB/3 = 21 GB per executor
• 21 x (1-0.07) ~ 19 GB (counting off heap overhead)

Correct Answer

The final settings are shown in our previous "spark-sumbit" command line, which has the following values:

• 17 executors
• Note that Executors are not be released until the job finishes, even if they are no longer in use. Therefore, do not overallocate executors above the estimated requirements
• 19 GB memory each
• 5 cores each

Note also that you should use the settings above as the starting point and fine tune them and/or other parameters further based on your network/disk capabilities and other factors (e.g., CPU intensive job or I/O intensive job).

Extra Tuning Ideas

Based on [4,6,9], here are more tuning ideas:

• --executor-cores or spark.executor.cores
• HDFS client could have trouble with tons of concurrent threads. A rough guess is that at most 5  tasks per executor can achieve full write throughput, so it’s good to keep the number of cores per executor equal to or below 5.
• --executor-memory or spark.executor.memory
• Controls the executor heap size, but JVMs can also use some memory off heap, for example for interned Strings and direct byte buffers. 
• spark.yarn.executor.memoryOverhead
• The value of the spark.yarn.executor.memoryOverhead property is added to the executor memory to determine the full memory request to YARN for each executor.
• Need to increase memoryOverhead size if you see:
• Container killed by YARN for exceeding memory limits. 7.0 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead
• --driver-memory
• Driver memory does not need to be large if the job does not aggregate much data (as with a collect() action)
• --num-executors vs --executor-memory
• There are tradeoffs between num-executors and executor-memory:
• Large executor memory does not imply better performance, due to JVM garbage collection. Sometimes it is better to configure a larger number of small JVMs than a small number of large JVMs.
• Running executors with too much memory often results in excessive garbage collection delays. 
• 64GB is a rough guess at an upper limit for a single executor
• spark.serializer
• Consider switching from the default serializer to the Kryo serializer (i.e., org.apache.spark.serializer.KryoSerializer) to improve performance
• spark.executor.extraJavaOptions
• Always print out the details of GC events with the following settings to aid debugging:
• -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:executor_gc.log 
• Note that executor_gc.log can be found on  Data Nodes where executors are running on
• spark.network.timeout
• Default timeout for all network interactions. This config will be used in place of 
• spark.core.connection.ack.wait.timeout
• spark.storage.blockManagerSlaveTimeoutMs
• spark.shuffle.io.connectionTimeout
• spark.rpc.askTimeout
• spark.rpc.lookupTimeout if they are not configured.
• If you run into the following timeouts:
• Executor heartbeat timed out after 121129 ms
• Consider set spark.network.tiemout to be higher value especially when you have heavy workloads^[9]
• spark.local.dir
• If you run into the following exception:
• java.io.IOException: No space left on device
• The best way to resolve this issue and to boost performance is to give as many disks as possible to handle scratch space disk IO
• Because Spark constantly writes to and reads from its scratch space, disk IO can be heavy and can slow down your workload
• Consider explicitly define parameter spark.local.dir in spark-defaults.conf configuration file to be something like:
• spark.local.dir   /data1/tmp,/data2/tmp, etc.
• yarn.nodemanager.resource.memory-mb 
• Controls the maximum sum of memory used by the containers on each node.
• yarn.nodemanager.resource.cpu-vcores 
• Controls the maximum sum of cores used by the containers on each node
• yarn.scheduler.minimum-allocation-mb 
• Controls the minimum request memory value
• yarn.scheduler.increment-allocation-mb 
• Control the increment request memory value
• Tuning parallelism
• Every Spark stage has a number of tasks, each of which processes data sequentially. In tuning Spark jobs, this number is probably the single most important parameter in determining performance.
• Read [4] for good tuning advice

References

1. Apache YARN一Knowing the Basics
2. Apache Spark Component Guide
3. Top 5 Mistakes to Avoid When Writing Apache Spark Applications
• How to adjust number of shuffle partitions
• Spark SQL
• use spark.sql.shuffle.partitions
• Others
• use rdd.repartition() or rdd.coalesce()
4. How-to: Tune Your Apache Spark Jobs (Part 2)
• The two main resources that Spark (and YARN) think about are CPU and memory. Disk and network I/O, of course, play a part in Spark performance as well, but neither Spark nor YARN currently do anything to actively manage them.
5. Spark Architecture: Shuffle
6. Tuning Spark (Hortonworkds)
7. Spark 2.1.0 Configuration
8. Spark 1.6.1 Configuration
9. Troubleshooting and Tuning Spark for Heavy Workloads
10. Untangling Apache Hadoop YARN, Part 4: Fair Scheduler Queue Basics (Cloudera)
11. Untangling Apache Hadoop YARN, Part 5: Using FairScheduler queue properties (Cloudera)
12. Untangling Apache Hadoop YARN, Part 3: Scheduler Concepts (Cloudera)
13. Tuning Spark (Apache Spark 2.1.0)

G1 GC: Humongous Objects and Humongous Allocations

For Garbage First Garbage Collector (G1 GC), any object that is more than half a region size is considered a Humongous Object.   A humongous object is directly allocated into Humongous Regions.^[1]

In this article, we will discuss the following topics:

• Humongous regions and humongous allocations
• How humongous allocations impact G1's performance?
• How to detect humongous allocations?
• Basic Investigation
• Advanced Investigation

Humongous Regions and Humongous Allocations

A humongous object is allocated directly in the humongous regions. These humongous regions are a contiguous set of regions. StartsHumongous marks the start of the contiguous set and ContinuesHumongous marks the continuation of the set.

Before allocating any humongous region, the marking threshold is checked, initiating a concurrent cycle, if necessary. Dead humongous objects are freed at the end of the marking cycle during the cleanup phase also during a full garbage collection cycle (but, a new implementation has changed this; see the next section).

Since each individual set of StartsHumongous and ContinuesHumongous regions contains just one humongous object, the space between the end of the humongous object and the end of the last region spanned by the object is unused. For objects that are just slightly larger than a multiple of the heap region size, this unused space can cause the heap to become fragmented.

How Humongous Allocations Impact G1's Performance?

In the old implementation, humongous objects are not released until after a full concurrent marking cycle.^[4]  This is far from ideal for many transaction-based enterprise applications that create short-lived (i.e., in the transaction scope) humongous objects such as ResultSet(s) generated from JDBC queries.  This results in the heap filling up relatively quickly and leads to unnecessary marking cycles to reclaim them.

A new implementation since:

java version "1.8.0_40-ea"
Java(TM) SE Runtime Environment (build 1.8.0_40-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b05, mixed mode)

handles humongous regions differently and can reclaim them earlier if they are short-lived (see [4] for details).

For investigating humongous allocations in G1, you should begin with the following minimal set of VM options:

• -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintReferenceGC (-XX:+UseG1GC)

Basic Humongous Allocation Investigation

For basic humongous allocations investigation, you simply add a new option:

• -XX:+PrintAdaptiveSizePolicy

This option will print out something like below:

12025.832: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 3660578816 bytes, allocation request: 1048592 bytes, threshold: 3650722120 bytes (85.00 %), source: concurrent humongous allocation]

From the above, we saw a humongous allocation is initiated with a request size of 1048592 bytes. Note that our region size is 1MBytes in this test. So, the request object is considered as humongous (i.e., 1048592 Bytes > half of 1 MBytes).

In the output, you can also find the following entries:

      [Humongous Reclaim: 0.0 ms]
         [Humongous Total: 54]
         [Humongous Candidate: 0]
         [Humongous Reclaimed: 0]

This shows that there are 54 humongous regions found in this GC event.   Also, none of them were reclaimed in this GC event. The reason is that the current algorithm is relatively conservative and it skips humongous objects for reclamation if there are sparse references into them.  In the case that those references belongs to sparse table entries, G1 may improve the reclamation rate by discovering if they are actually not referenced at all after iterating the table.  A performance enhancement is pending for this effort now.^[5]

Advanced Humongous Allocation Investigation

To further investigate humongous allocation in more details, you can add:

-XX:+PrintAdaptiveSizePolicy -XX:+UnlockExperimentalVMOptions -XX:+G1ReclaimDeadHumongousObjectsAtYoungGC -XX:G1LogLevel=finest -XX:+G1TraceReclaimDeadHumongousObjectsAtYoungGC

Note that the following VM options are experimental:

• G1TraceReclaimDeadHumongousObjectsAtYoungGC^[4]
• G1ReclaimDeadHumongousObjectsAtYoungGC^[3]
• G1LogLevel (i.e., [fine|finer|finest])

So, that's why we added the unlock option;

-XX:+UnlockExperimentalVMOptions

With extra printing options, you will find detailed description of humongous allocations as shown below:

3708.669: [SoftReference, 0 refs, 0.0000340 secs]3708.669: [WeakReference, 1023 refs, 0.0001790 secs]3708.669: [FinalReference, 295 refs, 0.0007020 secs]3708.670: [PhantomReference, 0 refs, 0.0000060 secs]3708.670: [JNI Weak Reference, 0.0000140 secs]Live humongous 1 region 9 size 65538 with remset 1 code roots 0 is marked 0 live-other 0 obj array 0

Live humongous 1 region 10 size 88066 with remset 1 code roots 0 is marked 0 live-other 0 obj array 0
Live humongous 1 region 15 size 262146 with remset 1 code roots 0 is marked 0 live-other 0 obj array 1
...

Finally, you can also use Java Flight Recorder (JFR) and Java Mission Control (JMC) to get information about humongous objects by showing their allocation sources with stack traces.^[7]

References

1. Garbage First Garbage Collector Tuning
2. G1GC: Migration to, Expectations and Advanced Tuning
3. Thread local allocation buffers (TLAB's)
4. Early reclamation of large objects in G1
• Introduced since 8u40 b02 and 8u45
• Was not back ported to 8u20
5. Early reclaim of large objects that are referenced by a few objects
6. G1TraceReclaimDeadHumongousObjectsAtYoungGC
• JDK-8058801
• Prints information about live and dead humongous objects
• (Before bug fix) prints that information when there is at least one humongous candidate.
• (After bug fix) prints that information even when there is no humongous candidates; it also prints humongous object size in the output
7. Java Mission Control (JMC) and Java Flight Recorder (JFR)
8. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)
9. Other JDK 8 articles on Xml and More
10. Concurrent Marking in G1

JDK 8: Is Tuning MaxNewSize in G1 GC a Good Idea?

For a throughput GC such as Parallel GC, it often helps performance by setting MaxNewSize (or -Xmn).  For G1 GC,^[1] is it the same? The short answer is yes/no and it depends (confession: I said "no" because I haven't done enough tests to exclude it).

In this article, we will demonstrate one case that setting MaxNewSize for G1 actually hurts the application's performance.  BTW, after benchmarking G1 in JDK 7, another author has claimed that:^[14]

• There is little to gain and much to lose from setting the new generation size explicitly.

Live Data Size

To tune the heap size of a Java Application, it's important to find out what its live data size is.  To learn how to estimate a Java application's live data size, read [2].  For the benchmark we are using, its live data size is around 1400 MB. With this benchmark, we are interested in knowing how G1 performs with MaxNewSize set to 400MB and Java heap set to 2GB.

Regularly we use the following settings for the G1 tuning:

• -Xms2g -Xmx2g -XX:+UseG1GC

plus some options for mixed GC (read [3]). So, for this experiment, we have added MaxNewSize setting like this:

• -Xms2g -Xmx2g -XX:+UseG1GC -Xmn400m

This has turned out hurting the performance a lot. Note that -Xmn400m is the optimal setting of our benchmark if Parallel GC is used.

Concurrent Cycle in G1 GC

G1 has four main operations:^[5]

• A young collection
• A background, concurrent cycle
• A mixed collection
• If necessary, a full GC

Without much ado, read [5-9] for the needed background of G1 GC.

G1 is a concurrent collector.  Its concurrent cycle has multiple phases—some of which stop all application threads (denoted by STW) and some of which do not:

• Initial-mark (STW)
• Denoted by: 
• [GC pause (G1 Evacuation Pause) (young) (initial-mark)
• [GC pause (Metadata GC Threshold) (young) (initial-mark)
• The G1 GC marks the roots during this phase. This phase is piggybacked on a normal (STW) young garbage collection.
• Root Region Scan
• Denoted by: 
• [GC concurrent-root-region-scan-start]
• [GC concurrent-root-region-scan-end
• The G1 GC scans survivor regions of the initial mark for references to the old generation and marks the referenced objects. 
• This phase runs concurrently with the application (not STW) and must complete before the next STW young garbage collection can start.
• Concurrent marking
• Denoted by:
• [GC concurrent-mark-start]
• [GC concurrent-mark-end
• The G1 GC finds reachable (live) objects across the entire heap. 
• This phase happens concurrently with the application, and can be interrupted by STW young garbage collections.
• Remark (STW)
• Denoted by:
• [GC remark
• This phase is STW collection and helps the completion of the marking cycle. 
• Finds objects that were missed by the concurrent mark phase due to updates by Java application threads to objects after the concurrent collector had finished tracing that object.
• G1 GC drains SATB buffers, traces unvisited live objects, and performs reference processing.
• Cleanup (STW)
• Denoted by:
• [GC cleanup 1936M->1931M(2048M)
• Prepare for next concurrent collection by clearing data structures.
• In this final phase, the G1 GC performs the STW operations of accounting and RSet scrubbing. 
• During accounting, the G1 GC identifies completely free regions and mixed garbage collection candidates. The cleanup phase is partly concurrent when it resets and returns the empty regions to the free list.
• Concurrent cleanup:
• Denoted by:
• [GC concurrent-cleanup-start]
• [GC concurrent-cleanup-end
• In this phase, G1 reclaims regions which were found empty during marking.
• Adds empty regions to the free list—i.e., thread local free lists are merged into global free list

G1 GC uses the Snapshot-At-The-Beginning (SATB) algorithm, which takes a snapshot of the set of live objects in the heap at the start of a marking cycle.  During the concurrent cycle, young garbage collections are allowed, which are triggered when eden fills up (note that initial-mark is implemented using a young collection cycle) .

The set of live objects after marking cycle is composed of the live objects in the snapshot, and the objects allocated since the start of the marking cycle. The G1 GC marking algorithm uses a pre-write barrier to record and mark objects that are part of the logical snapshot.

After the concurrent cycle, we expect to see:^[5]

• The eden regions before the marking cycle have been completely freed and new eden regions have been  allocated
• Old regions could be more occupied because the promotion of live objects from young regions
• Some old regions are identified to be mostly garbage and become candidates in later mixed or old collection cycles

Diagnosis

The reason of setting -Xmn400m has caused the regression of G1 is:

The option has forced G1 to use a young generation space of up to 400 MB and leaves G1 just 200 MB (i.e. 2048MB - 400MB - 1400MB) breathing room for shuffling live objects around.    

The small free space has negative impact on G1's concurrent cycles, which has caused most marking cycles not being completed in time.  

Before setting -Xmn400m on the command line, we have found only one instance of aborted marking cycle, which is denoted by:

• [GC concurrent-mark-abort]

After setting it, we have seen marking cycle was aborted 105 times out of 150 instances.  The lesson we have learnt here is that for a tight heap like our benchmark, it's not a good idea to set -Xmn400m for G1.  Without setting MaxNewSize, G1's ergonomics will dynamically adjust young generation space and it turns out that G1 can do a better job in this case.

Differences between Parallel GC and G1 GC

Note that -Xmn400m is the optimal setting for our benchmark if Parallel GC is used.  When we set MaxNewSize to be 400MB for G1, our benchmark regressed.  So, what's the difference between G1 GC and Parallel GC?

Initially, G1 GC was designed to replace CMS ^[10] for its relatively lower and more predictable pause times.  This is different from the design of Parallel GC which aims for higher throughput.  To help gain higher throughput, Parallel GC tries to:

• adjust young generation as large as possible and let it run into a full GC
• In our 4-hour experiments, we usually see around 50 full GC's in Parallel GC
• However, for G1's performance tuning, we want to avoid full GC's (see [11] for how)
• Which means more frequent full GC is expected
• Note that Parallel GC's Full GC pause time is shorter than G1's in the current JDK 8 releases

To conclude, if you upgrade JDK or switch garbage collector,^[7] always re-evaluate original optimal VM settings.

Acknowledgement

Some writings here are based on the feedback from Thomas Schatzl. However, the author would assume the full responsibility for the content himself.

References

1. Garbage First Garbage Collector Tuning
2. JRockit: How to Estimate the Size of Live Data Set
3. g1gc logs - basic - how to print and how to understand
4. g1gc logs - Ergonomics -how to print and how to understand
5. Java Performance: The Definitive Guide (Strongly recommended)
6. Garbage First Garbage Collector Tuning - Oracle
7. Our Collectors by Jon Masamitsu
8. Understanding Garbage Collection
9. HotSpot VM Performance Tuning Tips
10. Understanding CMS GC Logs
11. G1 GC: Tuning Mixed Garbage Collections in JDK 8
12. G1 GC Glossary of Terms
13. Learn More About Performance Improvements in JDK 8 
14. Benchmarking G1 and other Java 7 Garbage Collectors
15. HotSpot Virtual Machine Garbage Collection Tuning Guide
16. Getting Started with the G1 Garbage Collector
17. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)
18. Other JDK 8 articles on Xml and More
19. Tuning that was great in old JRockit versions might not be so good anymore
    
    • Trying to bring over each and every tuning option from a JR configuration to an HS one is probably a bad idea.
    • Even when moving between major versions of the same JVM, we usually recommend going back to the default (just pick a collector and heap size) and then redoing any tuning work from scratch (if even necessary).

G1 GC: Tuning Mixed Garbage Collections in JDK 8

To tune G1 GC (Garbage First Garbage Collector),  a place to start with is [1]. For the latest information, read [2].

Assuming you have minimal knowledge of G1 GC, the focus of this article is to tune the Mixed GC for better performance.

The Need to Tune Mixed GC

G1 GC is a generational garbage collector—read [3] to learn what eden, survivor, and old generation spaces are.  It also uses region-based architecture which divides large contiguous Java heap space into multiple fixed-sized heap regions.

The destination region for a particular object depends upon the object's age; an object that has aged sufficiently is evacuated to an old generation region (or promoted); otherwise, the object is evacuated to a survivor region and will be included in the CSet^[5] of the next young or mixed garbage collection.

During young collections, G1 GC adjusts its young generation (eden and survivor sizes) to meet its pause target. During mixed collections, the G1 GC adjusts the number of old gen regions that are collected based on a target number of mixed garbage collections, the percentage of live objects in each region of the heap, and the overall acceptable heap waste percentage (see details later).

Depending on your application's workload, Full GC could be expensive in G1 GC.  Since Mixed GC collects both young and old regions, you can get better performance by tuning Mixed GC—the goal is to reduce the number of Full GC's when your applications run.

More on Mixed Garbage Collections

Upon successful completion of a concurrent marking cycle, the G1 GC switches from performing young garbage collections to performing mixed garbage collections. In a mixed garbage collection, the G1 GC optionally adds some old regions to the set of eden and survivor regions that will be collected. The exact number of old regions added is controlled by a number of flags that will be discussed later. After the G1 GC collects a sufficient number of old regions (over multiple mixed garbage collections), G1 reverts to performing young garbage collections until the next marking cycle completes.

To summarize, Mixed GC can be characterized by:

• Collecting both young and old regions
• Denoted by: [GC pause (G1 Evacuation Pause) (mixed) in the GC log file^[2]
• Only after a completed marking cycle
• A sequence of mixed GC events to collect old regions, up to 8 by default

Tuning Mixed Garbage Collections

You can play with the following options to tune Mixed GC:^[1,2,4]

• -XX:InitiatingHeapOccupancyPercent 
• Percentage of the (entire) heap occupancy to start a concurrent GC cycle. It is used by GCs that trigger a concurrent GC cycle based on the occupancy of the entire heap, not just one of the generations.  The default value is 45.
• This option can be used to change the marking threshold
• If threshold is exceeded, a concurrent marking will be initiated next.
• The higher the threshold is, the less concurrent marking cycles will be, which also means the less mixed GC evacuation will be.
• -XX:G1MixedGCLiveThresholdPercent 
• This option can be used to change the threshold which determines whether a region should be added to the CSet or not.
• Only regions whose live data percentage are less than the threshold will be added to the CSet.
• The higher the threshold (default: 65) is, the more likely a region will be added to the CSet, which also means more mixed GC evacuation and longer evacuation time will happen. 
• -XX:G1HeapWastePercent
• Amount of reclaimable space, expressed as a percentage of the heap size that G1 will stop doing mixed GC's. If the amount of space that can be reclaimed from old generation regions compared to the total heap is less than this, G1 will stop mixed GC's.
• Current default is 10%.  A lower value, say 5%, will potentially cause G1 to add more expensive region(s) to evacuate for space reclamation.
• G1 will continue triggering mixed GC if the reclaimable is higher than the waste threshold.  So, there will be more mixed GC and maybe more expensive if you set this waste threshold lower.

Besides the above-mentioned options, you may also want to tune the following ones:

• -XX:G1MixedGCCountTarget 
• Sets the target number of mixed garbage collections after a marking cycle to collect old regions with at most G1MixedGCLIveThresholdPercent live data. 
• The default is 8 mixed garbage collections. The goal for mixed collections is to be within this target number.
• -XX:G1OldCSetRegionThresholdPercent
• Sets an upper limit on the number of old regions to be collected during a mixed garbage collection cycle. The default is 10 percent of the Java heap.
• -XX:G1HeapRegionSize
• Sets the size of a G1 region. The value will be a power of two and can range from 1MB to 32MB. 
• The goal is to have around 2048 regions based on the minimum Java heap size.

Conclusion

Each application is unique, you may need to tune G1 GC in an iterative process.  Note that all of the above options are product options except G1MixedGCLiveThresholdPercent and G1OldCSetRegionThresholdPercent, which are experimental options.  Be warned that: for the experimental options, Oracle may remove them at its discretion in the future releases.  

Acknowledgement

Some writings here are based on the feedback from Thomas Schatzl and Yu Zhang. However, the author would assume the full responsibility for the content himself.

References

1. Garbage First Garbage Collector Tuning
2. g1gc logs - Ergonomics -how to print and how to understand 
3. Understanding Garbage Collection  
4. Garbage First (G1) Garbage Collection Options
5. CSet
• The G1 GC reduces heap fragmentation by incremental parallel copying of live objects from one or more sets of regions (called Collection Set (CSet)) into different new region(s) to achieve compaction.
• The goal of G1 GC is to reclaim as much heap space as possible, starting with those regions that contain the most reclaimable space (i..e, garbage first), while attempting to not exceed the pause time goal.
6. G1 GC Glossary of Terms
7. Learn More About Performance Improvements in JDK 8 
8. HotSpot Virtual Machine Garbage Collection Tuning Guide
9. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)
10. G1 GC: Humongous Objects and Humongous Allocations (Xml and More)
11. Other JDK 8 articles on Xml and More
12. Concurrent Marking in G1

JDK 8: UseCompressedClassPointers vs. UseCompressedOops

A new JVM option was introduced into JDK 8 after PermGen removal:

UseCompressedClassPointers

In this article, we will discuss the difference between  UseCompressedOops and UseCompressedClassPointers.

Default Values

As described in [2], you can find out the default values of UseCompressedClassPointers and UseCompressedOops:

     bool UseCompressedClassPointers           := true   {lp64_product}
     bool UseCompressedOops                    := true   {lp64_product}

Our platform is linux-x64 and these options are both set to be true based on ergonomics.

UseCompressedOops vs. UseCompressedClassPointers

CompressedOops are for the compression of pointers to objects in the Java Heap.  Class data is no
longer in the Java Heap and the compression of pointers to class data is done under the flag
UseCompressedClassPointers.  In next sections, we will discuss them in more details.

Oops and Compressed Oops

Oops are "ordinary" object pointers.  Specifically, a pointer into the GC-managed heap. Implemented as a native machine address, not a handle. Oops may be directly manipulated by compiled or interpreted Java code, because the GC knows about the liveness and location of Oops within such code.  Oops can also be directly manipulated by short spans of C/C++ code, but must be kept by such code within handles across every safepoint.

Compressed Oops represent managed pointers (in many but not all places in the JVM) as 32-bit values which must be scaled by a factor of 8 and added to a 64-bit base address to find the object they refer to in Java Heap.

Compressed Class Pointers

Objects (in its 2nd word) have a pointer  to VM Metadata class, which can be compressed.  If compressed, it uses a base which points to the Compressed Class Pointer Space.

Before we continue, you need to know what Metaspace and Compressed Class Pointer Space  are. A Compressed Class Pointer Space (which is logically part of Metaspace) is introduced for 64 bit platforms. Whereas the Compressed Class Pointer Space contains only class metadata, the Metaspace can contain all other large class metadata including methods, bytecode etc.

For 64 bit platforms, the default behavior is using compressed (32 bit) object pointers (-XX:+UseCompressedOops) and compressed (32 bit) class pointers (-XX:+UseCompressedClassPointers).  However, you can modify default settings if you like.  When you do  modify them, be warned that there is a dependency between the two options—i.e., UseCompressedOops must be on for UseCompressedClassPointers to be on.

To summarize it, the differences between Metaspace and Compressed Class Pointer Space are :[3]

• Compressed Class Pointer Space contains only class metadata
• InstanceKlass, ArrayKlass
• Only when UseCompressedClassPointers true
• These include Java virtual tables for performance reasons
• Metaspace contains all other class metadata that can be large.
• Methods, Bytecodes, ConstantPool ...

 

References

1. HotSpot: Monitoring and Tuning Metaspace in JDK 8 (Xml and More)
2. What Are the Default HotSpot JVM Values?
3. Metaspace in Java 8 (good)  
4. HotSpot Glossary of Terms 
5. VM Class Loading
6. Learn More About Performance Improvements in JDK 8 
7. Garbage-First Garbage Collector Tuning
8. Understanding Compressed References (JRockit)
• Similar to HotSpot's CompressedOops, Jrockit uses -XXcompressedRefs command option.

HotSpot: Sizing System Dictionary in JDK 8

The system dictionary is an internal JVM data structure which holds all the classes loaded by the system.  As described in JDK-7114376,^[1]

The System Dictionary hashtable bucket array size is fixed at 1009. This value is too small for large programs with many names, too large for small programs and just about right for medium sized ones. The default should remain at 1009, but it should be possible to override it on the command line.

Finally, this has happened in JDK 8 and you can set hashtable bucket array size to be larger if you have more classes loaded in your applications by using:^[2]

• -XX:+UnlockExperimentalVMOptions  -XX:PredictedLoadedClassCount=<#>

This can help calls like Class.forName(), which do lookups into this data structure.

In this article, we will look into sizing system dictionary in more details.

Performance of System Dictionary

In 1.4.2, there were some performance enhancements.  One of them is making system dictionary reads lock-free.^[3] From this enhancement, you probably can guess that tuning system dictionary could be important to the JVM's performance.

The current number of buckets in hash table for system dictionary is set to be 1009 by default, which is relatively small for an application which has 49987 classes loaded as shown below:

Loaded   Bytes Unloaded   Bytes       Time
49987 110488.8     2453  7743.8     105.24

From the experience, we know:^[5]

In a good hash table, each bucket has zero or one entries, and sometimes two or three, but rarely more than that.

 Assuming the average ideal length of buckets is three, the hashtable bucket array size then should be:

Ideal hashtable bucket array size = 49987 / 3 = 16662

 

System Dictionary Tuning

Seeing 49987classes loaded in our application at end of its run, we decided to set:

-XX:PredictedLoadedClassCount=16661 (a prime)

Note that PredictedLoadedClassCount is an experimental flag.  So, you also need to set:

-XX:+UnlockExperimentalVMOptions

before it in the command line.

When the JVM allocates memory, the largest chunk is the java heap. The system dictionary is in C heap and it is the loaded class cache. The goal of setting PredictedLoadedClassCount flag is to increase the size of the system dictionary in order to make lookups of loaded classes faster. Before every class being loaded, it requires a checking to see if the class is already loaded.  Larger system dictionary will improve JVM performance during this class loading and resolution phase.

Note that the total number of entries in the hashtable does not change— that is based on the number of loaded classes. What sizing of system dictionary do is to increase the spread of the entries so that the average length of the buckets under a single hash result could be reduced. This would reduce the time it takes to find a given entry.

How to Find Number of Loaded Classes?

There are multiple ways to find out number of loaded classes in an application.  For example, you can use -Xverbose:class to determine what classes are loaded.   Another way to find out is what we have done in this article by setting -XX:+PerfDataSaveToFile and then use "jstat -class" command to decipher the class statistics offline:

$<snipped>/jdk-hs/bin/jstat -class file:////slot/myserver/appmgr/APPTOP/instance/domains/slcaf977.us.oracle.com/CRMDomain/hsperfdata_9872

Loaded   Bytes Unloaded   Bytes       Time
49987 110488.8     2453  7743.8     105.24

Acknowledgement

Some writings here are based on the feedback from Mikael Gerdin and Karen Kinnear. However, the author would assume the full responsibility for the content himself.

References

1. Make system dictionary hashtable bucket array size configurable  (JDK-7114376)
2. Tuning the JVM (at 43:02 mark)
3. Java 2 Platform, Standard Edition (J2SE Platform), version 1.4.2 Performance White Paper 
4. The Class Loader Subsystem 
5. Hash table (Wikipedia) 
6. VM Class Loading
• Actually, the HotSpot VM maintains three main hash tables to track class loading
• SystemDictionary  
• PlaceholderTable 
• LoaderConstraintTable
7. Garbage-First Garbage Collector Tuning

HotSpot: A Case Study of MetaspaceSize Tuning in JDK 8

This article is one of the Metaspace series in JDK 8 on Xml and More.  Please read previous articles (i.e. [1, 2, 3]) for the background information.

Here we use a case study to show you what we mean by:

Proper monitoring and tuning of the Metaspace is still required in order to limit the frequency or delay the garbage collections of metaspace.

 

JVM Setup

We have used the following print options:

•  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

to generate GC log files.  Besides that, we have set:

•  -XX:MetaspaceSize=128m

to specify the initial high water mark to trigger metaspace cleanup by inducing a GC event.

In the GC log file, you can find the following entries:

[Metaspace: 21013K->21013K(1069056K)]

The first value shows you the previously used size before GC, the second value shows you the current used size and the last value shows you the current reserved size.

Before Tuning

Below is the experiment running with all default values.  As you can see, there are six Full GC events triggered by "Metadata GC Threshold" being reached.  The initial high water mark was set by the default MetaspaceSize (i.e.,  21807104 Bytes).  So, the first Full GC event was induced when committed memory of all metaspaces reaches the initial high water mark.  To find HotSpot's default option values, read [4].

3.112: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 21013K->21013K(1069056K)]
5.440: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 34645K->34645K(1081344K)]
11.921: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 58159K->58152K(1101824K)]
18.321: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 97038K->97038K(1136640K)]
51.761: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 160475K->155432K(1193984K)]
319.406: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 267854K->267854K(1288192K)]

Note that the unit on X-axis is seconds and Y-axis KBytes.

After Tuning

Here we have set the experiment with the following extra option:

• -XX:MetaspaceSize=128m

Note that 128M is bigger than default value (i.e., 21807104 Bytes).  Because of the setting, we have reduced the frequency and delayed garbage collections due to "Metadata GC Threshold" being reached.  

25.863: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 127978K->126460K(1165312K)]
81.254: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 213481K->209918K(1241088K)]
3486.808: [Full GC (Allocation Failure) ...]
   [Eden: ..., [Metaspace: 306929K->298136K(1327104K)]
3631.001: [Full GC (Allocation Failure) ...]
   [Eden: ..., [Metaspace: 299979K->298792K(1329152K)]

Why to Tune?

So, you may ask what's the deal with tuning MetaspaceSize.  Yes, it's just setting the initial high water mark.  Later HotSpot will adjust high water mark based on ergonomics.  Hopefully, when HotSpot's ergonomics becomes more matured, no tuning is necessary at all.  But, even in that case, you may still want to set MetaspaceSize for some occasions.  One of them is when you run a benchmark in a short period and your focus may be on other GC activities than meta data being loaded or unloaded.

References

1. HotSpot: Understanding Metaspace in JDK 8
2. HotSpot: Monitoring and Tuning Metaspace in JDK 8
3. jstat Tool: New Metaspace Statistics from -gc Option
4. What Are the Default HotSpot JVM Values?
5. VM Class Loading 

HotSpot: What Are Those GC Worker Threads in G1 GC?

On the Internet, you can find some tuning guides for Garbage First Garbage Collector (G1 GC)^[1,5,8,9] and some of them may be outdated.

In this article, we try to update some of those information with the focus on thread tuning.

Ergonomics

Ergonomics for servers was first introduced in Java SE 5.0^[2]. It has greatly reduced application tuning time for server applications, particularly with heap sizing and advanced GC tuning. In many cases no tuning options when running on a server is the best tuning you can do.

Here we will focus on the number of GC worker threads chosen to run on a specific server based on the ergonomics in G1 GC.

Garbage Collector	Worker Threads Used
Parallel	ParallelGCThreads
CMS	ParallelGCThreads ConcGCThreads
G1	ParallelGCThreads ConcGCThreads G1ConcRefinementThreads

G1 Threads

In the table below, we have summarized the most important GC worker threads in Oracle's G1 GC implementation. For comparison, Parallel GC only uses ParallelGCThreads and CMS only uses ParallelGCThreads and ConcGCThreads. Be warned that these values can be changed in the future if Oracle sees appropriate.

GC thread counts are based on the number of processors (including virtual processor if hyper-threading^[3] is enabled) reported by the system. To see the number of processors you have in the system, check out the file /proc/cpuinfo. In this article, 32 processors are used in the demonstration.

Name	Controlled by	Default if Not Specified	Description
Parallel GC Threads	-XX:ParallelGCThreads	if #proc <=8, then #proc; if #proc >8, 8+(#proc-8)*(5/8)	Parallel operations
Parallel Marking Threads	-XX:ConcGCThreads	There are lot of cases, but mostly it uses max((ParallelGCThreads+2)/4, 1)
G1 Main Concurrent Mark GC Thread		1	Master concurrent thread
G1 Concurrent Refinement Thread	-XX:G1ConcRefinementThreads	ParallelGCThreads+1	The +1 is the master thread which control these worker threads.

What Are They?

G1 garbage collection cycle has multiple phases: it starts with an initial marking phase and ends with a concurrent-cleanup phase. During the cycle, there are some phases that pause all the application threads, and some that run concurrently. These GC worker threads used in G1 GC include, but not limited to:

• ParallelGCThreads
• ParallelGCThreads are the threads employed during GC pauses
• They are responsible for copying (and scanning for) live objects during stop-the-world (or STW) pauses.
• They are also employed during other STW pauses (such as Remark and Cleanup).
• ConcGCThreads
• In concurrent marking, there is one controller thread and another bunch of worker threads. The number of worker threads is set using ConcGCThreads or a fraction of ParallelGCThreads if ConcGCThreads is not set.
• Worker threads are the threads used for marking and determining the liveness of regions, which are run concurrently to the application threads.
• Increasing ConcGCThreads will make the marking cycle complete faster.
• G1ConcRefinementThreads
• G1ConcRefinementThreads are the threads used for updating remembered set while application threads are running concurrently.
• They are responsible for taking the buffers that are used to log object updates and updating the RememberedSets (or RSets)^[10] based upon the updates.
• If G1ConcRefinementThreads is not set, then its value is ParallelGCThreads + 1.
• For example, if ParallelGCThreads is 23 then we will have 24 refinement threads.
• One is used for sampling and updating the RSets of young regions (typically young regions have less cross-region references so one thread is enough); the other 23 threads fill update buffers.
• Typically they don't all run at once.
• They each have a stepped activation threshold and a thread is activated when the number of completed buffers exceeds it's activation threshold.
• They also have deactivation thresholds and when the number of filled update buffers falls below a thread's deactivation threshold, it is deactivated.

Thread Tuning

For our server with 32 processors, the number of GC worker threads chosen by G1's ergonomics are:

• # ParallelGCThreads = 23
• # ConcGCThreads = 6
• # G1ConcRefinementThreads = 23

To find out what the GC worker thread counts are for your server, you can use -XX:+PrintFlagsFinal.^[4] You can also change the default values if you see appropriate.^[6,7] For some systems, default values sometimes can be too high.

References

1. Garbage First Garbage Collector Tuning (Published August 2013)
2. Java Heap Sizing: How do I size my Java heap correctly?
3. Intel® Hyper-Threading Technology
4. What Are the Default HotSpot JVM Values?
5. Part #1 - Tuning Java Garbage Collection for HBase
6. Linux: Understanding Processor Queue in Vmstat Output
7. How to Troubleshoot High CPU Usage of Java Applications? 
8. Getting Started with the G1 Garbage Collector 
9. G1: One Garbage Collector To Rule Them All 
10. Remembered Sets (RSets) in G1 GC
• G1 GC uses independent Remembered Sets (RSets) to track references into regions. Independent RSets enable parallel and independent collection of regions because only a region's RSet must be scanned for references into that region, instead of the whole heap. To achieve the task, G1 GC uses a post-write barrier to record changes to the heap and update the RSets. 
11. D. L. Detlefs, C. H. Flood, S. Heller, and T. Printezis. Garbage-First Garbage Collection. In A. Diwan, editor, Proceedings of the 2004 International Symposium on Memory Management (ISMM 2004), pages 37-48, Vancouver, Canada. October 2004. ACM Press.
12. T. Printezis and D. L. Detlefs. A Generational Mostly-Concurrent Garbage Collector. In A. L. Hosking, editor, Proceedings of the 2000 International Symposium on Memory Management (ISMM 2000), pages 134-154,Minneapolis, MN, USA, October 2000. ACM Press. 
13. g1gc logs - how to print and how to understand - basic 
14. G1 GC Glossary of Terms
15. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)

Hotspot: Creating Flight Recording and Viewing Object Allocation

In [1], Marcus has written an excellent article on "Creating Flight Recording." Based on his advice, we have created a JFR recording with object allocations inside/outside TLAB (Thread Local Allocation Buffer) enabled. Then, we installed Java Mission Control (an Eclipse plug-in) on our 64-bit Windows platform and used it to investigate object allocations in the recording.

In this article, we will visit the following topics:

• Generating JFR recording
• Downloading and installing Java Mission Control
• Note that Java Mission Control (JMC) is for Hotspot as JRockit Mission Control (JRMC) is for JRockit.
• Viewing object allocations using Java Mission Control

Generating JFR Recording

Following Marcus' Time Fixed Recording example, we have used the following command line options:

• -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:StartFlightRecording=delay=7800s,duration=300s,name=MyRecording,filename=/tmp/myrecording.jfr,settings=default

In our example, we started the recording for 5 minutes after Hotspot has run for 7800 seconds. To configure the recording, you can use either template provided in jre/lib/jfr folder:

• default.jfc
• profile.jfc

For our example, we have chosen the default template (note that the settings parameter can also take a path to a template) in which we have enabled the following settings:

<flag name="allocation-profiling-enabled" label="Allocation Profiling">true</flag>

<event path="java/object_alloc_in_new_TLAB">
  <setting name="enabled" control="allocation-profiling-enabled">true</setting>
  <setting name="stackTrace">true</setting>
</event>

<event path="java/object_alloc_outside_TLAB">
  <setting name="enabled" control="allocation-profiling-enabled">true</setting>
  <setting name="stackTrace">true</setting>
</event>

Note that we have saved an original copy of default.jfc before making the changes. After running our benchmark, it has generated a recording file named:

• myrecording.jfr

Downloading and Installing Java Mission Control

Capability	Oracle JRockit JDK6 (R28+)	Oracle JDK 7 GA	Oracle JDK 7u40+
Host JRMC/JMC GUI	Yes (JRMC)	Yes (JMC)	Yes (JMC)
WLDF JFR Events and Analysis	Yes	Yes	Yes
JFR, JMC Convergence (JVM Events)	Yes	No	Yes

                 Table 1  Oracle JDK 7 Java Mission Control Support

You can follow the instructions from [2] to download Java Mission Control—an Eclipse plug-in. However, instead of using https, you should use http as shown above. Also, be noted that

Java Mission Control 5.2.0 does not support Kepler, only Juno.^[3,4]

So, you need to download Eclipse Juno from here. If you installed JMC in Eclipse Kepler, you would see the problem as reported in [5].

Viewing object allocations using Java Mission Control

After opening myrecording.jfr, select Memory tab group on the left as shown above.  The Memory tab group shows Information on memory management and garbage collections. It is comprised of these tabs:

• Overview Tab
• Garbage Collections Tab
• GC Times Tab
• GC Configuration Tab
• Allocations Tab
• Object Statistics Tab

The Allocation tab is a good starting point if you suspect that you have problems with object allocation. It contains data about allocated objects and Thread Local Area Buffers (TLABs). This tab has information about how much memory each thread has allocated.

To recap: when we did the recording, we have enabled the following events:

• java/object_alloc_in_new_TLAB
• java/object_alloc_outside_TLAB

which are displayed in the above figure:

• Allocation in new TLAB
• Allocation outside TLAB

References

1. Creating Flight Recordings
2. Oracle Java Mission Control Downloads
3. Java Mission Control for Eclipse
4. Oracle® Java Mission Control
• Oracle® Java Mission Control is a set of plug-ins for Eclipse 3.8 or 4.2.
5. Mission Control and Flight Recorder on HotSpot JVM
6. Which JVM?
• If opening JFR recordings takes forever,  you may need to increase heap size.
7. JDBC in Java Mission Control 
8. JDK Mission Control (JMC) 8.1.0 Downloads 

Linux: Understanding Processor Queue in Vmstat Output

Linux tools such as mpstat, iostat, vmstat are useful for evaluating overall system performance at the OS level.  In this article, we will look at what statistics vmstat provides, especially focusing on CPU Run Queue (or Processor Queue).

vmstat Sample Ouput

vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.  It takes an optional delay argument and other options.  For example, the following command asks vmstat make updates every 60 seconds.

$vmstat 60

The first report produced gives averages since the last reboot. Additional reports give information on a sampling period of length delay. The process and memory reports are instantaneous in either case. If no delay is specified, only one report is printed with the average values since boot.

procs -----------memory---------- -swap- ----io--- --system-- -----cpu------
 r  b  swpd   free   buff  cache   si so bi    bo   in   cs   us sy id wa st

 3  0   0 34138256 2654680 30542460 0  0 112 7052 1220 26560   11  9 79  0  0
134 0   0 34070712 2654812 30589228 0  0 19 15130 1174 2061612 79 10  9  1  0
13  0   0 34077516 2654868 30581300 0  0 32 10688 1123 1218261 68 17 15  0  0
 7  0   0 33928980 2654924 30538396 0  0 33  9762 1402 338231  41 12 46  1  0
 7  0   0 33904644 2654932 30556204 0  0 0   7094 1260 22498   12  8 80  0  0
 5  0   0 33869408 2654956 30564832 0  0 77  7298 1203 107386  15  9 76  0  0
 6  0   0 33907632 2654976 30532548 0  0 78  7220 1215 106477  14  9 76  0  0
 5  1   0 33898596 2655004 30539928 0  0 32  7172 1202 21369   10  8 81  0  0
 2  0   0 33883432 2655008 30547144 0  0 19  7158 1208 30007   10  8 81  0  0

As highlighted above, we have a high value (i.e., 134) of Run Queue, which will be discussed further later.

vmstat (Virtual Memory Statistics)

The summary information output by vmstat includes:

• r
• CPU Run Queue which is the actual number of lightweight processes in the run queue (including processes waiting to run but that are held up by the CPU)
• b
• Number of processes sleeping (usually waiting for IO)
• swpd
• Total swap space used (default: KB)
• free
• Total free memory (default: KB)
• buff
• Total buffer memory usage (default: KB)—represents how much portion of RAM is dedicated to cache disk block
• cache
• Total disk cache memory usage  (default: KB)—similar to buff, only this time it caches pages from file reading.
• si
• Memory swapped in from disk (in KB/sec)—the amount of memory paged-in 
• If you see high si/so values, it indicates the system is swapping 
• so
• Memory swapped out to disk (in KB/sec)—the amount of memory paged-out
• If you see high si/so values, it indicates the system is swapping 
• bi
• Blocks read in from IO devices (blocks per sec)
• bo
• Blocks written to IO devices (blocks per sec)
• in
• Interrupts per second
• cs
• Context switch per second
• us
  • Percentage of user CPU utilization
• sy
  • Percentage of kernel or system CPU utilization
  • High levels of system time mean something is wrong, or the application is making many system calls. Investigating the cause of high system time is  always worthwhile.
• id
• Percentage of idle or available CPU
• The sum of the “us” column and “sy” column should be equal to 100 minus the value in the “id” column, that is, 100 – (“id” column value)
• wa
• Time spent waiting for IO—Prior to Linux 2.5.41, included in idle
• st
• Time stolen from a virtual machine—Prior to Linux 2.6.11, unknown

CPU Run Queue (Processor Queue)

As we have seen that there was 134 threads (note that we have a 24-processor Linux system) in the Processor Queue at one time, this may be something to be alerted.  It means that there are 134 threads that were running or that could run if there were available CPU.  Keep in mind that the run queue length represents everything on the machine, so sometimes there are other threads from completely separate processes that want to run.

If there are more threads to run than available CPUs, performance will begin to degrade. In general, you want the processor queue length to be 0 on Windows and equal to (or less than) the number of CPUs on Unix systems.

If the run queue length is too long for any significant period of time, it is an indication that your machine is overloaded and you should look into reducing the amount of work the machine is doing (either by moving jobs to another machine, or optimizing your code).^[1]

Context Switches

When the machine is overloaded, you will also witness high number of context switches occurring at the same time.  From the line showing high run-queue length (i.e., 134), we also see high number of context switch per second (i.e., 2061612).

There are two types of context switches:

• Voluntary thread context switches 
• An executing thread voluntarily takes itself off the CPU
• Involuntary thread context switches 
• A thread is taken off the CPU as a result of an expiring time quantum or has been preempted by a higher priority thread

High involuntary context switches are an indication there are more threads ready to run than there are virtual processors available to run them, which results in:

• A high run queue depth in vmstat
• A high CPU utilization
• A high number of migrations

For our case, the high run queue length occurred only for a short duration.  And, it may be due to the fact that our application and database were run on the same system.  At the time of high run queue lengths, database may be performing lots of I/O's.

References

1. How to Troubleshoot High CPU Usage of Java Applications? (Xml and More)

Eclipse MAT: Querying Heap Objects Using OQL

This article is a follow-up of [1]. Here we continue to explore on how to investigate memory leaks of an application using Memory Analyzer.^[2]

Querying Heap Objects (OQL)

Memory Analyzer allows you to query the heap dump^[3] with custom SQL-like queries (OQL). OQL represents classes as tables, objects as rows, and fields as columns:^[4]

SELECT *
FROM [ INSTANCEOF ] <class name="name">
[ WHERE <filter-expression> ]
</filter-expression></class>To open an OQL editor use the toolbar button :

For instance, we have used the following SQL statement:

select * from java.lang.String where toString().startsWith("http://xmlns.oracle.com/bpel")

to query String objects with a certain prefix (i.e., "http://xmlns.oracle.com/bpel") and calculate the total size of retained heap associated with the interested objects.  Note that you need to press red "!" button to execute the OQL.

Shallow vs. Retained Heap

As shown in the Figure, two sizes of an object are displayed in the Result area:

• Shallow heap
• Retained heap

Generally speaking, shallow heap of an object is its size in the heap and retained size of the same object is the amount of heap memory that will be freed when the object is garbage collected. In other words, retained heap of object X is the sum of shallow sizes of all objects in the retained set of X, the set of objects which would be removed by Garbage Collector when X is garbage collected.

As said in [6], while Shallow Heap can be interesting, the more useful metric is the Retained Heap. For example, you can benchmark retained sizes of interested objects before and after your code optimizations. Below, we will show how to compute the total retained size of our interested objects.

Exporting to CSV...

Analyzed data can be exported from the heap editor by:^[5]

• Using the toolbar export menu (you can choose between export to HTML, CSV, and TXT)

Let's say we have exported it to a CSV file named RetainedHeap.txt.

Importing CSV File into Excel

You can use Java code to parse the CSV file and compute the retained heap of interested objects. An alternative way is using Excel, which is demonstrated here.

First, you open RetainedHeap.txt and specify both comma and space as the delimiters of fields.

Then, select all "Retained Heap" fields (shown in red) and compute the Sum as shown below:

Finding Responsible Objects

To investigate your potential memory leaks, it will be important to answer the following question:

Who has kept these objects alive?

To answer that, you can use Immediate Dominators (an Object X is said to dominate an Object Y if every path from the Root to Y must pass through X) from the context menu. This query finds and aggregates all objects dominating a given set of objects on class level. It is very useful to quickly find out who is responsible for a set of objects. Using the fact that every object has just one immediate dominator (unlike multiple incoming references) the tool offers possibility to filter "uninteresting" dominators (e.g. java.* classes) and directly see the responsible application classes.

For example, if your interested objects are char arrays. The immediate dominators of all char arrays are all objects responsible for keeping the char[] alive. The result will contain most likely java.lang.String objects. If you add the skip pattern java.* , and you will see the non-JDK classes responsible for the char arrays.

  

Bonus OQL Example

In the above OQL example, it shows that two columns were selected:

• toString(s.sqlObject.actualSql)
• s.@retainedHeapSize

from oracle.jdbc.driver.T4CPreparedStatement (alias: s).  Also, a filter was added (highlighted in red):

• .*SELECT TerritoryResource.*

References

1. Diagnosing Java.lang.OutOfMemoryError (Xml and More)
2. Memory Analyzer Downloads
• The stand-alone Memory Analyzer is based on Eclipse RCP.
• Can find the update site here too.
3. Diagnosing Heap Stress in HotSpot (Xml and More)
4. Querying Heap Objects (OQL)
5. Export Data
6. 10 Tips for using the Eclipse Memory Analyzer
7. Analyzing a Heap Dump Using Object Query Language (OQL)
8. Basic Tutorial of MAT