G1 GC: Humongous Objects and Humongous Allocations

For Garbage First Garbage Collector (G1 GC), any object that is more than half a region size is considered a Humongous Object.   A humongous object is directly allocated into Humongous Regions.^[1]

In this article, we will discuss the following topics:

• Humongous regions and humongous allocations
• How humongous allocations impact G1's performance?
• How to detect humongous allocations?
• Basic Investigation
• Advanced Investigation

Humongous Regions and Humongous Allocations

A humongous object is allocated directly in the humongous regions. These humongous regions are a contiguous set of regions. StartsHumongous marks the start of the contiguous set and ContinuesHumongous marks the continuation of the set.

Before allocating any humongous region, the marking threshold is checked, initiating a concurrent cycle, if necessary. Dead humongous objects are freed at the end of the marking cycle during the cleanup phase also during a full garbage collection cycle (but, a new implementation has changed this; see the next section).

Since each individual set of StartsHumongous and ContinuesHumongous regions contains just one humongous object, the space between the end of the humongous object and the end of the last region spanned by the object is unused. For objects that are just slightly larger than a multiple of the heap region size, this unused space can cause the heap to become fragmented.

How Humongous Allocations Impact G1's Performance?

In the old implementation, humongous objects are not released until after a full concurrent marking cycle.^[4]  This is far from ideal for many transaction-based enterprise applications that create short-lived (i.e., in the transaction scope) humongous objects such as ResultSet(s) generated from JDBC queries.  This results in the heap filling up relatively quickly and leads to unnecessary marking cycles to reclaim them.

A new implementation since:

java version "1.8.0_40-ea"
Java(TM) SE Runtime Environment (build 1.8.0_40-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b05, mixed mode)

handles humongous regions differently and can reclaim them earlier if they are short-lived (see [4] for details).

For investigating humongous allocations in G1, you should begin with the following minimal set of VM options:

• -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintReferenceGC (-XX:+UseG1GC)

Basic Humongous Allocation Investigation

For basic humongous allocations investigation, you simply add a new option:

• -XX:+PrintAdaptiveSizePolicy

This option will print out something like below:

12025.832: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 3660578816 bytes, allocation request: 1048592 bytes, threshold: 3650722120 bytes (85.00 %), source: concurrent humongous allocation]

From the above, we saw a humongous allocation is initiated with a request size of 1048592 bytes. Note that our region size is 1MBytes in this test. So, the request object is considered as humongous (i.e., 1048592 Bytes > half of 1 MBytes).

In the output, you can also find the following entries:

      [Humongous Reclaim: 0.0 ms]
         [Humongous Total: 54]
         [Humongous Candidate: 0]
         [Humongous Reclaimed: 0]

This shows that there are 54 humongous regions found in this GC event.   Also, none of them were reclaimed in this GC event. The reason is that the current algorithm is relatively conservative and it skips humongous objects for reclamation if there are sparse references into them.  In the case that those references belongs to sparse table entries, G1 may improve the reclamation rate by discovering if they are actually not referenced at all after iterating the table.  A performance enhancement is pending for this effort now.^[5]

Advanced Humongous Allocation Investigation

To further investigate humongous allocation in more details, you can add:

-XX:+PrintAdaptiveSizePolicy -XX:+UnlockExperimentalVMOptions -XX:+G1ReclaimDeadHumongousObjectsAtYoungGC -XX:G1LogLevel=finest -XX:+G1TraceReclaimDeadHumongousObjectsAtYoungGC

Note that the following VM options are experimental:

• G1TraceReclaimDeadHumongousObjectsAtYoungGC^[4]
• G1ReclaimDeadHumongousObjectsAtYoungGC^[3]
• G1LogLevel (i.e., [fine|finer|finest])

So, that's why we added the unlock option;

-XX:+UnlockExperimentalVMOptions

With extra printing options, you will find detailed description of humongous allocations as shown below:

3708.669: [SoftReference, 0 refs, 0.0000340 secs]3708.669: [WeakReference, 1023 refs, 0.0001790 secs]3708.669: [FinalReference, 295 refs, 0.0007020 secs]3708.670: [PhantomReference, 0 refs, 0.0000060 secs]3708.670: [JNI Weak Reference, 0.0000140 secs]Live humongous 1 region 9 size 65538 with remset 1 code roots 0 is marked 0 live-other 0 obj array 0

Live humongous 1 region 10 size 88066 with remset 1 code roots 0 is marked 0 live-other 0 obj array 0
Live humongous 1 region 15 size 262146 with remset 1 code roots 0 is marked 0 live-other 0 obj array 1
...

Finally, you can also use Java Flight Recorder (JFR) and Java Mission Control (JMC) to get information about humongous objects by showing their allocation sources with stack traces.^[7]

References

1. Garbage First Garbage Collector Tuning
2. G1GC: Migration to, Expectations and Advanced Tuning
3. Thread local allocation buffers (TLAB's)
4. Early reclamation of large objects in G1
• Introduced since 8u40 b02 and 8u45
• Was not back ported to 8u20
5. Early reclaim of large objects that are referenced by a few objects
6. G1TraceReclaimDeadHumongousObjectsAtYoungGC
• JDK-8058801
• Prints information about live and dead humongous objects
• (Before bug fix) prints that information when there is at least one humongous candidate.
• (After bug fix) prints that information even when there is no humongous candidates; it also prints humongous object size in the output
7. Java Mission Control (JMC) and Java Flight Recorder (JFR)
8. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)
9. Other JDK 8 articles on Xml and More
10. Concurrent Marking in G1

G1 GC: What Is "to-space exhausted" in GC Log?

As shown in [1], you can enable basic garbage collection (GC) event printing using two options:

• -XX:+PrintGCTimeStamps (or -XX:+PrintGCDateStamps) 
• -XX:+PrintGCDetails

These additional printing incurs minimal overhead while provide useful information for understanding the healthiness of your running garbage collector (e.g., Generation First GC).

In this article, we will discuss the significance of to-space exhausted events in your GC logs.

What Is "to-space exhausted"?

When you see to-space exhausted messages in GC logs, it means that G1 does not have enough memory for either survived or tenured objects, or for both, and the Java heap cannot be further expanded since it is already at its max.^[5]  This event only happens in mixed GC 's when G1 tries to copy live objects from the source space to the destination space.  Note that, in G1, it avoids heap fragmentation by compaction (i.e., done when objects are copied from source region to destination region).

Example message:

5843.494: [GC pause (G1 Evacuation Pause) (mixed)... (to-space exhausted), 0.1145790 secs]

When that happens, GC pause time will be longer (i.e., since both RSet and CSet^[5] need to be re-generated) and sometimes it may require a Full GC to resolve the issue. 

A Case Study of "to-space exhausted" Events

Using a benchmark with large live data size,^[3] a "to-space exhausted" event can be triggered if there is not enough breathing room for shuffling live data objects in the Java heap.  For example, we have a test case which its live data set occupies 68% of the Java heap at run time.  In a 4-hour run, we have seen the following sequence of events:

5843.494: [GC pause (G1 Evacuation Pause) (mixed)... (to-space exhausted), 0.1145790 secs]
5848.831: [GC pause (G1 Evacuation Pause) (mixed)... (to-space exhausted), 0.0847160 secs]
5856.640: [GC pause (G1 Evacuation Pause) (mixed)... 0.0864060 secs]
5866.182: [GC pause (G1 Evacuation Pause) (mixed)... 0.0821220 secs]
5875.550: [GC pause (G1 Evacuation Pause) (mixed)... 0.0549090 secs]
5882.353: [GC pause (G1 Evacuation Pause) (mixed)... 0.0827630 secs]
5896.331: [GC pause (G1 Evacuation Pause) (mixed)... 0.0632660 secs]
5901.966: [GC pause (G1 Evacuation Pause) (mixed)... 0.5356580 secs]
5902.813: [GC pause (G1 Evacuation Pause) (mixed)... (to-space exhausted), 1.3480320 secs]
5904.162: [Full GC (Allocation Failure) ... 5.0413590 secs]

A series of "to-space exhausted" events eventually led to a Full GC.  As discussed in [4], adding -Xmn400m on the command line to the same test case further add insult to injury—instead of seeing 3 "to-space exhausted" events, now we saw 20 of them.  This is due to we have further cut down the breathing room for G1.

How to Deal with "to-space exhausted" Events?

"to-space exhausted" events happen when G1 runs out of space (either survivor space or old space; see diagram above) to copy live objects to.  When it happens, sometimes G1's ergonomics can get by the issue by dynamically re-sizing heap regions.  If not, it eventually leads to a Full GC event.

When "to-space exhausted" events happen, it could mean you have a very tight heap allocated for your application.  The easiest approach to resolve this is increasing your heap size if it's possible. Alternatively, sometimes it's possible to tune G1's mixed GC to get by (see [4]). As shown in the above section, we have a high live data occupancy (i.e., 68%) in Java heap.  However, after tuning mixed GC, we can get by with a couple of Full GC's in a 4-hour run, which yields satisfactory result.

Note that mixed GC can clean up both young and old regions. If it does not take an old region, it would be a young(-only) GC. G1's mixed GC's are meant as a replacement for Full GC's. Instead of doing all the work at once, the idea is to achieve the same task in steps. Each small steps in mixed GC is smaller than a Full GC. Because G1 does not intend to reclaim all garbage as in a Full GC event, mixed GC uses less time.  However, Full GC is a compaction event which it can walk around "to-space exhausted" issue without needing extra space.

Acknowledgement

Some writings here are based on the feedback from Thomas Schatzl and Yu Zhang. However, the author would assume the full responsibility for the content himself.

References

1. g1gc logs - basic - how to print and how to understand
2. g1gc logs - Ergonomics -how to print and how to understand
3. JRockit: How to Estimate the Size of Live Data Set
4. JDK 8: Is Tuning MaxNewSize in G1 GC a Good Idea?
5. Garbage First Garbage Collector Tuning
• Note that "to-space overflow" message was removed and only "to-space exhausted" remains.
• RSet (Remembered Set)—Tracks object references into a given region. There is one RSet per region in the heap. The RSet enables the parallel and independent collection of a region. The overall footprint impact of RSets is less than 5%.
• CSet (Collection Set)—Are the set of regions that will be collected in a GC. All live data in a CSet is evacuated (copied/moved) during a GC. Sets of regions can be Eden, survivor, and/or old generation. CSets have a less than 1% impact on the size of the JVM. 
6. HotSpot Virtual Machine Garbage Collection Tuning Guide
7. Garbage-First Garbage Collector Tuning
8. Getting Started with the G1 Garbage Collector
9. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)
10. Other JDK 8 articles on Xml and More
11. Concurrent Marking in G1

G1 GC: Tuning Mixed Garbage Collections in JDK 8

To tune G1 GC (Garbage First Garbage Collector),  a place to start with is [1]. For the latest information, read [2].

Assuming you have minimal knowledge of G1 GC, the focus of this article is to tune the Mixed GC for better performance.

The Need to Tune Mixed GC

G1 GC is a generational garbage collector—read [3] to learn what eden, survivor, and old generation spaces are.  It also uses region-based architecture which divides large contiguous Java heap space into multiple fixed-sized heap regions.

The destination region for a particular object depends upon the object's age; an object that has aged sufficiently is evacuated to an old generation region (or promoted); otherwise, the object is evacuated to a survivor region and will be included in the CSet^[5] of the next young or mixed garbage collection.

During young collections, G1 GC adjusts its young generation (eden and survivor sizes) to meet its pause target. During mixed collections, the G1 GC adjusts the number of old gen regions that are collected based on a target number of mixed garbage collections, the percentage of live objects in each region of the heap, and the overall acceptable heap waste percentage (see details later).

Depending on your application's workload, Full GC could be expensive in G1 GC.  Since Mixed GC collects both young and old regions, you can get better performance by tuning Mixed GC—the goal is to reduce the number of Full GC's when your applications run.

More on Mixed Garbage Collections

Upon successful completion of a concurrent marking cycle, the G1 GC switches from performing young garbage collections to performing mixed garbage collections. In a mixed garbage collection, the G1 GC optionally adds some old regions to the set of eden and survivor regions that will be collected. The exact number of old regions added is controlled by a number of flags that will be discussed later. After the G1 GC collects a sufficient number of old regions (over multiple mixed garbage collections), G1 reverts to performing young garbage collections until the next marking cycle completes.

To summarize, Mixed GC can be characterized by:

• Collecting both young and old regions
• Denoted by: [GC pause (G1 Evacuation Pause) (mixed) in the GC log file^[2]
• Only after a completed marking cycle
• A sequence of mixed GC events to collect old regions, up to 8 by default

Tuning Mixed Garbage Collections

You can play with the following options to tune Mixed GC:^[1,2,4]

• -XX:InitiatingHeapOccupancyPercent 
• Percentage of the (entire) heap occupancy to start a concurrent GC cycle. It is used by GCs that trigger a concurrent GC cycle based on the occupancy of the entire heap, not just one of the generations.  The default value is 45.
• This option can be used to change the marking threshold
• If threshold is exceeded, a concurrent marking will be initiated next.
• The higher the threshold is, the less concurrent marking cycles will be, which also means the less mixed GC evacuation will be.
• -XX:G1MixedGCLiveThresholdPercent 
• This option can be used to change the threshold which determines whether a region should be added to the CSet or not.
• Only regions whose live data percentage are less than the threshold will be added to the CSet.
• The higher the threshold (default: 65) is, the more likely a region will be added to the CSet, which also means more mixed GC evacuation and longer evacuation time will happen. 
• -XX:G1HeapWastePercent
• Amount of reclaimable space, expressed as a percentage of the heap size that G1 will stop doing mixed GC's. If the amount of space that can be reclaimed from old generation regions compared to the total heap is less than this, G1 will stop mixed GC's.
• Current default is 10%.  A lower value, say 5%, will potentially cause G1 to add more expensive region(s) to evacuate for space reclamation.
• G1 will continue triggering mixed GC if the reclaimable is higher than the waste threshold.  So, there will be more mixed GC and maybe more expensive if you set this waste threshold lower.

Besides the above-mentioned options, you may also want to tune the following ones:

• -XX:G1MixedGCCountTarget 
• Sets the target number of mixed garbage collections after a marking cycle to collect old regions with at most G1MixedGCLIveThresholdPercent live data. 
• The default is 8 mixed garbage collections. The goal for mixed collections is to be within this target number.
• -XX:G1OldCSetRegionThresholdPercent
• Sets an upper limit on the number of old regions to be collected during a mixed garbage collection cycle. The default is 10 percent of the Java heap.
• -XX:G1HeapRegionSize
• Sets the size of a G1 region. The value will be a power of two and can range from 1MB to 32MB. 
• The goal is to have around 2048 regions based on the minimum Java heap size.

Conclusion

Each application is unique, you may need to tune G1 GC in an iterative process.  Note that all of the above options are product options except G1MixedGCLiveThresholdPercent and G1OldCSetRegionThresholdPercent, which are experimental options.  Be warned that: for the experimental options, Oracle may remove them at its discretion in the future releases.  

Acknowledgement

Some writings here are based on the feedback from Thomas Schatzl and Yu Zhang. However, the author would assume the full responsibility for the content himself.

References

1. Garbage First Garbage Collector Tuning
2. g1gc logs - Ergonomics -how to print and how to understand 
3. Understanding Garbage Collection  
4. Garbage First (G1) Garbage Collection Options
5. CSet
• The G1 GC reduces heap fragmentation by incremental parallel copying of live objects from one or more sets of regions (called Collection Set (CSet)) into different new region(s) to achieve compaction.
• The goal of G1 GC is to reclaim as much heap space as possible, starting with those regions that contain the most reclaimable space (i..e, garbage first), while attempting to not exceed the pause time goal.
6. G1 GC Glossary of Terms
7. Learn More About Performance Improvements in JDK 8 
8. HotSpot Virtual Machine Garbage Collection Tuning Guide
9. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)
10. G1 GC: Humongous Objects and Humongous Allocations (Xml and More)
11. Other JDK 8 articles on Xml and More
12. Concurrent Marking in G1

HotSpot: What Are Those GC Worker Threads in G1 GC?

On the Internet, you can find some tuning guides for Garbage First Garbage Collector (G1 GC)^[1,5,8,9] and some of them may be outdated.

In this article, we try to update some of those information with the focus on thread tuning.

Ergonomics

Ergonomics for servers was first introduced in Java SE 5.0^[2]. It has greatly reduced application tuning time for server applications, particularly with heap sizing and advanced GC tuning. In many cases no tuning options when running on a server is the best tuning you can do.

Here we will focus on the number of GC worker threads chosen to run on a specific server based on the ergonomics in G1 GC.

Garbage Collector	Worker Threads Used
Parallel	ParallelGCThreads
CMS	ParallelGCThreads ConcGCThreads
G1	ParallelGCThreads ConcGCThreads G1ConcRefinementThreads

G1 Threads

In the table below, we have summarized the most important GC worker threads in Oracle's G1 GC implementation. For comparison, Parallel GC only uses ParallelGCThreads and CMS only uses ParallelGCThreads and ConcGCThreads. Be warned that these values can be changed in the future if Oracle sees appropriate.

GC thread counts are based on the number of processors (including virtual processor if hyper-threading^[3] is enabled) reported by the system. To see the number of processors you have in the system, check out the file /proc/cpuinfo. In this article, 32 processors are used in the demonstration.

Name	Controlled by	Default if Not Specified	Description
Parallel GC Threads	-XX:ParallelGCThreads	if #proc <=8, then #proc; if #proc >8, 8+(#proc-8)*(5/8)	Parallel operations
Parallel Marking Threads	-XX:ConcGCThreads	There are lot of cases, but mostly it uses max((ParallelGCThreads+2)/4, 1)
G1 Main Concurrent Mark GC Thread		1	Master concurrent thread
G1 Concurrent Refinement Thread	-XX:G1ConcRefinementThreads	ParallelGCThreads+1	The +1 is the master thread which control these worker threads.

What Are They?

G1 garbage collection cycle has multiple phases: it starts with an initial marking phase and ends with a concurrent-cleanup phase. During the cycle, there are some phases that pause all the application threads, and some that run concurrently. These GC worker threads used in G1 GC include, but not limited to:

• ParallelGCThreads
• ParallelGCThreads are the threads employed during GC pauses
• They are responsible for copying (and scanning for) live objects during stop-the-world (or STW) pauses.
• They are also employed during other STW pauses (such as Remark and Cleanup).
• ConcGCThreads
• In concurrent marking, there is one controller thread and another bunch of worker threads. The number of worker threads is set using ConcGCThreads or a fraction of ParallelGCThreads if ConcGCThreads is not set.
• Worker threads are the threads used for marking and determining the liveness of regions, which are run concurrently to the application threads.
• Increasing ConcGCThreads will make the marking cycle complete faster.
• G1ConcRefinementThreads
• G1ConcRefinementThreads are the threads used for updating remembered set while application threads are running concurrently.
• They are responsible for taking the buffers that are used to log object updates and updating the RememberedSets (or RSets)^[10] based upon the updates.
• If G1ConcRefinementThreads is not set, then its value is ParallelGCThreads + 1.
• For example, if ParallelGCThreads is 23 then we will have 24 refinement threads.
• One is used for sampling and updating the RSets of young regions (typically young regions have less cross-region references so one thread is enough); the other 23 threads fill update buffers.
• Typically they don't all run at once.
• They each have a stepped activation threshold and a thread is activated when the number of completed buffers exceeds it's activation threshold.
• They also have deactivation thresholds and when the number of filled update buffers falls below a thread's deactivation threshold, it is deactivated.

Thread Tuning

For our server with 32 processors, the number of GC worker threads chosen by G1's ergonomics are:

• # ParallelGCThreads = 23
• # ConcGCThreads = 6
• # G1ConcRefinementThreads = 23

To find out what the GC worker thread counts are for your server, you can use -XX:+PrintFlagsFinal.^[4] You can also change the default values if you see appropriate.^[6,7] For some systems, default values sometimes can be too high.

References

1. Garbage First Garbage Collector Tuning (Published August 2013)
2. Java Heap Sizing: How do I size my Java heap correctly?
3. Intel® Hyper-Threading Technology
4. What Are the Default HotSpot JVM Values?
5. Part #1 - Tuning Java Garbage Collection for HBase
6. Linux: Understanding Processor Queue in Vmstat Output
7. How to Troubleshoot High CPU Usage of Java Applications? 
8. Getting Started with the G1 Garbage Collector 
9. G1: One Garbage Collector To Rule Them All 
10. Remembered Sets (RSets) in G1 GC
• G1 GC uses independent Remembered Sets (RSets) to track references into regions. Independent RSets enable parallel and independent collection of regions because only a region's RSet must be scanned for references into that region, instead of the whole heap. To achieve the task, G1 GC uses a post-write barrier to record changes to the heap and update the RSets. 
11. D. L. Detlefs, C. H. Flood, S. Heller, and T. Printezis. Garbage-First Garbage Collection. In A. Diwan, editor, Proceedings of the 2004 International Symposium on Memory Management (ISMM 2004), pages 37-48, Vancouver, Canada. October 2004. ACM Press.
12. T. Printezis and D. L. Detlefs. A Generational Mostly-Concurrent Garbage Collector. In A. L. Hosking, editor, Proceedings of the 2000 International Symposium on Memory Management (ISMM 2000), pages 134-154,Minneapolis, MN, USA, October 2000. ACM Press. 
13. g1gc logs - how to print and how to understand - basic 
14. G1 GC Glossary of Terms
15. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)