HotSpot: What Are Those GC Worker Threads in G1 GC?

On the Internet, you can find some tuning guides for Garbage First Garbage Collector (G1 GC)^[1,5,8,9] and some of them may be outdated.

In this article, we try to update some of those information with the focus on thread tuning.

Ergonomics

Ergonomics for servers was first introduced in Java SE 5.0^[2]. It has greatly reduced application tuning time for server applications, particularly with heap sizing and advanced GC tuning. In many cases no tuning options when running on a server is the best tuning you can do.

Here we will focus on the number of GC worker threads chosen to run on a specific server based on the ergonomics in G1 GC.

Garbage Collector	Worker Threads Used
Parallel	ParallelGCThreads
CMS	ParallelGCThreads ConcGCThreads
G1	ParallelGCThreads ConcGCThreads G1ConcRefinementThreads

G1 Threads

In the table below, we have summarized the most important GC worker threads in Oracle's G1 GC implementation. For comparison, Parallel GC only uses ParallelGCThreads and CMS only uses ParallelGCThreads and ConcGCThreads. Be warned that these values can be changed in the future if Oracle sees appropriate.

GC thread counts are based on the number of processors (including virtual processor if hyper-threading^[3] is enabled) reported by the system. To see the number of processors you have in the system, check out the file /proc/cpuinfo. In this article, 32 processors are used in the demonstration.

Name	Controlled by	Default if Not Specified	Description
Parallel GC Threads	-XX:ParallelGCThreads	if #proc <=8, then #proc; if #proc >8, 8+(#proc-8)*(5/8)	Parallel operations
Parallel Marking Threads	-XX:ConcGCThreads	There are lot of cases, but mostly it uses max((ParallelGCThreads+2)/4, 1)
G1 Main Concurrent Mark GC Thread		1	Master concurrent thread
G1 Concurrent Refinement Thread	-XX:G1ConcRefinementThreads	ParallelGCThreads+1	The +1 is the master thread which control these worker threads.

What Are They?

G1 garbage collection cycle has multiple phases: it starts with an initial marking phase and ends with a concurrent-cleanup phase. During the cycle, there are some phases that pause all the application threads, and some that run concurrently. These GC worker threads used in G1 GC include, but not limited to:

• ParallelGCThreads
• ParallelGCThreads are the threads employed during GC pauses
• They are responsible for copying (and scanning for) live objects during stop-the-world (or STW) pauses.
• They are also employed during other STW pauses (such as Remark and Cleanup).
• ConcGCThreads
• In concurrent marking, there is one controller thread and another bunch of worker threads. The number of worker threads is set using ConcGCThreads or a fraction of ParallelGCThreads if ConcGCThreads is not set.
• Worker threads are the threads used for marking and determining the liveness of regions, which are run concurrently to the application threads.
• Increasing ConcGCThreads will make the marking cycle complete faster.
• G1ConcRefinementThreads
• G1ConcRefinementThreads are the threads used for updating remembered set while application threads are running concurrently.
• They are responsible for taking the buffers that are used to log object updates and updating the RememberedSets (or RSets)^[10] based upon the updates.
• If G1ConcRefinementThreads is not set, then its value is ParallelGCThreads + 1.
• For example, if ParallelGCThreads is 23 then we will have 24 refinement threads.
• One is used for sampling and updating the RSets of young regions (typically young regions have less cross-region references so one thread is enough); the other 23 threads fill update buffers.
• Typically they don't all run at once.
• They each have a stepped activation threshold and a thread is activated when the number of completed buffers exceeds it's activation threshold.
• They also have deactivation thresholds and when the number of filled update buffers falls below a thread's deactivation threshold, it is deactivated.

Thread Tuning

For our server with 32 processors, the number of GC worker threads chosen by G1's ergonomics are:

• # ParallelGCThreads = 23
• # ConcGCThreads = 6
• # G1ConcRefinementThreads = 23

To find out what the GC worker thread counts are for your server, you can use -XX:+PrintFlagsFinal.^[4] You can also change the default values if you see appropriate.^[6,7] For some systems, default values sometimes can be too high.

References

1. Garbage First Garbage Collector Tuning (Published August 2013)
2. Java Heap Sizing: How do I size my Java heap correctly?
3. Intel® Hyper-Threading Technology
4. What Are the Default HotSpot JVM Values?
5. Part #1 - Tuning Java Garbage Collection for HBase
6. Linux: Understanding Processor Queue in Vmstat Output
7. How to Troubleshoot High CPU Usage of Java Applications? 
8. Getting Started with the G1 Garbage Collector 
9. G1: One Garbage Collector To Rule Them All 
10. Remembered Sets (RSets) in G1 GC
• G1 GC uses independent Remembered Sets (RSets) to track references into regions. Independent RSets enable parallel and independent collection of regions because only a region's RSet must be scanned for references into that region, instead of the whole heap. To achieve the task, G1 GC uses a post-write barrier to record changes to the heap and update the RSets. 
11. D. L. Detlefs, C. H. Flood, S. Heller, and T. Printezis. Garbage-First Garbage Collection. In A. Diwan, editor, Proceedings of the 2004 International Symposium on Memory Management (ISMM 2004), pages 37-48, Vancouver, Canada. October 2004. ACM Press.
12. T. Printezis and D. L. Detlefs. A Generational Mostly-Concurrent Garbage Collector. In A. L. Hosking, editor, Proceedings of the 2000 International Symposium on Memory Management (ISMM 2000), pages 134-154,Minneapolis, MN, USA, October 2000. ACM Press. 
13. g1gc logs - how to print and how to understand - basic 
14. G1 GC Glossary of Terms
15. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)

Monitoring WebLogic Server Thread Pool at Runtime

There are two critical areas for WebLogic Server performance tuning:

• Thread management
• Network I/O tuning

Before any tuning, you need to identify areas of bottlenecks first.  In this article, we will focus on thread management and specifically its run-time health monitoring.

Thread Pool

There are thread pool implementation changes since WLS 9.0: 

• Previous versions
• Multiple pools of threads 
• See [7] on how to use the WebLogic 8.1 thread pool model for backward compatibility
• WLS 9.0 and above
• A single dynamically sized pool of threads (or self-tuning thread pool)
• Self-tuning work manager^[8,9]

New WebLogic Server uses a single thread pool, in which all types of work are executed. Here we summarize how the new implementation works:

• WebLogic Server prioritizes work and allocates threads based on an execution model that takes into account 
• • Administrator-defined parameters 
  • Actual run-time performance 
  • Throughput
• The common thread pool changes its size automatically to maximize throughput.
• This new strategy makes it easier for administrators to allocate processing resources and manage performance, avoiding the effort and complexity involved in configuring, monitoring, and tuning custom executes queues. 
• The queue monitors throughput over time and based on history, determines whether to adjust the thread count.  For example,
• Thread count will be increased when:
• If historical throughput statistics indicate that a higher thread count increased throughput, WebLogic increases the thread count. 
• Thread count will be reduced when
• If statistics indicate that fewer threads did not reduce throughput, WebLogic decreases the thread count. 

In general, the self-tuning thread pool changes its size automatically to maximize throughput, so in normal cases there is nothing you need to do aside from monitoring it to understand the behavior of your server under different types of load.

WebLogic Server Monitoring Dashboard

From the Oracle WebLogic Administration Console, you can navigate to

• Servers | SalesServer_1 | Monitoring | Threads 

to view important statistics related to thread pool and thread pool threads.

Everything from Active Execute Threads to Min Threads Contrain Complete is shown on this page.

From the dashboard, the thread pool runtime can be monitored in real-time. The key KPI's to monitor include^[2]:

• Hogging Thread Count 
• The threads that are being held by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
• Pending User Request Count
• The number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
• Queue Length
• The number of requests queued up when you don’t have idle threads.  

On a low usage environment, these should ideally be hovering around zero.

Another KPI worth of monitoring is

• Throughput
• The Throughput is a single value that denotes the mean number of requests completed per second. 

The higher this value is, the better it is.

Pool Size

In some cases, you do want to set the range of pool size.  For example, for our Fusion Application benchmarks, our area of interest is in the JVM (i.e, not upper layers).  In that case, we want to reduce the variation introduced by the self-tuning of thread pool and set our thread pool size to be:

• -Dweblogic.threadpool.MinPoolSize=32 -Dweblogic.threadpool.MaxPoolSize=32

The general rule^[4] for pool sizing or other kinds of tuning is to start with no specific tuning and then configure Work Managers only to address specific problems that might arise. Aggressively configuring Work Managers for a specific environment can end up hurting performance when your application, workload, or underlying system changes.

Stuck Threads

If an execute thread is being hogged by a request for much more than the normal execution time (as automatically observed by the scheduler), it's declared as a hogger.  These threads will either be declared as stuck after the configured timeout (by default, 10 min of processing time) or will return to the pool before that.

If you find any thread's state become STUCK, it's the time you start investigating—does the stuck thread ever recover or does it stay stuck indefinitely?  By default, Oracle Fusion Apps is configured to generate an incident when a STUCK thread is detected.  You can find them here:

• <DOMAIN_HOME>/servers/<server_name>/adr/diag/ofm/<domain_name>/<server_name>/incident

The incident contains some key diagnostic information that can be used to help understand why the request took so long.

What to Expect if Things Work Normally?

This thread below is what an execute thread in the WebLogic Server self-tuning pool looks like when there is nothing for it to do.

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x3c tid=3810 prio=5 alive, waiting, native_blocked, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0xa0c21480[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xa0c21480[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace

The thread below is a timer thread waiting to be notified that it is time to wake up and do whatever it is supposed to do:

"JFR request timer" id=16 idx=0x40 tid=3811 prio=5 alive, waiting, native_blocked, daemon
    -- Waiting for notification on: java/util/TaskQueue@0xa0c20b28[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at java/util/TimerThread.mainLoop(Timer.java:483)
    ^-- Lock released while waiting: java/util/TaskQueue@0xa0c20b28[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace

What state is the “main” thread in? It should look something like this one:

"main" prio=6 tid=0x000000000224f000 nid=0x2e64 runnable [0x00000000023de000]
   java.lang.Thread.State: RUNNABLE
        at weblogic.i18n.Localizer.prune(Localizer.java:358)
        at weblogic.i18n.Localizer.getObject(Localizer.java:164)
        at weblogic.i18n.Localizer.getDiagnosticVolume(Localizer.java:344)
        at weblogic.i18n.logging.CatalogMessage.(CatalogMessage.java:53)
        at weblogic.kernel.T3SrvrLogger.logServerStateChange(T3SrvrLogger.java:2084)
        at weblogic.t3.srvr.T3Srvr.setState(T3Srvr.java:211)
        - locked <0x00000000e0a30d78> (a weblogic.t3.srvr.T3Srvr)
        at weblogic.t3.srvr.T3Srvr.initializeAdmin(T3Srvr.java:921)
        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:589)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:471)
        at weblogic.Server.main(Server.java:74)

References

1. Oracle® Fusion Applications Performance and Tuning Guide 11g Release 1 (11.1.4)
2. Oracle SOA Suite 11g Administrator's Handbook
3. Tuning WebLogic Server
4. Oracle WebLogic Server 11g Administration Handbook
5. Controlling Thread Pool Size in WebLogic Server
6. Understanding JVM Thread States
7. Using the WebLogic 8.1 Thread Pool Model
   • Describes how to use and tune WebLogic 8.1 thread pools
8. Using Work Managers to Optimize Scheduled Work
   
9. Understanding WebLogic Work Manager
10. Oracle® Fusion Middleware Tuning Performance of Oracle WebLogic Server 12c (12.2.1)
11. Data Source Connection Pool Sizing (The Weblogic Server Blog)