Monitoring WebLogic Server Thread Pool at Runtime

There are two critical areas for WebLogic Server performance tuning:

• Thread management
• Network I/O tuning

Before any tuning, you need to identify areas of bottlenecks first.  In this article, we will focus on thread management and specifically its run-time health monitoring.

Thread Pool

There are thread pool implementation changes since WLS 9.0: 

• Previous versions
• Multiple pools of threads 
• See [7] on how to use the WebLogic 8.1 thread pool model for backward compatibility
• WLS 9.0 and above
• A single dynamically sized pool of threads (or self-tuning thread pool)
• Self-tuning work manager^[8,9]

New WebLogic Server uses a single thread pool, in which all types of work are executed. Here we summarize how the new implementation works:

• WebLogic Server prioritizes work and allocates threads based on an execution model that takes into account 
• • Administrator-defined parameters 
  • Actual run-time performance 
  • Throughput
• The common thread pool changes its size automatically to maximize throughput.
• This new strategy makes it easier for administrators to allocate processing resources and manage performance, avoiding the effort and complexity involved in configuring, monitoring, and tuning custom executes queues. 
• The queue monitors throughput over time and based on history, determines whether to adjust the thread count.  For example,
• Thread count will be increased when:
• If historical throughput statistics indicate that a higher thread count increased throughput, WebLogic increases the thread count. 
• Thread count will be reduced when
• If statistics indicate that fewer threads did not reduce throughput, WebLogic decreases the thread count. 

In general, the self-tuning thread pool changes its size automatically to maximize throughput, so in normal cases there is nothing you need to do aside from monitoring it to understand the behavior of your server under different types of load.

WebLogic Server Monitoring Dashboard

From the Oracle WebLogic Administration Console, you can navigate to

• Servers | SalesServer_1 | Monitoring | Threads 

to view important statistics related to thread pool and thread pool threads.

Everything from Active Execute Threads to Min Threads Contrain Complete is shown on this page.

From the dashboard, the thread pool runtime can be monitored in real-time. The key KPI's to monitor include^[2]:

• Hogging Thread Count 
• The threads that are being held by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
• Pending User Request Count
• The number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
• Queue Length
• The number of requests queued up when you don’t have idle threads.  

On a low usage environment, these should ideally be hovering around zero.

Another KPI worth of monitoring is

• Throughput
• The Throughput is a single value that denotes the mean number of requests completed per second. 

The higher this value is, the better it is.

Pool Size

In some cases, you do want to set the range of pool size.  For example, for our Fusion Application benchmarks, our area of interest is in the JVM (i.e, not upper layers).  In that case, we want to reduce the variation introduced by the self-tuning of thread pool and set our thread pool size to be:

• -Dweblogic.threadpool.MinPoolSize=32 -Dweblogic.threadpool.MaxPoolSize=32

The general rule^[4] for pool sizing or other kinds of tuning is to start with no specific tuning and then configure Work Managers only to address specific problems that might arise. Aggressively configuring Work Managers for a specific environment can end up hurting performance when your application, workload, or underlying system changes.

Stuck Threads

If an execute thread is being hogged by a request for much more than the normal execution time (as automatically observed by the scheduler), it's declared as a hogger.  These threads will either be declared as stuck after the configured timeout (by default, 10 min of processing time) or will return to the pool before that.

If you find any thread's state become STUCK, it's the time you start investigating—does the stuck thread ever recover or does it stay stuck indefinitely?  By default, Oracle Fusion Apps is configured to generate an incident when a STUCK thread is detected.  You can find them here:

• <DOMAIN_HOME>/servers/<server_name>/adr/diag/ofm/<domain_name>/<server_name>/incident

The incident contains some key diagnostic information that can be used to help understand why the request took so long.

What to Expect if Things Work Normally?

This thread below is what an execute thread in the WebLogic Server self-tuning pool looks like when there is nothing for it to do.

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x3c tid=3810 prio=5 alive, waiting, native_blocked, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0xa0c21480[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xa0c21480[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace

The thread below is a timer thread waiting to be notified that it is time to wake up and do whatever it is supposed to do:

"JFR request timer" id=16 idx=0x40 tid=3811 prio=5 alive, waiting, native_blocked, daemon
    -- Waiting for notification on: java/util/TaskQueue@0xa0c20b28[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at java/util/TimerThread.mainLoop(Timer.java:483)
    ^-- Lock released while waiting: java/util/TaskQueue@0xa0c20b28[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace

What state is the “main” thread in? It should look something like this one:

"main" prio=6 tid=0x000000000224f000 nid=0x2e64 runnable [0x00000000023de000]
   java.lang.Thread.State: RUNNABLE
        at weblogic.i18n.Localizer.prune(Localizer.java:358)
        at weblogic.i18n.Localizer.getObject(Localizer.java:164)
        at weblogic.i18n.Localizer.getDiagnosticVolume(Localizer.java:344)
        at weblogic.i18n.logging.CatalogMessage.(CatalogMessage.java:53)
        at weblogic.kernel.T3SrvrLogger.logServerStateChange(T3SrvrLogger.java:2084)
        at weblogic.t3.srvr.T3Srvr.setState(T3Srvr.java:211)
        - locked <0x00000000e0a30d78> (a weblogic.t3.srvr.T3Srvr)
        at weblogic.t3.srvr.T3Srvr.initializeAdmin(T3Srvr.java:921)
        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:589)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:471)
        at weblogic.Server.main(Server.java:74)

References

1. Oracle® Fusion Applications Performance and Tuning Guide 11g Release 1 (11.1.4)
2. Oracle SOA Suite 11g Administrator's Handbook
3. Tuning WebLogic Server
4. Oracle WebLogic Server 11g Administration Handbook
5. Controlling Thread Pool Size in WebLogic Server
6. Understanding JVM Thread States
7. Using the WebLogic 8.1 Thread Pool Model
   • Describes how to use and tune WebLogic 8.1 thread pools
8. Using Work Managers to Optimize Scheduled Work
   
9. Understanding WebLogic Work Manager
10. Oracle® Fusion Middleware Tuning Performance of Oracle WebLogic Server 12c (12.2.1)
11. Data Source Connection Pool Sizing (The Weblogic Server Blog)

Controlling Thread Pool Size in WebLogic Server

One of the critical areas that relate to the tuning of WebLogic Server is thread management^[1].  In previous versions of WebLogic Server, processing was performed in multiple execute queues. Different classes of work were executed in different queues, based on priority and ordering requirements, and to avoid deadlocks.  However, in WLS 9.0 and above, it uses a single thread pool, in which all types of work are executed.^[10] WebLogic Server prioritizes work based on rules you define, and run-time metrics, including the actual time it takes to execute a request and the rate at which requests are entering and leaving the pool.

Self-tuning Thread Pool^[7]

WebLogic uses work managers with a variable and self-tuning number of worker threads. By default, the self-tuning thread pool size limit is 400. This limit includes all running and idle threads, but does not include any standby threads. The size of thread pool grows and shrinks automatically to improve throughput.  Measurements are taken every 2 seconds and the decision to increase or decrease the thread count is based on the current throughput measurement versus past values.

Thread Management^[4]

If your server has four physical processors, theoretically you only need a thread pool with four threads. When you have more threads than there are CPU resources, the throughput may suffer. However, if your threads often make database connections or call some other long-running tasks where they need to wait, you do want to have more threads around so that the ones that aren't waiting can do some work.

In a 3-tiered architecture, you can also have a situation like this: the clients make requests coming into the application server faster than the database server can handle.  Then the clients keep adding requests on the application server until all its threads are busy, all of which just adds load to the database.  The more load you add to the application server that is overloaded, the worse you make the situation.  In other cases that clients cannot keep all threads on the application server busy and leave some of them idle, you may still lose throughput because the cache will be less efficient when a new thread takes a new request versus when a just-used thread takes a new request.

At any rate, tuning the size of thread pool is challenging and time consuming.  Internally Weblogic Server has many work managers configured for different types of work. If WLS runs out of threads in the self-tuning pool (because of system property -Dweblogic.threadpool.MaxPoolSize) due to being undersized, then important work that WLS might need to do could be starved.  While limiting the self-tuning would limit the default WorkManager and internally it also limits all other internal WorkManagers which WLS uses.  So, leaving that task to WebLogic Server seems to be a wise choice. 

However, there are some cases that we do need to set the size of thread pool manually.  For example, to make performance comparison between two different test cases, you may want to eliminate the thread-pool-size variance from the performance results.  In this article, we will show you how to set up minimum and maximum thread pool sizes and how to examine the results of the settings.

Controlling the Size of Thread Pool

There are different ways of changing the size of thread pool.  One way of doing it is by setting them from the command line:

• -Dweblogic.threadpool.MinPoolSize=5 -Dweblogic.threadpool.MaxPoolSize=5

By setting both MinPoolSize and MaxPoolSize to be the same value, we have forced WLS to use exactly five worker threads.  In our case, our two test cases will be compared with the same number of worker threads and prevent the self-tuning effects from contaminating our performance results.  In [8], it also tells us how to make similar changes via config.xml.

Threads Page on WLS Console

For a WebLogic Server administrator, the WLS console is indispensable for monitoring running server instances, including the various subsystems such as security, JTA, and JDBC. The Threads page on the WLS console provides information on the thread activity for the current server.  In Figure 1, it shows that there are five Active Execute Threads based on our configuration.  If we didn't configure the thread pool size, you could see number of Active Execute Threads changing dynamically due to WLS' self-tuning activities.


.

Default Execute Queue from Thread Dump

Besides monitoring number of worker threads from the WLS console, you can also examine them from the thread dump as generated from JStack^[3].

Unless you've customized the execute queue (or thread pool) that your application gets deployed to, you can look for "Default" execute queue.  In the dump file, you'll look for the threads marked as 'weblogic.kernel.Default' to see what's running.  As work enters an instance of WLS, it is placed in the default execute queue.  This work is then assigned to a worker thread that does the work on it.

$ grep weblogic.kernel.Default threadDump.fod1
"[STANDBY] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00000000202d9800 nid=0x404a in Object.wait() [0x0000000040801000]
"[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x0000000021813800 nid=0x3d13 in Object.wait() [0x000000004c52d000]
"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002aaabc0c6800 nid=0x3811 runnable [0x000000004a107000]
"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002aaabc0c5000 nid=0x3810 in Object.wait() [0x000000004a008000]
"[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002aaabc0c1800 nid=0x380f in Object.wait() [0x0000000049f06000]
"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002aaabc0db800 nid=0x380e in Object.wait() [0x000000004194e000]
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002aaabc0bc800 nid=0x380d in Object.wait() [0x000000004184d000]

As shown above, you can find there are seven instances of 'weblogic.kernel.Default (self-tuning)'.  Five of them are active and two of them are in standby.^[11]  These five active instances match what we've found as "Active Execute Thread" on the WLS console.

References

1. Oracle WebLogic Server 11g Administration Handbook by Sam Alapati
2. Using Work Managers to Optimize Scheduled Work
3. Fun with JStack by Scott Oaks
4. Rewritten from personal's email exchanges with Scott Oaks
5. Monitoring WebLogic Server Thread Pool at Runtime
6. Understanding JVM Thread States
7. Self-Tuning Thread Pool
8. Tuning Default WorkManager - Advantages and Disadvantages
9. Fusion Middleware Performance and Tuning for Oracle WebLogic Server
10. Understanding the Differences Between Work Managers and Execute Queues
11. STANDBY thread (WLS)
• ACTIVE threads can go to STANDBY when it is deemed that you don’t need that many active threads.
• But, a STANDBY thread can still be used (without transitioning to ACTIVE) in order to satisfy a min threads constraint.
12. Top Tuning Recommendations for WebLogic Server (12.2.1.3.0)
13. Analyzing Thread Dumps in Middleware - Part 2