How to Troubleshoot High CPU Usage of Java Applications?

A Java application that is constantly maxing out the CPU load sometimes can be a good thing^[3]. For instance, for a batch application that is computationally bound, it would normally be a best case scenario for it to complete as soon as possible. Also idled CPU could be a waste and should be avoided. CPU idle happens when the system:

• Needs to wait for locks or external resources
• The application might be blocked on a synchronization primitive and unable to execute until that lock is released
• The application might be waiting for something, such as a response to come back from a call to the database
• Has no threads available to handle the work in a multithreaded, multi-CPU case
• Has nothing to do

Looking at the % CPU utilization is a first step in understanding your application performance, but it is only that—Use it to see if you are using all the CPU you expect, or if it points to some synchronization or resource issue.

Normally, some over-provisioning is needed to keep an application responsive. If the CPU usage is very high (i.e, consistently over 95%), you may want to invest in better hardware, or look over the data structures and algorithms employed by the application.

In this article, we will show you how to investigate where all those CPU cycles are being spent in your Java applications.

How to Troubleshoot High CPU Usage?

The easiest approach is to generate a sequence of thread dumps to see what's keeping the processor busy. Note that you can't tell much from a single thread dump. So, you need to generate a sequence of thread dumps.

Thread dumps generated at high CPU times are the most useful. To monitor CPU usage, you can use Linux tools like top ^[2] or prstat^[4] to see which threads are consuming the most CPU and get thread dumps at the same time. Then you can map the ids. It may end up being GC that is taking the CPU if your memory pressure is high. In that case, you also need to gather GC logs for further analysis.

Using top Linux command, Java threads (or Linux LWP's) will be sorted based on the %CPU by default. Pressing Shift+F, you will be shown a screen specifying the current sort field. Then you can select different sort field by selecting different field letter. For example, select "n" for sorting by memory usage (RES).

User Time vs. System Time

Some Linux commands (i.e., vmstat)^[1] can report CPU time spent in either system or user space. User time (including nice time for vmstat) is the percentage of time the CPU is executing application code (including GC code), while system time is the percentage of time the CPU is executing kernel code.

System time could be related to your application too. For example, if your application performs I/O, the kernel will execute the code to read the file from disk, or write the network buffer, and so on. High levels of system time often mean something is wrong, or the application is making many system calls. Investigating the cause of high system time is always worthwhile.

CPU Tuning

The goal in performance is always to drive the CPU usage as high as possible (for as short a time as possible). The CPU number is an indication of how effectively the program is using the expensive CPU, and so the higher the number the better. As previously mentioned, in some CPU-bound applications (i.e., the CPU is the limiting factor), for example batch jobs, it is normally a good thing for the system to be completely saturated during the run. However, for a standard server-side application it is probably more beneficial if the system is able to handle some extra load in addition to the expected one.

Based on [8], Oracle has provided the following tuning guidelines (including CPU tuning) for its Fusion Applications:

Metric Category	Metric Name	Warning Threshold	Critical Threshold	Comments
Disk Activity	Disk Device Busy	>80%	>95%
Filesystems	Filesystem Space Available	<20	<5
Load	CPU in I/O wait	>60%	>80%
	CPU Utilization	>80%	>95%
	Run Queue (5 min average)	>2	>4	The run queue is normalized by the number of CPU cores.
	Swap Utilization	>75%	>90%
	Total Processes	>15000	>25000
	Logical Free Memory %	<20	<10
	CPU in System Mode	>20%	>40%
Network Interfaces Summary	All Network Interfaces Combined Utilization	>80%	>95%
Switch/Swap Activity	Total System Swaps	>3	>5	Value is per second.
Paging Activity	Pages Paged-in (per second)
	Pages Paged-out (per second)			The combined value of Pages Paged-in and Pages Paged-out should be <=1000

Oracle Performance Tools

To analyze high CPU usage in Java applications, the best approach is to use enterprise profilers. For example, Oracle Solaris Studio^[5] can offer more performance details and better measurements.  Now it can run on Oracle Solaris, Oracle Linux, and Red Hat Enterprise Linux operating systems.

The Oracle Solaris Studio Performance Analyzer can be extremely useful to identify bottlenecks and provide advanced profiling for your applications. The key features of it includes^[6]:

• Low overhead for fast and accurate results
• Advanced profiling of single-threaded and multithreaded applications
• Support for multiprocess and system-wide profiling
• Ability to analyze MPI applications
• Support for C, C++, Fortran, and Java code
• Optimized for the latest Oracle systems

If your applications run in JRockit, another good way to profile CPU usage is to capture JRockit Flight Recordings^[7]. JFR can provide extremely detailed level of profiling with little impact of your application performance.  If you use HotSpot, Java Mission Control and Java Flight Recorder (commercial features) are now available for Java SE 7u40 as well as for JRockit^[9].

References

1. iostat, vmstat and mpstat Examples for Linux Performance Monitoring
2. Linux / Unix Command: top
3. Oracle JRockit: The Definitive Guide Paperback by Marcus Hirt , Marcus Lagergren
4. PRSTAT Linux - How to pinpoint high CPU Java VM Threads (good)
5. Oracle Solaris Studio
6. Oracle Solaris Studio Performance Analyzer
7. Oracle® JRockit Flight Recorder Run Time Guide Release R28
8. Oracle® Fusion Applications Performance and Tuning Guide 11g Release 1 (11.1.2)
9. New Diagnostic, Monitoring, Security and Deployment Capabilities for Java SE 7
10. All other performance tuning articles on XML and More
11. Guide to Advanced Linux Command Mastery, Part 3: Resource Management (OTN)
12. Oracle® Fusion Middleware Tuning Performance of Oracle WebLogic Server 12c (12.2.1)

openSubTask and closeSubTask APIs

openSubTask and openMainTask seem to be related and have similar parameter lists. However, they behave differently and are used for different purposes. This article examines what openSubTask and closeSubTask does in details.

In a nutshell, openSubTask and closeSubTask APIs are programmatic support in Oracle Applications Core to notify recent items about a sub-flow launch.

Recent Items and Favorites^[1]

Recent Items in Oracle Fusion Web Applications tracks a list of the last 20 task flows visited by a user. The Recent Items list is persistent across user sessions and a task can be relaunched from the Recent Items list.

The Favorites menu is implemented on top of Recent Items. Any current task on the Recent Items list can be bookmarked and placed in Favorites' folders.

openMainTask and closeMainTask

Both openMainTask and openSubTask (or their close counterparts) are provided on FndUIShellController ^[2]data control. At design time, you can drag and drop both operations to UI components (for example, your page fragments) to create needed bindings objects in XML meta data files:

openMainTask is used to open a new Task in the Main Area of Oracle Fusion Web Applications that use UIShell templates^[5]. Besides opening a new tab, openMainTask also pushes a new Task Flow History object onto a stack, which is used to keep track of all task flows that have been opened. The Task Flow ID and its associated parameter values are encapsulated in the Task Flow History object.

Having this information, the call to closeMainTask pops the stack to get the last Task Flow ID and its parameter values that were displayed, and re-initializes the Main Area with that Task Flow and parameter information.

Sub-Flow

When a task flow is called from the local area task flow via task flow call activity, it is called a 'sub-flow'. By default, subflows will not be recorded on the stack as described above. Two new APIs^[3] are exposed in FndUIShellController^[2] data control for registering sub-flows:

• openSubTask
• closeSubTask.

openSubTask and closeSubTask

When openSubTask is called before a sub-flow is launched, sub-flow ID and its parameter values are pushed onto the stack. Besides that, Applications Core implementation also notifies Recent Items & Favorites^[1] (or RI & Favs) implementation with recorded task flow information. Once tasks are recorded on the Recent Items list, they are also eligible for Favorites.
This essentially makes a sub-flow bookmarkable by RI & Favs and can be launched directly from the selection of menu items on Recent Items.
When a sub flow finishes execution, it should call closeSubTask on all of its exit paths. What closeSubTask API does is:

if topOfStack is SUB_FLOW type {
  Pop it from the stack;
  Notify RI&Favs for the new topOfStack entry;
} else {
  Look inside the stack to find a match based on sub flow ID and its keyList;
  if (found the first one) {
    Remove it from the Stack;
    Notify RI&Favs for the new topOfStack entry;
  }
}

Contextual events are fired when openSubTask or closeSubTask excute. The method outcome (i.e., sub-flow information encapsulated in FndMethodParameters) is delivered as the contextual event payload.
For openSubTask, an FND_OPEN_SUB_TASK event is fired and the event consumer is handleOpenSubTaskEvent method on the MainAreaHandler object.
Similarly, a FND_CLOSE_SUB_TASK event is fired when closeSubTask method executes and the event consumer is handleCloseSubTaskEvent method on the MainAreaHandler object. The main functionality of openSubTask and closeSubTask from the Applications Core side are both implemented in these event handlers.
When RI & Favs is notified with a new task flow, it does the following:

if user session (i.e., FND session) is not enabled or user is anonymous {
  do nothing and return;
}
if any of the following is not set, do nothing and return:
  Task flow ID
  Task flow label
  Task flow view ID
  Task flow webApp name
if an existing task flow on the RI list matches the new task flow based on the following condition:
  Same Task flow ID
  Same Task flow label
  Same Task flow view ID
  Same webApp name
  Same reuseInstance value
  Same keyList
then
Remove it from the list and move it to the top and update its information with new entry
else
  Add a new task flow history object to the list

Hope this article can help Oracle Fusion Developers to understand more on the functionality of Recent Items and Favorites within Oracle Fusion Web Applications. However, the details provided here are based on author's own understanding of Recent Items and Favorites implementation. These implementation details may evolve over time and there is no guarantee that they will remain valid in the future.

References

1. Implementing Recent Items
2. FndUIShellController
3. openSubTask and closeSubTask
4. openMainTask
5. Implementing the UI Shell
6. Oracle ADF Task Flow in a Nutshell