On Stack Replacement in HotSpot JVM

In JVM, JIT Compiler is used to convert bytecode at runtime prior to executing it natively.  Since the JIT does not have time to compile every single method in an Java application, all code starts out initially running in the interpreter, and once it becomes hot enough it gets scheduled for compilation.

The HotSpot VM can perform special compiles called On Stack Replacement compiles, or OSRs.  These are used when Java code contains a long-running loop (see the example here) that started executing in the interpreter.

As described in [1], it is important to know about OSR if you want to benchmark Java programs and have put everything in a loop to be run in a method such as main.  You may run into some pitfalls caused by OSR on your benchmark results.  Without further ado, I'll refer you to read [1] for more details.

On Stack Replacement (OSR)^[2]

Normally the way Java code ends up in compiled code is that when invoking a method the interpreter detects that there's compiled code for it, and it dispatches to that instead of staying in the interpreter.  This does not help long-running loops that started in the interpreter since they are not being invoked again.

When a long-running loop is detected at runtime, HotSpot VM requests a compile that starts its execution at the first bytecode of loop instead of starting at the first bytecode in the method.  The resulting generated code takes an interpreter frame as its input and uses that state to begin its execution.  In this way, long-running loops are able to take advantage of compiled code.  The act of the generated code taking an interpreter frame as input to be execution is called On Stack Replace.

How to Detect If OSR Happened?

The diagnostic options -XX:+LogCompilation can emit a structured XML log of compilation related activity during a run of the virtual machine.  Because this is a diagnostic option, to enable it, you need to specify  -XX:+UnlockDiagnosticVMOptions first.  For example, to generate compilation log file, you can specify:

• -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation -XX:LogFile=/<path to log>/logs/<name of log>.log 

From the log file, you can search "osr" to see if any loop in your application has been compiled:

<task_queued compile_id='1' compile_kind='osr' 

  method='java/util/jar/JarFile hasClassPathAttribute ()Z' bytes='275' 

  count='223' backedge_count='57584' iicount='223' osr_bci='157' stamp='0.233' 

  comment='tiered' hot_count='57584'/>
<nmethod compile_id='1' compile_kind='osr' compiler='C2' level='4' 

  entry='0x00002aaaab4d4d40' size='1576' address='0x00002aaaab4d4bd0' 

  relocation_offset='288' insts_offset='368' stub_offset='1072' 

  scopes_data_offset='1120' scopes_pcs_offset='1352' dependencies_offset='1512' 

  nul_chk_table_offset='1520' oops_offset='1096'

  method='java/util/jar/JarFile hasClassPathAttribute ()Z' bytes='275' 

  count='241' backedge_count='62159' iicount='241' stamp='0.237'/>

Every compile is assigned a compile id by the system at the point is enqueued and that's recorded as the compile_id attribute. The above example shows an on stack replacement (OSR) compile where the code is going to be used to replace an already existing activation. It is tagged by the compile_kind attribute as:

•  compile_kind='osr'. 

The 'method' attribute is a string version of the VM name of the method with spaces separating the class, method name signature.

The 'bytes' attribute is number of bytecodes in the method.

'count' is the invocation count as recorded by the method invocation counters. Note that these counters are mainly used for triggering compiles and are not guaranteed to be an accurate reflection of the number of times a method has actually executed. Multiple threads may be updated these counters and sometimes the VM will reset a counter to a lower value to delay retriggering of compiles.

'iicount' is the interpreter invocation count. This is a separate copy of the invocation count which is maintained by the profiling support. Again it's not guaranteed to be accurate since multiple threads may update it but it's never reset so it's reasonably accurate.

The 'backedge_count' attribute is used to detect methods that contain loops and to cause them to get compiled earlier than they would with just an invocation counter. Whenever this counter is incremented by the interpreter it checks it against a threshold, and if it crosses this threshold the interpreter requests a compile of that loop.

'stamp' gives a timestamp for the start time.  Many elements contain time stamps as the end which can be used to order them relative to events in other threads and the measure elapsed time. The time stamp is in seconds since the start of the VM and the start time of the VM is recorded in the hotspot_log element in the time_ms attribute.

References

1. Robust Java benchmarking, Part 1: Issues 
2. Java Performance by Charlie Hunt and Binu John
3. LogCompilation Overview

Performance Tuning with Hotspot VM Option: -XX:+TieredCompilation

Updated (09/26/2014): In the latest JDK 8 release,[20] TieredCompilation is ON by default. Depending on platforms, default ReservedCodeCacheSize is bigger too.  For example, on Linux 64-bit platforms, its default size is 240MB.

Hotspot has two JITs named c1 (i.e., client JIT) and c2 (i.e., server JIT)^[1,5,6]. The client JIT starts fast but provides less optimizations. So, it is used for GUI application. The server JIT starts more slowly but provide very good optimizations. The idea of tiered compilation is to get the best of both compilers, first JITs the code with c1 and then if the code is really hot to recompile it with c2.

The tiered server runtime is enabled with the following Hotspot VM options:

• -server -XX:+TieredCompilation

In this article, we will show you how to tune code cache size (i.e., -XX:ReservedCodeCacheSize^[4]) if tiered compilation is enabled. With the latest Hotspot builds,^[19] we did see some performance improvement if tiered compilation is enabled. However, we also noticed that if your code cache size is not big enough, it may negatively impact your application's performance (i.e., Response Time).

PrintCompilation

Before any performance tuning, you would like to learn how JIT compiler behaves first. You can track the behavior of the JIT compiler with:

• -XX:+PrintCompilation ^[4]

With this VM option, JVM will print message when a method is compiled as follows:

 66    4     n 0     java.lang.System::arraycopy (0 bytes)   (static)
 66    5       3     java.lang.String::hashCode (67 bytes)
 67    1       3     java.lang.String::equals (88 bytes)
 68    2       3     java.lang.String::charAt (33 bytes)
504   69   !         java.util.jar.Attributes::read (410 bytes)
522   70             java.util.jar.Attributes$Name::isValid (45 bytes)
522   69   !         java.util.jar.Attributes::read (410 bytes)   made not entrant
864   69   !         java.util.jar.Attributes::read (410 bytes)   made zombie

Without much ado, we will refer you to read [5] for detailed explanation of each field in the output. For our purpose, we are interested in finding out:

• How many methods have been compiled at the beginning of, say, 10 minutes?

You can find out that by counting the lines in the log file:

• grep :: <logfile> | wc -l

since the compilation lines all have "::" in them. Based on this information, you can estimate what size of code cache size should be for better performance. You can also use this command line option along with instrumenting the benchmark to indicate when it has completed the warm-up period. For example, if you are observing -XX:+PrintCompilation output during the measurement interval, then benchmark has not reached JIT compiler steady state.

ReservedCodeCacheSize

To improve application's performance, you can set the maximum code cache size:

• -XX:ReservedCodeCacheSize=256m

when tiered compilation is enabled for your JVM. ReservedCodeCacheSize (and InitialCodeCacheSize) is an option for the (just-in-time) compiler of the Java Hotspot VM. Basically it sets the maximum size for the compiler's code cache. In 1.5.0_06 and earlier, the default was 1024MB for Solaris 64-bit and amd64. Now the default is 48MB for Solaris 64-bit, amd64, and -server x86.

In our experiments, we have measured the performance of ATG CRM Demo application with different maximum code cache sizes. Here are our findings:

	-TC 64M	+TC 64M	+TC 128M	+TC 256M	+TC 512M	+TC 1024M
Response Time (seconds)	0.236	Failed	0.496	0.227	0.226	0.223

TC: Tiered Compilation

nnnM: Reserved Code Cache Size

Conclusion

As the benchmark results show, you can tune your application's performance by enabling tiered compilation with appropriate setting of code cache size. To estimate how much code cache to reserve, you can connect to the JVM using jconsole and use the memory tab to see how much code cache is filled.  The performance improvement may vary based on your JVM versions and system capabilities.  But, before you do any fine tuning, read [9] first.

Finally, be warned that options that are specified with -XX are not stable and are not recommended for casual use. These options are subject to change without notice^[4].

References

1. Tiered Compilation 
• See also:  HotSpot Glossary of Terms
2. PrintCompilation JVM flag
3. Infrastructure for Tiered Compilation Support
4. Java Hotspot VM Options
5. About PrintCompilation
6. JVM Runtime Compilers: -client or -server 
7. Java Tuning White Paper
8. A Case Study of Using Tiered Compilation in HotSpot 
9. HotSpot VM Performance Tuning Tips (XML and More)
10. Java Performance by Charlie Hunt, Binu John, David Dagastine
11. All other performance tuning articles on XML and More
12. Using JConsole
13. How to Troubleshoot High CPU Usage of Java Applications? (XML and More)
14. HotSpot: Using jstat to Explore the Performance Data Memory (XML and More)
15. Understanding String Table Size in HotSpot (XML and More)
16. HotSpot VM Performance Tuning Tips (XML and More)
17. HotSpot Performance Option — SurvivorRatio (XML and More)
18. All other performance tuning articles on XML and More
19. JDK 7u40 (Java(TM) SE Runtime Environment: build 1.7.0_40-b43)
• Default values for linux-x64:
• ReservedCodeCacheSize: 50331648 {pd product}
• TieredCompilation: false  {pd product}
20. JDK 8: Revisiting ReservedCodeCacheSize and CompileThreshold (Xml and More)
21. HotSpot Virtual Machine Garbage Collection Tuning Guide
22. Garbage-First Garbage Collector Tuning