JRockit: Unable to open temporary file /mnt/hugepages/jrock8SadIG

After system maintenance, our JRockit failed to create the Java virtual machine because it could not acquire the large pages as shown below:

$ bin/java -Xms2560m -Xmx2560m -XlargePages -Xgc:genpar -XlargePages:exitOnFailure -version
[ERROR][osal   ] Unable to open temporary file /mnt/hugepages/jrock8SadIG
[ERROR][memory ] Could not acquire large pages for Java heap.
[ERROR][memory ] Could not setup java heap.
Could not create the Java virtual machine.

In this article, we will discuss how to investigate and resolve this issue.

What to Check?

When using JRockit we have to make a hugetblfs file system available in the directory /mnt/hugepages. Since the message says that it cannot open temporary file /mnt/hugepages/jrock8SadIG, the first thing to check is if that directory was mounted (note that it was mounted before system maintenance).

$ mount -l
nodev on /mnt/hugepages type hugetlbfs (rw)

As shown above, /mnt/hugepages directory was mounted.  The next thing to check is why JRockit cannot open temporary file in that directory.  Could it be privilege?

As described in [1], to make the /mnt/hugepages directory accessible for the oracle user, you need to do:

• chmod -R 777 /mnt/hugepages

It turns out that it was the culprit.  After issuing the above command, JRockit was able to create Java virtual machine.

References

1. Tune the JVM that runs Coherence
2. JRockit: Could not acquire large pages for 2Mbytes code
3. How to Test Large Page Support on Your Linux System
4. Understanding Application Memory Performance - Red Hat

JRockit: Could not acquire large pages for 2Mbytes code

When we tried to enable Large Pages for JRockit, we have seen the following message from the WebLogic Server console output:

• WARN codegc Could not acquire large pages for 2Mbytes code (at 0x2aaab0622000).

In this article, we will show you how to investigate and resolve this issue.

How to Test for Large Pages Support?

Similar to [1], here are the VM options for testing large-pages support for JRockit:

$  bin/java -Xms2560m -Xmx2560m -XlargePages -Xgc:genpar -XlargePages:exitOnFailure -version

When we ran the above command, we have seen the following warning:

[WARN ][codegc ] Could not acquire large pages for 2Mbytes code (at 0x2aaab0622000).
[WARN ][codegc ] Falling back to normal page size.
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64, compiled mode)

Enabling Large Pages Support in Linux Kernel

We have followed the procedure described in [2,5] to enable huge-pages support in the Linux kernel.  One of the requirements is to mount hugepages directory on the hugetlbfs type filesystem^[6] (note that this step is required for JRockit, but not HotSpot).  For example, we have hugepages directory mounted as follows:

$ mount -l
nodev on /mnt/hugepages type hugetlbfs (rw,noexec,nosuid,nodev,sync,uid=59951)

How to Investigate?

When we tried to explicitly disabled large pages for Java code (but not Java heap) , the following VM options ran fine:

$  bin/java -Xms2560m -Xmx2560m -XlargePages -Xgc:genpar -XlargePages:exitOnFailure -XX:+UseLargePagesForHeap -XX:-UseLargePagesForCode -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=false  -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64, compiled mode)


So, the issue is related to Java code reservation in Huge Pages.  After further investigation, we have found the real problem is that: when we mounted hugepages directory, we have chosen the following option:

• noexec
• Do not allow direct execution of any binaries on the mounted filesystem

After we have removed that constraint and rebooted the system, we have finally resolved the issue as shown below:

$  bin/java -Xms2560m -Xmx2560m -XlargePages -Xgc:genpar -XlargePages:exitOnFailure -XX:+UseLargePagesForHeap -XX:+UseLargePagesForCode -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=false -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64, compiled mode)

Here are the settings of newly mounted hugepages directory:

$ mount -l
nodev on /mnt/hugepages type hugetlbfs (rw)

Acknowledgement

This issue was resolved based on the feedbacks from Scott Oats.

References

1. How to Test Large Page Support on Your Linux System?
2. Java SE Tuning Tip: Large Pages on Windows and Linux 
3. Oracle® JRockit Command-Line Reference Release R28
4. Memlock limit too small (one of the requirements for large page support)
5. How to acquire large pages for Java heap
6. Linux / Unix Command: mount
7. Oracle® JRockit Performance Tuning Guide Release R28