Wednesday, August 28, 2013

Auto-Correlating Session IDs in Oracle Application Test Suite (OATS)

Similar to HP LoadRunner, Oracle Application Test Suite (OATS)[1] is an automated performance and test automation product from Oracle for monitoring system behavior and application performance. It's especially useful for Oracle Fusion Application's performance evaluation.

Oracle Open Script (or Oracle Functional Testing) is one of the components in OATS, which is integrated with Oracle Load Testing and Oracle Test Manager. It is also a load testing script generator, which is integrated with Eclipse to support script development and debugging. In the current offering, it only runs on Windows.

Correlation


Correlation of dynamic session values is a major task for load test scripting[2]. When a server in AUT (application under test) exchanges dynamic session values with the browser. OpenScript can auto-correlate dynamic session values—For example session IDs.

What's Session ID?


Session ID is used in session tracking.  Session tracking enables you to track a user's progress over multiple servlets or HTML pages, which, by nature, are stateless. A session is defined as a series of related browser requests that come from the same client during a certain time period. Session tracking ties together a series of browser requests—think of these requests as pages—that may have some meaning as a whole, such as a shopping cart application.

Session ID is a piece of data that is exchanged between the application's web server and the user agent (or browser). It is typically used to identify a specific user logged on to the application for a particular duration of his/her visit (or session).

Session ID is given per Session. It is often destroyed when the user logs off from the application. Next time you visit the same site, you will have a different session ID. The correlation task is to identify these dynamic values and substitute variables for them in the load testing scripts.

As you know, Oracle Fusion Applications maintain a rich set of dynamic session values. Correlation done manually requires in-depth knowledge of the application itself and can also be error prone. Fortunately, most correlations needed for successful playbacks can be done automatically by Oracle Open Script. For example, it auto-correlates Session IDs.

Different Ways of Storing Session IDs


There are multiple ways for a web page to pass session ID to a web server. Session ID can be stored in:
  • Cookie[6]
  • URL
  • HTML page

Storing Session ID in Cookies


Cookie is the text information that application places in the client's hard disk. Browser sends the cookie back to the application to keep the state. On WebLogic Server, use of session cookies is enabled by default and is recommended, but you can disable them by setting cookies-enabled property[3] to false.
If cookie is enabled on the browser, you often find the following entry in the HTTP headers:
  • JSESSIONID=HDe6IhnMFZFtKrVsNi0eUsZ0NWaaIaw_OT2vW7CDpZ8sfz9v4Hqf!-777642468!-553692576;
Note that JSESSIONID is the default session tracking cookie name used by WLS. You can configure WebLogic Server session tracking by defining properties in the WebLogic-specific deployment descriptor,weblogic.xml. For a complete list of session attributes, see session-descriptor[3].

Storing Session ID in URL


Session ID can be sent back to the server as a string appended to URL following a question mark (i.e., "?")
On WLS, you can enable URL rewriting by setting url-rewriting-enabled property, which encodes the session ID into the URL and provides session tracking if cookies are disabled in the browser. However, storing Session ID in URLs is less secure than storing it in cookies[5].

Storing Session ID in HTML Page


Finally, session ID can also be stored in the hidden field of a HTML page and submitted by the Post Command:

  • <input type="hidden" name="sessionID" value="54321abcd">

Most user agents (or browsers) allow you to store information in a HiddenField control, which renders as a standard HTML hidden field. A hidden field does not render visibly in the browser, but you can set its properties just as you can with a standard control. When a page is submitted to the server, the content of a hidden field is sent in the HTTP form collection along with the values of other controls. A hidden field acts as a repository for any page-specific information (including Session ID) that you want to store directly in the page.

References

  1. Oracle Application Testing Suite
  2. OpenScript for Load Testing Script Troubleshooting (Tutorial)
  3. weblogic.xml Deployment Descriptor Elements
  4. Extended Session ID format in WebLogic Server (12.1.1)
    • A server startup flag, -Dweblogic.servlet.useExtendedSessionFormat=true, retains the information that the load-balancing application needs for session stickiness. 
    • The extended session ID format will be part of the URL if URL rewriting is activated, and the startup flag is set to true.
  5. Why is passing the session id as url parameter insecure?
  6. OAM 11g Single Sign-On and OAM 11g Cookies
    • Note that the cookie model is different between 10g and 11g.
  7. OATS: Tie All Processes Together — from OpenScript to Scenario (Xml and More)

Monday, August 12, 2013

How to Investigate: Failed to Bind to Port on Linux

From the server log file (i.e., CRMCommonServer_1.log) of WebLogic, I have found the following messages:

####<Aug 12, 2013 10:40:43 AM PDT> <Emergency> <Security> <myserver> <CRMCommonServer_1> <[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1376329243268> <BEA-090087> <Server failed to bind to the configured Admin port. The port may already be used by another process.>
####<Aug 12, 2013 10:40:43 AM PDT> <Error> <Server> <myserver> <CRMCommonServer_1> <DynamicListenThread[Default]> <<WLS Kernel>> <> <> <1376329243268> <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 10.241.88.31 might be incorrect or another process is using port 9004: java.net.BindException: Address already in use.>

In this article, I will show you how to investigate: 
  • Which process is using port 9004?

Netstat Command on Linux


To investigate failed-to-bind-to -port issue, netstat comes in handy on Linux systems.  netstat command can be used to:
  • Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

In this detective work, we have used the following options:

   -a, --all
       Show both listening and non-listening sockets.  With the --interfaces  option,  show  inter-
       faces that are not marked
   -p, --program
       Show the PID and name of the program to which each socket belongs.

The results are shown below:

$ netstat -ap | grep 9004 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 myserver.us.ora:interserver myserver.oracle.com:9004 ESTABLISHED 12550/oidldapd tcp 0 0 myserver.us.oracle.com:9004 myserver.ora:interserver ESTABLISHED 22328/java


From the output, we know a Java application (i.e., process 22328) is using port 9004. When the first socket is bound to that port, then no other socket could be bound on port 9004 as long as the first socket remains open.  To know which application it is, we check out that process' command line:
  • $ vi /proc/22328/cmdline 
On the command line, we have found the following information:
  • -Dweblogic.Name=AdminServer
Also, BIDomain was mentioned there. So, that process is the AdminServer of BIDomain.

Port 7020


Similarly, we have seen port 7020 was used in another server's log file:
  • <BEA-002606> <Unable to create a server socket for listening on channel "Default". The address 10.241.88.31 might be incorrect or another process is using port 7020: java.net.BindException: Address already in use.>
When you tried:
    # netstat -ap  |grep 7020

    No entries have been returned.   However, if you use:

    # netstat -an  |grep 7020

    You could find one entry:

    tcp        0      0 ::ffff:10.241.88.31:7020    :::*                        LISTEN

    In this case, we need to use the following command line:

    # netstat -ap --numeric-ports |grep 7020
    tcp        0      0 slcag044.us.oracle.com:7020 *:*                         LISTEN      21696/java      

    So, we know process 21696 is using port 7020.  To investigate further, we typed:
    # netstat -ap  |grep  21696
    tcp        0      0 slcag044.us.oracle.:dpserve *:*                         LISTEN      21696/java

    It shows dpserve in the place of 7020.  So, that's why our first search ended up with no entries. Now we know port 7020 was used by the dpserve protocol for service type dpserve[2,3].

    Our Solution


    In our case, we need to re-order our start-up steps (see [4] for another approach). Instead of starting BIDomain first, we need to start it last. To fix our issue, we have done:
    • Shut down BIDomain 
    • Start up CRMDomain 
    • Start up BIDomain

    References

    Saturday, August 10, 2013

    Diagnosing Heap Stress in HotSpot

    Heap stress is characterized as OutOfMemory conditions or frequent Full GCs accounting for a certain percentage of CPU time[6].  To diagnose heap stress, either heap dumps or heap histograms can help.

    In this article, we will discuss the following topics:
    1. Heap histogram vs. heap dump[1]
    2. How to generate heap histogram or heap dump in HotSpot


    Heap Histogram vs. Heap Dump 


    Without much ado, read this companion article for the comparison.  For heap analysis, you can use either jmap or jcmd to do the job[5].  Here we focus only on using jmap.

    $ jdk-hs/bin/jmap -help
    Usage:
        jmap [option] 
            (to connect to running process)
        jmap [option] 
            (to connect to a core file)
        jmap [option] [server_id@]
            (to connect to remote debug server)
    
    where <option> is one of:
        <none>               to print same info as Solaris pmap
        -heap                to print java heap summary
        -histo[:live]        to print histogram of java object heap; if the "live"
                             suboption is specified, only count live objects
        -permstat            to print permanent generation statistics
        -finalizerinfo       to print information on objects awaiting finalization
        -dump:<dump-options> to dump java heap in hprof binary format
                             dump-options:
                               live         dump only live objects; if not specified,
                                            all objects in the heap are dumped.
                               format=b     binary format
                               file=  dump heap to 
                             Example: jmap -dump:live,format=b,file=heap.bin <pid>
        -F                   force. Use with -dump:<dump-options> <pid> or -histo
                             to force a heap dump or histogram when <pid> does not
                             respond. The "live" suboption is not supported
                             in this mode.
        -h | -help           to print this help message
        -J<flag>             to pass <flag> directly to the runtime system
    


    Generating Heap Histogram


    Heap histograms can be obtained by using jmap (note that you need to use jmap from the same JDK installation which is used to run your applications):

    $~/JVMs/jdk-hs/bin/jmap -histo:live 7891 >hs_jmap_7891.txt


     num     #instances         #bytes  class name
    ----------------------------------------------
       1:       2099805      195645632  [C
       2:        347553       49534472  <constMethodKlass>
       3:       2055692       49336608  java.lang.String
       4:        347553       44501600  <methodKlass>
       5:         30089       36612792  <constantPoolKlass>
       6:       1044560       33425920  java.util.HashMap$Entry
       7:         90868       24909264  [B
       8:         30089       23289072  <instanceKlassKlass>
       9:         22323       18194144  <constantPoolCacheKlass>
      10:        177458       15661816  [Ljava.util.HashMap$Entry;
      11:        642260       15414240  javax.management.ObjectName$Property
      12:        159785       15405144  [Ljava.lang.Object;
    

    In the output, it shows the total size and instance count for each class type in the heap.  For example, there are 2099805 instances of character arrays (i..e, [C), which has a total size of 195645632 bytes.  Because the suboption live was specified, only live objects were counted (i.e., a full GC was forced before histogram was collected).

    Generating Heap Dump


    Heap dump is a file containing all the memory contents of a Java application. It can be generated via:

    $ ~/JVMs/jdk-hs/bin/jmap -dump:live,file=/tmp/hs_jmap_dump.hprof 7891
    Dumping heap to /tmp/hs_jmap_dump.hprof ...
    Heap dump file created

    Including the live option in jmap will force a full GC to occur before the heap is dumped so that it contains only live objects.  We recommend taking multiple heap dumps.  For example, 30 minutes and 1 hour into the run.  Then use Eclipse MAT[2,3] to examine the heap dumps.

    Finally, there are other ways to generate a java heap dump:
    • Use jconsole option to obtain a heap dump via HotSpotDiagnosticMXBean at runtime
    • Heap dump will be generated when OutOfMemoryError is thrown by specifying
      • -XX:+HeapDumpOnOutOfMemoryError VM option
    • Use hprof[7]


    References

    Thursday, August 8, 2013

    Default Values of JRockit's VM Options

    Updated (09/19/2014):

    You can also add -XX:+UnlockInternalVMOptions to the command line to access JVM internal flags.

    To find out what the JRockit's default VM options are, you type:
    •  jdk-jr/bin/java -XX >JR_defaults.txt
    Note that -XX is an alias for -Xprintflags in JRockit.

    Default Values


    Here are the output from this JRockit version:
      java version "1.6.0_37"
      Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
      Oracle JRockit(R) (build R28.2.5-20-152429-1.6.0_37-20120927-1915-linux-x86_64, compiled mode)


    Global:
            UnlockDiagnosticVMOptions = false (default, writeable)
                    - Enable processing of flags relating to field diagnostics
            UnlockInternalVMOptions = false (default)
                    - Enable processing of internal, unsupported flags
    Class:
            FailOverToOldVerifier = true (default, writeable)
                    - Fail over to old verifier when split verifier fails
            UseVerifierClassCache = true (default)
                    - Try to cache java.lang.Class lookups for old verifier.
            UseClassGC = true (default)
                    (Alias: -Xnoclassgc)
                    - Allow GC of Java classes
    Threads:
            UseThreadPriorities = false (default)
                    - Use native thread priorities
            DeferThrSuspendLoopCount = 4000 (default, writeable)
                    - Number of iterations in safepoint loop until we try blocking
            SafepointSpinBeforeYield = 2000 (default, writeable)
                    - Number of iterations in safepoint loop until we yield instead
                      of pause (MP only)
            UseCompilerSafepoints = true (default)
                    - Insert safepoint polls in compiled code
            DeferPollingPageLoopCount = -1 (default)
                    - Number of iterations in safepoint loop before arming
                      safepoint poll page
            UseMembarForTransitions = false (default)
                    - Use membar to serialize thread states.
            UseNativeLockProfiling = false (default)
                    - Profile use of internal JVM monitors
            TrustPThreadStackInfo = false (default)
                    - Trust information from pthreads about stack start and size
    JNI:
            CheckJNICalls = false (default)
                    - Verify all arguments to JNI calls
            AbortOnFailedJNICheck = true (default)
                    - Used with CheckJNICalls. If true, abort the JVM upon first
                      JNI parameter error.
            ErrorOnFailedJNICheck = false (default)
                    - Used with CheckJNICalls. If true, any errors will be
                      signalled through a java.lang.Error.
    JDK:
            UseNewHashFunction = false (default)
                    - Use HashMaps new hash function on jdks that does not do so by
                      default
            TreeMapNodeSize = 64 (default)
                    - Size of entry array in each java.util.TreeMap node
            MaxDirectMemorySize = 0 (default)
                    - Maximum total size of NIO direct-buffer allocations
            UseLazyStackTraces = true (default)
                    - Generate stacktraces lazily for thrown Exceptions
            ShowInternalMethodsInStackTrace = false (default, writeable)
                    - Show JVM internal code in java stacktraces.
            ExceptionTraceFilter = (null) (default)
                    - Pattern that limits what exceptions are logged.
    OS:
            ReduceSignalUsage = false (default)
                    (Alias: -Xrs)
                    - Reduce the use of OS signals in Java and/or the VM
            MaxFDLimit = true (default)
                    - Maximize the available number of filedescriptors.
            MaxLargePageSize = 256M (default)
                    - Use value as maximum size for large pages (if possible).
    GC:
            UseLowAddressForHeap = true (default)
                    - Use low 4Gb address space for Java heap if possible.
            UseLargePagesForHeap = false (default)
                    - Attempt to use large page translation for the Java heap.
            ForceLargePagesForHeap = false (default)
                    - Force the use of large page translation for the Java heap.
            CompressedRefs = false (default)
                    (Alias: -XXcompressedRefs)
                    - Use 32-bit java references on 64-bit OS - implies a heap
                      maximum of 4Gb (probably less)
            InitialHeapSize = 0 (default)
                    (Alias: -Xms)
                    - Initial size of Java Object heap
            MaxHeapSize = 0 (default)
                    (Alias: -Xmx)
                    - Maximum size of Java Object heap
            GCTimeRatio = 19 (default)
                    - The ratio of time spent in garbage collection compared to
                      outside of garbage collection.
            GCTimePercentage = 0.000000 (default)
                    - The percentage of time spent in garbage collection of total
                      run time.
            GCTrigger = 0 (default)
                    (Alias: -XXgcTrigger)
                    - The threshold of free heap before a concurrent GC is started
            ForceEarlyOC = true (default)
                    - Force an early OC before old space is empty to avoid
                      promotion failed.
            ForceEarlyOCMaxPercentage = 5.000000 (default)
                    - Maximum percentage of heap that is allowed to have left
                      before doing an early OC.
            ForceYCOnLargeAllocationFailed = false (default)
                    - Force YC on a large allocation failure.
            UseNurseryEvacuation = false (default)
                    - Try to evacuate the nursery when a promotion failed has
                      occured.
            DisableEvacuationToNursery = false (default)
                    - Disallows evacuation to move objects to where the nursery is.
            NurseryPartsLimit = 10000 (default)
                    - The maximum number of nursery parts we will allow before
                      forcing an early OC to fight fragmentation, or 0 for no
                      limit.
            SemiRefPostponedPacketSize = 492 (default)
                    - The number of references in a postponed semiref packet.
            SemiRefPrefetchDistance = 0 (default)
                    - The number of reference packet indexes to prefetch, or 0 for
                      no prefetch.
            FinalHandleParallelThreshold = -1 (default)
                    - The minimum number of final handles needed to process them in
                      parallel.
            FinalHandlePacketSize = 200 (default)
                    - The number of handles in a final handle packet.
            MaximumNurseryPercentage = 95 (default)
                    - Sets the maximum size of the nursery relative to the amount
                      of free heap after the last old collection.
            AllowYCDuringOC = true (default)
                    - Allow young collections during old collections.
            YcAlignAll = false (default)
                    - Align all objects (with regards to YcAlignMaxSpill) during YC
            YcAlignMaxSpill = 40 (default)
                    - Max spill allowed when aligning objects during YC.
            FullSystemGC = false (default)
                    (Alias: -XXfullSystemGC)
                    - Always run full GC (with full compaction) when System.gc() is
                      called
            AllowSystemGC = true (default)
                    (Alias: -XXnoSystemGC)
                    - Run a GC when System.gc() is called
            GcCardTableParts = 1024 (default)
                    - Initial number of parts of the card table array
            GcBalancePrefetchDistance = 4 (default)
                    - Prefetch distance in a GC balance system workpacket
            GcBalancePacketSize = 493 (default)
                    - Packet size of GC balance system workpackets
            NumGenConPrecleaningIterations = 3 (default)
                    - Number of precleaning iterations for gencon.
            AllowEmergencyParSweep = true (default)
                    - Allow the OC to temporarily change concurrent sweep to
                      parallel if needed.
            UseCfsAdaptedYield = false (default)
                    - Use a version of yield adapted for the CFS scheduler. Only
                      for use on CFS.
            TlaWasteLimit = 0 (default)
                    (Alias: -XXlargeObjectLimit)
                    - Internal. Use -XXtlaSize:wasteLimit instead.
            TlaMinSize = 0 (default)
                    (Alias: -XXminBlockSize)
                    - Internal. Use -XXtlaSize:min instead.
            TlaPreferredSize = 0 (default)
                    - Internal. Use -XXtlaSize:preferred instead.
    GC::Compaction:
            UseFullCompaction = false (default)
                    - All compactions will be full compactions. Internal. Use
                      -XXcompaction:full instead.
            InternalCompactionPercentage = -1.000000 (default)
                    - The percentage of the heap to compact for internal
                      compaction. Internal. Use -XXcompaction:internalPercentage
                      instead.
            ExternalCompactionPercentage = -1.000000 (default)
                    - The percentage of the heap to compact for external
                      compaction. Internal. Use -XXcompaction:externalPercentage
                      instead.
            InitialCompactionPercentage = -1.000000 (default)
                    - The initial percentage of the heap to compact, for both
                      internal and external compaction. Internal. Use
                      -XXcompaction:initialPercentage instead.
            UseCompaction = true (default)
                    - Use compaction to reduce fragmentation. Internal. Use
                      -XXcompaction:enable instead.
            UseAbortableCompaction = false (default)
                    - The compactions should be possible to abort. Internal. Use
                      -XXcompaction:abortable instead.
            NumCompactionHeapParts = 4096 (default)
                    (Alias: -XXheapParts)
                    - The number of heap parts in compaction heuristics. Internal.
                      Use -XXcompaction:heapParts instead.
            InitialExternalReservedHeap = 4M (default)
                    - The initial size of the memory reserved by external
                      compaction. Internal. Use
                      -XXcompaction:initialExternalReservedHeap instead.
            UseFixedExternalReservedHeap = false (default)
                    - The size of the memory reserved by external compaction is
                      fixed. Internal. Use
                      -XXcompaction:externalReservedHeapIsFixed instead.
            MaxCompactionReferences = 0 (default)
                    (Alias: -XXcompactSetLimit)
                    - The maximum number of references to store in compaction
                      before skipping. Internal. Use -XXcompaction:maxReferences
                      instead.
            MaxCompactionReferencesPerObject = 0 (default)
                    (Alias: -XXcompactSetLimitPerObject)
                    - The maximum number of references to store per object in
                      compaction. Internal. Use
                      -XXcompaction:maxReferencesPerObject instead.
            InternalCompactionParts = -1 (default)
                    (Alias: -XXinternalCompactRatio)
                    - Deprecated. Use -XXcompaction:internalPercentage instead.
            ExternalCompactionParts = -1 (default)
                    (Alias: -XXexternalCompactRatio)
                    - Deprecated. Use -XXcompaction:externalPercentage instead.
    Object allocation:
            UseAllocPrefetch = true (default)
                    - Use prefetch on object allocation
            RedoAllocPrefetch = true (default)
                    - Do prefetch on object allocation from start of the allocated
                      object
            AllocPrefetchLineLength = -1 (default)
                    - Line length for allocation prefetch
            AllocPrefetchDistance = -1 (default)
                    - Distance for allocation prefetch
            AllocChunkSize = -1 (default)
                    - Size of chunks to clear/prefetch
    Javalock:
            UseLockProfiling = false (default)
                    - Enable Java lock profiling.
            ThinLockContendedSpinCount = -1 (default)
                    - Number of spins between each poll when acquiring a thin lock
            ThinLockContendedPollCount = -1 (default)
                    - Number of polls between each short nap when acquiring a thin
                      lock
            ThinLockConvertToFatThreshold = -1 (default)
                    - Number of of short naps before converting thin lock to fat
            FatLockContendedSpinCount = -1 (default)
                    - Number of spins between each poll when acquiring a fat lock
            FatLockContendedPollCount = -1 (default)
                    - Number of polls between each short nap when acquiring a fat
                      lock
            MonitorContendedSpinCount = -1 (default)
                    - Number of spins between each poll when acquiring a monitor
                      lock
            MonitorContendedPollCount = -1 (default)
                    - Number of polls between each short nap when acquiring a
                      monitor lock
            UseFatLockDeflation = true (default)
                    - Try to deflate fat locks to thin
            FatLockDeflationThreshold = 50 (default)
                    - Number of uncontended entries on lock before deflation occurs
            UseLockQueueLength = true (default)
                    - Make threads go to sleep if contention exceeds # cpus
            UseFatSpin = true (default)
                    (Alias: -XXdisableFatSpin)
                    - Should we spin-try then acquiring a fat lock
            UseAdaptiveFatSpin = false (default)
                    - Should we use adaptive spinning acquiring a fat lock
            UseThreadContentionMonitoring = true (default)
                    - Allow thread contention monitoring
    JavaLock::LazyUnlocking:
            UseLazyUnlocking = true (default)
                    (Alias: -XXlazyUnlocking)
                    - Enable lazy unlocking
            UseLazyUnlockingInJIT = true (default)
                    - Use lazy locks in JIT code
            UseLazyUnlockingClassBan = true (default)
                    - Use class banning
            UseLazyUnlockingTransferClassBan = true (default)
                    - Use transfer class banning
    JFR:
            FlightRecorder = true (default)
                    - Enable flightrecorder
            FlightRecorderOptions = (null) (default)
                    - Flight recorder arguments
            StartFlightRecording = (null) (default)
                    - Start a Flight recording with args. Equivalent to using
                      "start_flightrecording".
    Code memory:
            CodeBlockAbsorbtionSize = 32 (default)
                    - Maximum extra size allowed for fitting code memory chunks
            FreeEmptyCodeBlocks = true (default)
                    - Free unused code memory
            UseLargePagesForCode = false (default)
                    - Attempt to use large page translation compiled code.
            MaxCodeMemory = 0 (default)
                    - Maximum amount of memory used for generated code
            ReserveCodeMemory = true (default)
                    - Reserve all memory for code at startup
            UseCodeGC = true (default)
                    - Allow GC of discarded compiled code
            CodeGCThreshold = 0 (default)
                    - Released byte threshold for initiating a code GC
            CodeGCReclaimThreshold = 0 (default)
                    - Released byte threshold before compiler attempts to reclaim
                      unused code space
            CodeGCUseReclaim = true (default)
                    - Should Code GC attempt reclamation of unused code memory
            CodeGCTaskInterval = 5 (default)
                    - Interval in secs between background scans for unused code
                      blocks
    Compiler broker:
            MaxOptQueueLength = 0 (default)
                    - Maximum allowed optimization queue length before JIT thread
                      helps generate code
            OptThreads = 1 (default)
                    - Number of background optimization threads
            JITThreads = 1 (default)
                    - Number of background JIT threads
            JITThreadPrio = 5 (default)
                    - Priority of background JIT threads
            OptThreadPrio = 5 (default)
                    - Priority of background optimization threads
            DisableOptsAfter = -1 (default)
                    (Alias: -Xnoopt)
                    - Disable optimizations after n seconds
    Compiler:
            PreOpt = false (default)
                    - Optimize all code on jit (first generation)
            UseCallProfiling = false (default)
                    - Use call profiling on unoptimized code
            StrictFP = false (default)
                    (Alias: -Xstrictfp)
                    - Force strict FP for all methods
            CheckStacks = false (default)
                    (Alias: -Xcheckedstacks)
                    - Do explicit checks for stack overflow
            DevirtualizeAlways = false (default)
                    - Forces devirtualization in jitted code
            UseStringCache = false (default)
                    - Cache common arrays used in String constructor
            MethodCodeAlignment = 32 (default)
                    - Byte alignment for start of method code
            UseInlineObjectAlloc = true (default)
                    - Generate inlined object allocation code.
            UseSafeTimer = false (default)
                    - Use fast, signal based timer for System.currentTimeMillis.
            UseOldLockMatching = false (default)
                    - Compatibility mode lock matching
    JVMTI:
            JavaDebug = false (default)
                    (Alias: -Xdebug)
                    - Enable java debugging
    Management:
            DisableAttachMechanism = false (default)
                    - Disable mechanism that allows tools to attach to this VM
            CrashOnOutOfMemoryError = false (default, writeable)
                    - Crash JVM process on OutOfMemory
            ExitOnOutOfMemoryError = false (default, writeable)
                    - Terminate JVM process on OutOfMemory
            ExitOnOutOfMemoryErrorExitCode = 51 (default, writeable)
                    - Exit code for termination of  JVM process on OutOfMemory
            HeapDiagnosticsOnOutOfMemoryError = false (default, writeable)
                    - Print Java heap diagnostics on OutOfMemory
            HeapDiagnosticsPath = (null) (default, writeable)
                    - When HeapDiagnosticsOnOutOfMemoryError is on, the path
                      (filename or directory) of the dump file (defaults to
                      jrockit_.oomdiag in the working directory)
            HeapDumpOnOutOfMemoryError = false (default, writeable)
                    - Dump Java heap to a hprof binary format file on OutOfMemory
            HeapDumpOnCtrlBreak = false (default)
                    - Dump heap to file in Ctrl-Break handler
            HeapDumpPath = (null) (default, writeable)
                    - When HeapDumpOnOutOfMemoryError is on, the path (filename or
                      directory) of the dump file (defaults to jrockit_.hprof
                      in the working directory)
            SegmentedHeapDumpThreshold = 2G (default, writeable)
                    - Generate a segmented heap dump (JAVA PROFILE 1.0.2 format)
                      when the heap usage is larger than this
            HeapDumpSegmentSize = 1G (default)
                    - Approximate segment size when generating a segmented heap
                      dump
            StartMemleakOnPort = 0 (default, writeable)
                    (Alias: -XXmemleak)
                    - Listen for memleak connections on this port (0 for default)
            FlightRecordingDumpOnUnhandledException = false (default, writeable)
                    - Generate a Flight Recording dump when a thread is terminated
                      due to an unhandled exception
            FlightRecordingDumpPath = (null) (default, writeable)
                    - When FlightRecordingDumpOnUnhandledException is on, the path
                      (filename or directory) of the dump file (defaults to
                      jrockit__.jfr in the working directory)
    Runtime:
            AbortOnCrash = false (default, writeable)
                    - Abort the JVM in case of an crash.
            DumpOnCrash = true (default, writeable)
                    - Generate a dump of the JVM state in case of a crash.
            CoreOnCrash = true (default, writeable)
                    - Generate a core dump file of the JVM state in case of a
                      crash.
            WaitOnCrash = false (default, writeable)
                    - Wait for user debugger attach in case of a crash.
            AbortOnAssert = true (default, writeable)
                    - Abort JVM on assertion
            CrashOnAssert = false (default, writeable)
                    - Crash dump the JVM on assertion
            WaitOnAssert = false (default, writeable)
                    - Spin and wait JVM on assertion
            NumaMemoryPolicy = (null) (default)
                    - Numa memory policy (interleave, preferredlocal, strictlocal)
            BindToNumaNodes = (null) (default)
                    - Bind process to Numa nodes
            BindToCPUs = (null) (default)
                    - Bind process to CPUs
            UseFastTime = true (default)
                    - Force/disable usage of hardware platform support for fast
                      time
            UseJNIPinning = true (default)
                    - Use pinning for Objects in call to GetPrimitiveArrayCritical
                      etc
    

    See Also

    Wednesday, August 7, 2013

    Diagnosing OutOfMemoryError or Memory Leaks in JRockit

    When you run into OutOfMemoryError or other memory-leak issues, generating a heap histogram or heap dump can help you diagnose the memory-bloating issues.

    In [1], it lists the following Java Heap related problems:
    • Exceeding max heap 
      • The heap is full and cannot fit a new object
    • Large allocation
      • The new object is too large for the contiguous free space
    • Native exhaustion
      • There is not enough native heap for the requested object[10]
    • GC Starvation
      • The heap is almost full and causing frequent garbage collection[9]
    • Optimization
      • Heap utilization is higher than expected for the current number of users
    In this article, we will discuss the following topics:
    1. Heap histogram vs. heap dump (see also [8])
    2. How to generate heap histogram or heap dump in JRockit
    3. JVM options that are useful for heap analysis

    Heap Histogram vs. Heap Dump


    A heap dump is a snapshot of all the objects in the Java Virtual Machine (JVM) heap at a certain point in time. The JVM software allocates memory for objects from the heap for all class instances and arrays. The garbage collector reclaims the heap memory when an object is no longer needed and there are no references to the object. By examining the heap you can locate where objects are created and find the references to those objects in the source.  However, dumping of Java heap is time-consuming and lengthy in size.

    On the other hand, heap histogram gives a very good summary of heap objects used in the application without doing a full heap dump. It can help you quickly narrow down a memory leak. This information can be obtained in several way:
    • Attach a running process using the command jrcmd.
    • Generate from a core file or heap dump
    Note that we refer to heap histogram, heap summary, or heap diagnostics interchangeably in this article.

    Generating Heap Histogram


    A heap histogram can be obtained from a running process using the command:
    • jrcmd 20488 heap_diagnostics

    
    
    --------- Detailed Heap Statistics: ---------
    30.0% 65027k   672176     -1k [C
    10.2% 22119k   943754     -2k java/lang/String
    10.0% 21592k   183036     -3k [Ljava/lang/Object;
     4.7% 10185k   434587     +0k java/util/HashMap$Entry
     4.7% 10114k    27029  -1254k [B
     4.5% 9783k   111539     +0k [Ljava/util/HashMap$Entry;
     1.9% 4075k    34777     +0k java/lang/Class
     1.9% 4058k    86590     +0k java/util/HashMap
     1.6% 3448k   147156     +0k javax/management/ObjectName$Property
     1.2% 2593k    82994     +0k java/util/LinkedHashMap$Entry
     1.1% 2398k    76765     +0k java/util/concurrent/ConcurrentHashMap$Segment
     1.0% 2215k     9311     +0k [I
     0.9% 1975k    18469     +0k [J
    
    
    In the output, there is a "Detailed Heap Statistics" section, which shows the total size and instance count for each class type in the heap:
    • The first column corresponds to the Class object type contribution to the Java Heap footprint in %
    • The second column correponds to the Class object type memory footprint in K
    • The third column correponds to the # of Class instances of a particular type
    • The fourth column correponds to the delta - / + memory footprint of a particular type
    As you can see from the above snapshot, the biggest data type is [C (i.e., character array) and java.lang.String. In order to see which data types are leaking, you will probably need to generate several snapshots, which you might be able to observe a trend that can lead to further analysis.

    Generating Heap Dump


    Heap dump is a file containing all the memory contents of a Java application. It can be generated via 
    • jrcmd 20488 hprofdump
      • Wrote dump to /.../appmgr/APPTOP/instance/debug/jrockit_20488.hprof
    Then you can use various tools to load that file and look at various things in the heap: how much each kind of object is using, what things are holding onto the most amount of memory, and so on.  The size of heap dump file is proportional to the size of Java Heap and can be large.

    Three of the most common tools are:

    • jhat
      • This is the original heap analyzer tool, which reads the heap dump and runs a small HTTP server that lets you look at the dump through a series of web page links.
    • VisualVM [3]
    • MAT [4,5]

    Heap-Related JVM Options


    When your JVM runs into OutOfMemoryError, you can set:
    • -XX:+HeapDumpOnOutOfMemoryError
    to get a heap dump after the heap is big and bloated just before the JVM dies.  Also, you can provide the following flags:
    • -XX:HeapDumpPath=<path to the destination>
    • -XX:+ExitOnOutOfMemoryError
    Similarly, you can get a heap histogram instead of a full heap dump using[7]:
    • -XX:+HeapDiagnosticsOnOutOfMemoryError 
    • -XX:HeapDiagnosticsPath=<path to the destination>

    hotspot.log from the HotSpot's Fastdebug Build

    I was asked to use a fastbebug VM build to run my benchmark.  I have noticed that a new file was generated in our WebLogic domain folder:
    • hotspot.log[1]
    To track which methods are getting inlined, you can use DEBUG VM with the following option:
    • -XX:PrintInlining
    Note that there are some interesting options available only in DEBUG build[4].


    What Is a Fastdebug Build?


    There are only scarce information for fastdebug VM build on the internet.  Here are some description of it (but, it may be obsolete)[2]:

    • Hotspot, fastdebug builds, plug&play shared libraries (removal of java_g)
      These may seem unrelated, but when the Hotspot team started building "fastdebug" VM libraries (using -g -O and including assertions), that could just be plugged into any JDK image, that was a game changer. It became possible to plug&play native components when building this way, instead of the java_g builds where all the components had to be built the same way, an all or nothing run environment that was horribly slow and limiting. So we tried to create a single build flow, with just variations on the builds (I sometimes called them "build flavors" of product, debug, and fastdebug). Of course, at the same time, the machines got faster, and perhaps now using a complete debug build of a jdk make more sense? In any case, that fastdebug event influenced matters. We do want different build flavors, but we don't want separate make logic for them.


    For a fastdebug build, I've downloaded the following zip file:
    • linux_x64_2.6-fastdebug.zip

    hotspot.log


    The contents of hotspot.log look like below:

    <?xml version='1.0' encoding='UTF-8'?>
    <hotspot_log version='160 1' process='7063' time_ms='1375898373959'>
    <vm_version>
      <name>Java HotSpot(TM) 64-Bit Server VM</name>
      <release>25.0-b43-fastdebug</release>
      <info>
       Java HotSpot(TM) 64-Bit Server VM (25.0-b43-fastdebug) for linux-amd64 JRE ...
      </info>
    </vm_version>
    <vm_arguments>
      <args>
       ...
      </args>
      <command>weblogic.Server</command>
      <launcher>SUN_STANDARD</launcher>
      <properties>
       java.vm.specification.name=Java Virtual Machine Specification
       ...
      </properties>
    </vm_arguments>

    Obviously, there are tons of other information in the file that can help you to identify any VM issues.

    References

    Friday, August 2, 2013

    Linux: "File size limit exceeded" or "Too many open files in system"

    When running my benchmark, I have run into the following exception:
    • java.io.FileNotFoundException

    Too many open files in system


    The above exception is caused by:
    • Too many open files in system
    as found in the MyServer_1-diagnostic.log

    [2013-06-27T15:40:03.611-07:00] [CRMCommonServer_1] [ERROR] [] [oracle.security.audit.ajl.loader.AuditLoaderManager] [tid: AuditLoaderRunner] [ecid: 0000Jy7mm0L7y0I_IpG7yf1Hn9FC0001na,0] IAU:IAU-5046: Stopping AuditLoader, caught exception: oracle.security.audit.AuditException: java.io.FileNotFoundException: /slot/.../MyDomain/servers/myserver_1/logs/iau/state/auditloader.state (Too many open files in system)[[
            at oracle.security.audit.service.AuditLoaderManager.readMessages(AuditLoaderManager.java:276)
            at oracle.security.audit.service.AuditLoaderManager$Runner.run(AuditLoaderManager.java:335)
    Caused by: java.io.FileNotFoundException: /slot/.../MyDomain/servers/MyServer_1/logs/iau/state/auditloader.state (Too many open files in system)
            at java.io.FileOutputStream.open(Native Method)
            at java.io.FileOutputStream.(FileOutputStream.java:194)
            at java.io.FileOutputStream.(FileOutputStream.java:145)
            at java.io.FileWriter.(FileWriter.java:73)
            at oracle.security.audit.ajl.loader.AuditLoader.saveState(AuditLoader.java:213)
            at oracle.security.audit.service.AuditLoaderManager.readMessages(AuditLoaderManager.java:262)
            ... 1 more

    User Level File Descriptor Limits



    To view current open file limit for the current Linux user, run command:

    $ulimit -n
    8192

    To set it to a new value for this running session, which takes effect immediately, run command:

    $ ulimit -n 16384


    Alternatively, if you want the changes to survive reboot, do the following:
    1. Exit all shell sessions for the user you want to change limits on.
    2. As root, edit the file /etc/security/limits.conf and add these two lines toward the end:
      • user1 soft nofile 16384
        user1 hard nofile 16384
    The two lines above changes the max number of file handles - nofile - to new settings.
    • Save the file.
    • Login as the user1 again. The new changes will be in effect.

    System-wide File Descriptors Limits


    On Linux, there is also a system-wide configuration parameter named:
    • fs.file_max
    Use the following command to display maximum number of open file descriptors allowed on the system:

    $cat /proc/sys/fs/file-max
    100000

    Many application such as Oracle database or WebLogic server needs this setting quite higher. So you can increase the maximum number of open files by setting a new value in kernel variable /proc/sys/fs/file-max as follows (login as the root):

    # sysctl -w fs.file-max=262144

    Above command forces the limit to 262144 files. You need to edit /etc/sysctl.conf file and put following line so that after reboot the setting will remain as it is:

    # vi /etc/sysctl.conf

    Append a configuration directive as follows:

    fs.file-max = 262144

    Save and close the file. Users need to log out and log back in again to changes take effect or just type the following command:

    # sysctl -p

    Verify your settings with command:

    # cat /proc/sys/fs/file-max

    OR

    # sysctl fs.file-max

    Final Words


    Note that commands used in this article are good for the following Linux release:

    $ cat /etc/*-release
    Enterprise Linux Enterprise Linux Server release 5.8 (Carthage)
    Oracle Linux Server release 5.8
    Red Hat Enterprise Linux Server release 5.8 (Tikanga)


    References

    1. Need to “calculate” optimum ulimit and fs.file-max values according to my own server needs
    2. Verifying Kernel Parameters
    3. Linux Increase The Maximum Number Of Open Files / File Descriptors (FD)