Monday, December 30, 2013

JRockit: What's the Total Memory Footprint of a Java Process?

In [1], we have shown a case of performance tuning by sizing JRockit's Thread Local Area (TLA). For both test runs—TLA default and TLA tuned, they have been given the same heap size (i.e, 2g). However, in the conclusion, I have said:
Better performance is achieved by reducing pause time % in GC andTotal CPU % at the expense of total memory footprint (-1.6%).
In this article, we will clarify what the following phrase:
at the expense of total memory footprint (-1.6%)

Java Heap vs Native Memory

As described in [2], total memory footprint of a Java process includes not only Java Heap, but also Native Memory. For example, you can use print_memusage to analyze the memory (including native memory) allocated by a Java process:

> ./jrcmd 26413 print_memusage >JR_print_memusage.txt
>cat JR_print_memusage.txt

Total mapped                  4897980KB           (reserved=1384484KB)
-              Java heap      2097152KB           (reserved=0KB)
-              GC tables        70156KB
-          Thread stacks        48324KB           (#threads=141)
-          Compiled code      1048576KB           (used=41160KB)
-               Internal         1480KB
-                     OS       401856KB
-                  Other       760932KB
-            Classblocks        27136KB           (malloced=26821KB #58548)
-        Java class data       441344KB           (malloced=439083KB #271063 in 58548 classes)
- Native memory tracking         1024KB           (malloced=242KB #10)

Native Memory

Internal JVM memory management is, to a large extent, kept off the Java heap and allocated natively in the operating system, through system calls like malloc. This non-heap system memory allocated by the JVM is referred to as native memory.

For JRockit, you can constraint the amount of memory allocated to the Java heap, but not the native memory, in a Java process.[3] As shown above, process 26413 has been allocated 4897980KB (i.e., VSZ) in the virtual address space. However, this VSZ value is not very useful. What counts is the one reported by Resident Set Size (RSS; or physically resident memory). To find out the RSS of a Java process, you can do:

>ps -o pid,uid,state,rss,vsz,minflt,majflt,args -p 26413 >ps_26413.tmp
>cat ps_26413.tmp

26413 60000 S 3295216 4889792 15194852 1 /scratch/user1/JVMs/jdk-jr/bin/java ...

For the process 26413, its total memory footprint (or RSS) is 3295216 KB and this is the KPI we have quoted in our benchmark comparison.


  1. JRockit: A Case Study of Thread Local Area (TLA) Tuning
  2. Why is my JVM process larger than max heap size?
  3. How to Debug Native OutOfMemory in JRockit (Xml and More)

Sunday, December 29, 2013

JRockit: Analyzing GC With JRockit Verbose Output (-Xverbose:memdbg)

Before any JRockit performance tuning, you need to assess the sizes of the objects allocated by your application.[4] One way to access live data size is to view object allocation statistics from the GC log file.

In this article, we will show you how to access object allocation statistics by enabling the JRockit verbose output:
  • -Xverbose:memdbg

Verbose memdbg Log Module[1]

The -Xverbose:memdbg log module is an alias that starts several memory-related log modules to provide a complete picture of the memory management. The memdbg log module sets the memory module to the debug level. This information can also be obtained by setting:
  • -Xverbose:memory=debug

The overhead of the verbose memdbg output is low, which means it can be used in production environments. For example, after enabling -Xverbose:memdb, the overhead has caused the Average Response Time (ART) to deteriorate by 2.25% using our benchmark.

Understanding Verbose Output

After setting the memory module to the debug level, JRockit provides us tons of information. To understand its formats, read [1]. Here we will focus only on OC's (old collections). For example, here is a typical output from the memory log module at the debug level:

[DEBUG][memory ][06310] [OC#41] GC reason: Allocation request failed (1048592 bytes).
[DEBUG][memory ][06310] [OC#41] 2607.371: OC started.
[INFO ][alloc  ][06310] [OC#41] Pending requests at 'Before OC' - Total: 1, TLAs: 0(approx 0 bytes), objects: 1 (1048592 bytes). Max age: 0.
[INFO ][compact][06310] [OC#41] Compaction reason: Normal.
[INFO ][compact][06310] [OC#41] Compacting 128 of 4096 parts at index 384. Compaction type is internal. Exceptional: No.
[INFO ][compact][06310] [OC#41] Compaction area start: 0x8c000000, end: 0x90000000.
Timeout: 411.471 ms.
[INFO ][compact][06310] [OC#41] Compactset limit (per thread): 316839 (dynamic), not using matrixes.
[DEBUG][memory ][06310] [OC#41] Starting parallel marking phase.
[DEBUG][memory ][06310] [OC#41] SemiRef phase WeakJNIHandles run in single threadedmode.
[DEBUG][memory ][06310] [OC#41] SemiRef phase ClassConstraints run in single threaded mode.
[DEBUG][memory ][06310] [OC#41] SemiRef phase FinalMemleak run in single threaded mode.
[DEBUG][memory ][06310] [OC#41] Adding 48 temporary work packets to permanent pool,now 811 packets.
[DEBUG][memory ] [OC#41] Total mark time: 307.204 ms.
[DEBUG][memory ] [OC#41] Ending marking phase.
[DEBUG][memory ] [OC#41] Starting parallel sweeping phase.
[INFO ][nursery] [OC#41] Setting keepAreaMarkers[0] to 0xc9872bc0.
[INFO ][nursery] [OC#41] Setting keepAreaMarkers[1] to 0xcc97f7a0.
[INFO ][nursery] [OC#41] Setting keepAreaMarkers[2] to 0xd20f6088.
[INFO ][nursery] [OC#41] Next keeparea will start at 0xcc97f7a0 and end at 0xd20f6088.
[INFO ][nursery] [OC#41] Nursery size increased from 0KB to 120414KB. Nursery list consists of 2760 parts.
[INFO ][nursery] [OC#41] Average part size: 44KB. Contraharmonic mean (CHM):946KB. CHM per part: 0KB. Normalized CHM: 0.007856.
[DEBUG][memory ][ [OC#41] Total sweep time: 159.954 ms.
[DEBUG][memory ] [OC#41] Ending sweeping phase.
[INFO ][nursery] [OC#41] Nursery size remains at 120414KB. Nursery list consists of 2760 parts.
[INFO ][nursery] [OC#41] Average part size: 44KB. Contraharmonic mean (CHM):946KB. CHM per part: 0KB. Normalized CHM: 0.007856.
[INFO ][alloc  ] [OC#41] Satisfied 0 object and 0 tla allocations. Pending requests went from 1 to 1.
[INFO ][compact] [OC#41] Average compact time ratio (move phase/total time):0.655049.
[INFO ][compact] [OC#41] Compaction time, total: 158.705 ms (target 411.471 ms).
[INFO ][compact] [OC#41] Compaction moved 1295486 objects and left 349 objects. Total moved size 64028976B.
[INFO ][compact] [OC#41] Compaction added 3047752B of free memory in 2 parts.
[INFO ][compact] [OC#41] Compaction time, move phase: 58.023 ms (target 205.735 ms).
[INFO ][compact] [OC#41] Compaction time, update phase: 100.669 ms (target 205.735 ms).
[INFO ][compact] [OC#41] Found 3350566 references. Internal: 1807732  External: 1542834.
[INFO ][compact] [OC#41] Updated 3339159 references. Internal: 1807048  External: 1532111.
[INFO ][alloc  ] [OC#41] Pending requests at 'After OC' - Total: 1, TLAs: 0 (approx 0 bytes), objects: 1 (1048592 bytes). Max age: 1.
[DEBUG][memory ] [OC#41] Page faults before OC: 1, page faults after OC: 1, pages in heap: 524288.
[DEBUG][memory ] [OC#41] Nursery size after OC: 120414KB. (Free: 120371KB Parts: 2760)
[INFO ][memory ] [OC#41] 2607.371-2607.928: OC 1890153KB->1756549KB (2097152KB), 0.558 s, sum of pauses 472.469 ms, longest pause 472.469 ms.

Note that we have used the following JRockit options:
  • -Xverbose:memdbg -Xverbose:gc -Xverbosedecorations=level,module,timestamp,millis,pid

GC Reason

Old collections can be triggered by different events. For example, in the above example, it shows the reason for OC#41 is:
  • Allocation request failed
There are other events can trigger an OC. For example, it can be:
  • GC reason: Artificial, description: Print Heap Summary
  • GC reason: Artificial, description: Print Heap Diagnostics

From our GC log file, we have found the following OC statistics (total: 235)
  • GC reason: Allocation request failed (total: 219)
  • GC reason: Artificial, description: Print Heap Summary (total: 8)
  • GC reason: Artificial, description: Print Heap Diagnostics (total: 8)
Among all events, you can count those artificial heap printing events as the overhead of enabling:
  • -Xverbose:memdbg


  1. Understanding Verbose Output
  2. JRockit: A Case Study of Thread Local Area (TLA) Tuning (Xml and More)
  3. JRockit: Thread Local Area Size and Large Objects (Xml and More)
  4. Analyzing the Performance Impact of Memory Utilization and Garbage Collection

Saturday, December 28, 2013

JRockit: A Case Study of Thread Local Area (TLA) Tuning

Increasing the Thread Local Area (TLA) size is beneficial for multi-threaded applications where each thread allocates a lot of objects. Increasing the TLA size is also beneficial when the average size of the allocated objects is large, as this allows larger objects to be allocated in the TLAs.

In this article, we will present a case of TLA tuning using an application with the following characteristics:
  • A multi-threaded application
  • Allocating a lot of objects
  • Allocating large objects 

TLA Basics

Without much ado, see [1] for the differences between JRockit's large-object implementations between pre- and post-R28.   The version of JRockit used in this article is R28.2.5.  For this version, TLA sizes can be tuned using the following option:
  • -XXtlaSize:min=size,preferred=size,wasteLimit=size

Chaning Default TLA Sizes

For our Linux system, the default TLA settings are:
  • min=2k
  • preferred=16k
  • wasteLimit=2k
To tune for better performance, read [2,4] for advice.  For example, one recommendation offered by Oracle is:
  • Setting  -XXtlaSize:wasteLimit to the same value as -XXtlaSize:min.
After trials and errors, we have found the following TLA settings to be better than the defaults:
  • -XXtlaSize:min=8k,preferred=512k,wasteLimit=8k

Performance Comparison: Default vs. TLA Tuned

Looking at different KPIs, we have found the application's performance improved by 5.9% in Average Response Time (ART) and 3.32% in 90% Response Time.  Better performance is achieved by reducing pause time % in GC and Total CPU % at the expense of total memory footprint (-1.6%).


As the benchmark results show, you can tune your application's performance by tuning TLA sizes. The performance improvement may vary based on your JRockit versions and system capabilities. But, before you do any fine tuning, read [2, 4] first.

Finally, be warned that options that are specified with -XX are not stable and are not recommended for casual use. These options are subject to change without notice.[3]


Friday, December 27, 2013

JRockit: Thread Local Area Size and Large Objects

Large objects can be a problem for an application. If your Java application allocates a lot of objects, especially large objects, it helps to play around with various Thread Local Area (TLA) settings.[1] For a case study, read [11].

If you try to find the default settings of TLA by using:
  • -XX (an alias for -Xprintflags)
it prints out the following information in JRockit R28.2.5:
        TlaWasteLimit = 0 (default)
                (Alias: -XXlargeObjectLimit)
                - Internal. Use -XXtlaSize:wasteLimit instead.
        TlaMinSize = 0 (default)
                (Alias: -XXminBlockSize)
                - Internal. Use -XXtlaSize:min instead.
        TlaPreferredSize = 0 (default)
                - Internal. Use -XXtlaSize:preferred instead.

Hmm. That is not very helpful. In this article, we will discuss:
  • Thread local area size and large objects
  • How to find out the default TLA settings?

Thread Local Area Size and Large Objects[1-4]

The thread local area (TLA) is a chunk of free space reserved on the heap or the nursery and given to a thread for its exclusive use. A thread can allocate small objects in its own TLA without synchronizing with other threads. When the TLA gets full the thread simply requests a new TLA. The objects allocated in a TLA are accessible to all Java threads and are not considered “thread local” in any way after they have been allocated.

Increasing the TLA size is beneficial for multi-threaded applications where each thread allocates a lot of objects. Increasing the TLA size is also beneficial when the average size of the allocated objects is large, as this allows larger objects to be allocated in the TLAs. Increasing the TLA size too much may however cause more fragmentation and more frequent garbage collections. Before any JRockit performance tuning, you need to assess the sizes of the objects allocated by your application. One way to access live data size is to view object allocation statistics. There are multiple ways of achieving that:
  • You can create a JRockit Flight Recorder recording and view object allocation statistics in the JRockit Flight Recorder[7]
  • You can use the following JVM option:
    • -Xverbose:memdbg Xverbose:gc -Xverbosedecorations=level,module,timestamp,millis,pid

Large Object: Pre-R28 vs. Post-R28[1]

In JRockit versions earlier than R28, large objects were allocated immediately on the heap and never in a TLA. A flag called –XXlargeObjectLimit was provided to tell JRockit the minimum number of bytes an object should be of in order to be treated as "large". The default was 2 KB.

JRockit post R28 uses a waste limit for TLA space instead. This constrains the amount of TLA space that can be thrown away for each TLA when large objects are allocated and is a more flexible solution.

The R28 allocation algorithm now works like this—JRockit tries to allocate every object regardless of its size in the current TLA. If it doesn't fit and the waste limit is less than the space left in the TLA, the object
goes directly on the heap. Otherwise, JRockit will "waste" the rest of this TLA and try to allocate the object in a new TLA or directly on the heap, depending on the size of the object.

The TLA sizes are set using the following option:
  • -XXtlaSize:min=size,preferred=size,wasteLimit=size[8]
    • min
      • Sets the minimum TLA size. 
    • preferred
      • Sets a preferred TLA size.  This means that TLAs will be the preferred size whenever possible, but can be as small as the min size.
    • wasteLimit
      • Sets the waste limit for TLAs. This is the maximum amount of free memory that a TLA is allowed to have when a thread requires a new TLA.
The following relation is true for the TLA size parameters:
  • -XXtlaSize:wasteLimit <= -XXtlaSize:min <= -XXtlaSize:preferred
Read [9] for the tuning advice.

What are the default TLA settings?

To find out the default TLA settings, you can use the following JVM option:[12]
  • -Xverbose:memdbg
For example, here is the output:

[DEBUG][memory ][19709] Minimum TLA size is 2048 bytes.
[DEBUG][memory ][19709] Preferred TLA size is 16384 bytes.
[DEBUG][memory ][1388121278654][19709] TLA waste limit is 2048 bytes.

from the following JVM options:
  • -Xms2g -Xmx2g -Xverbose:memdbg -Xgc:pausetime -Xverbose:gc -Xverbosedecorations=level,module,timestamp,millis,pid


  1. Oracle JRockit- The Definitive Guide
  2. Tuning Java Virtual Machines (JVMs)
  3. On Nursery Sizing (Migrated)
    • The goal of nursery sizing to get as much memory as possible freed by young collections rather than old collections.
    • The rule of thumb is that a nursery size of approximately half of the free memory on the heap is nearly optimal.
    • If your application is sensitive to latencies you may want to decrease the nursery size to shorten the young collection pause times.
    • If you're using the garbage collection mode optimizing for short pauses (-Xgcprio:pausetime) or the static generational concurrent garbage collector (-Xgc:gencon) you will most likely want to tune the nursery size manually.
  4. First Steps for Tuning the Oracle JRockit JVM
  5. Oracle® JRockit Diagnostics and Troubleshooting Guide (Release R28)
  6. Oracle® JRockit Performance Tuning Guide (Release R28)
  7. Oracle® JRockit Flight Recorder Run Time Guide (Release R28)
  8. -XXtlaSize Parameters
  9. Optimizing Memory Allocation Performance (Section 4.4 of this pdf)
  10. Oracle® JRockit Command-Line Reference (Release R28)
  11. JRockit: A Case Study of Thread Local Area (TLA) Tuning (Xml and More)
  12. JRockit: Analyzing GC With JRockit Verbose Output (-Xverbose:memdbg)  (Xml and More)
  13. Where did all of these ConstPoolWrapper objects come from?!

Friday, November 22, 2013

AWR Wait Events: Free Buffer Waits vs. Buffer Busy Waits

To diagnose Oracle performance problems, you can use Automatic Workload Repository (AWR)[1] to collect, process, and maintain performance statistics for problem detection. This data is both in memory and stored in the database. The gathered data can be displayed in both reports and views.

In this article, we will discuss the following topics:
  • Top 5 Timed Foreground Events[2]
  • Free buffer Waits vs. Buffer Busy Waits[3]

Top 5 Timed Foreground Events

The Top 5 Timed Foreground Events section is where you can usually spot the problem, by showing you why the sessions are “waiting.” Make sure to analyze the total waits and average waits (ms) separately, in order to determine if the waits are significant. 

For a given workload, this list of events should be relatively stable; you should investigate any significant variation. You will generally see DB CPU at the top of the list. However, CPU usage is not necessarily an indication of a healthy system, as the application may be CPU-bound. The most common wait events are:
  • db file sequential read
  • db file scattered read
  • db file parallel read
  • log file sync

In a nicely performing database, you should see CPU and I/O as the top wait events. If any wait events from the Concurrency wait class or Configuration wait class,[7] investigate those waits further.  For example, in one of the AWR reports, we have found:

EventWaitsTime(s)Avg wait (ms)% DB timeWait Class
free buffer waits1,336,44420,8801655.33Configuration
db file sequential read11,443,82012,942134.30User I/O
DB CPU1,5284.05
undo segment extension5,0315471091.45Configuration
buffer busy waits106847950.22Concurrency

Free Buffer Waits

The Free Buffer Waits event indicates that your session is waiting for a buffer to become empty.  For instance, the database writer is copying the current contents to disc and your session has to wait for this to complete before you can empty the buffer and put something else in it.

Possible Causes and Solutions:
  • DB_BUFFER_CACHE may need to be tuned[4]
    • If the cache hit ratio is low and your application is tuned to avoid performing full table scans, consider increasing the size of the buffer cache.
      • If it is possible to allocate the extra memory required to the buffer cache without causing the system to page, then allocate extra memory.
      • To increase the amount of memory allocated to the buffer cache, increase the value of the DB_CACHE_SIZE initialization parameter.
  • If all your SQL is tuned, free buffer waits could also indicate that:
    • Unselective SQL is causing data to flood the buffer cache with index blocks, leaving none for this particular statement that is waiting for the system to process. 
    • This normally indicates that there is a substantial amount of DML (insert/update/delete) being done and that the Database Writer (DBWR) is not writing quickly enough; the buffer cache could be full of multiple versions of the same buffer, causing great inefficiency. 
    • The solutions include, but not limited to:
      • Avoid performing full table scans
      • Accelerating incremental checkpointing, using more DBWR processes, increasing the size of buffer cache, or increasing the number of physical disks.
  • Investigate if this is an I/O problem

Buffer Busy Waits

The Buffer Busy Waits event indicates that an Oracle session needs to access a block in the buffer cache,
but cannot because the buffer copy of the data block is locked. Buffer busy waits should not be greater than 1 percent.  This buffer busy wait condition can happen for either of the following reasons:
  • The block is being read into the buffer by another session, so the waiting session must wait for the block read to complete. 
  • Another session has the buffer block locked in a mode that is incompatible with the waiting session's request.

Possible Causes and Solutions:[2,3,5]
  • Check the Buffer Wait Statistics section (see next section) to find out if the wait is on:
    • segment header
      • Increase the freelist groups or increase the pctused to pctfree gap
    • file header block
      • If you are hitting file header block contention (Reason Code 13) on the temp files, increase the number of header blocks by adding more tempfiles.[5]
    • undo header
      • Add rollback segments
    • undo block
      • Reduce the data density on the table driving this consistent read or increase the DB_CACHE_SIZE. 
    • data block
      • Move data to another block to avoid this hot block, increase the freelists on the table, or use Locally Managed Tablespaces (LMTs). 
      • Use a smaller block size: fewer records fall within a single block in this case, so it's not as "hot." 
    • index block
      • Rebuild the index, partition the index, or use a reverse key index. 
  • When a DML (insert/update/ delete) occurs, Oracle Database writes information into the block, including all users who are "interested" in the state of the block (Interested Transaction List, ITL). 
    • To decrease waits in this area, you can:
      • Increase the initrans, which will create the space in the block to allow multiple ITL slots. 
      • Increase the pctfree on the table where this block exists (this writes the ITL information up to the number specified by maxtrans, when there are not enough slots built with the initrans that is specified).
  • Other tunings
    • Database writer (DBWR) contention tuning
    • Implementing Automatic Segment Storage Management (ASSM, a.k.a bitmap freelists)

Buffer Wait Statistics

Here is the Buffer Wait Statistics section from our AWR snapshot:
  • ordered by wait time desc, waits desc
ClassWaitsTotal Wait Time (s)Avg Time (ms)
undo header827752909
file header block4540896
data block1107

SQL ordered by Elapsed Time

Finally, from the SQL Ordered by Elapsed Time section, it seems to indicate that a DML (i.e., delete) operation poorly performed.  This has given us another clue to investigate our performance issue.  From the SQL Id, you should be able to find out more details of the interested SQL in your AWR snapshot.

Elapsed Time (s) CPU Time (s) Executions Elap per Exec (s) % Total DB Time SQL Id SQL Module SQL Text
35,931 1,457 1,000 35.93 95.22 83uuf84dqpbaq   delete /*+ INDEX (a EQ$ATTR_VA...
35,925 1,479 0 95.20 6vyng4f5vktyb   DECLARE job BINARY_INTEGER := ...


  1. Overview of the Automatic Workload Repository
  2. AWR Top 5 Timed Events - Top 10
  3. What is the difference between buffer busy waits and free buffer waits
  4. Tuning the Database Buffer Cache
  5. Resolving file header contention on tempfiles
  6. Oracle® Database Performance Tuning Guide 10g Release 2 (10.2)
  7. Classes of Wait Events
    • Configuration: Waits caused by inadequate configuration of database or instance resources (for example, undersized log file sizes, shared pool size)
    • Concurrency: Waits for internal database resources (for example, latches)
  8. Oracle® Database Reference 10g Release 2 (10.2)
  9. More on Interested Transaction Lists
  10. Oracle automatic segment space management:  ASSM internal structures

Saturday, November 16, 2013

Healthy Pursuits Are like Traveling

The American travel writer Paul Theroux once wrote: "Tourists don't know where they've been, travelers don't know where they're going."

Healthy pursuits are like traveling. We know there is a wonderland called wellness. But, there are no fixed routes to reach there. The pursuits need effort and determination. We cannot act like tourists who don't know where they've been. We must take notes of warning signs sent by our bodies.

On the journey to wellness, there are many dangers to be avoided; there are many footsteps to be taken; unfortunately, there are no shortcuts.

Travelers walking in the night follow North Star. Healthy pursuits work similarly. We know our goal; we know the signposts; we know the dangers on the road; we can adjust pace if we are tired; we can change travel plans due to the weather. But, We walk steadily and persistently. With my companionship, hopefully, your journey will be made easier.

May all be fed. May all be healed. May all be loved.

Travel and Health

Be Healthy!

Tuesday, November 12, 2013

How to Restart OATS Services

This article is one of the Oracle Application Test Suite (OATS)[1] series published on Xml and More, which includes the following:
In this article, we will show how to restart OATS services on the Linux system. The Linux system referenced here is Oracle Linux:[2]

$ cat /etc/*-release

Enterprise Linux Enterprise Linux Server release 5.8 (Carthage)
Oracle Linux Server release 5.8
Red Hat Enterprise Linux Server release 5.8 (Tikanga)


Red Hat (i.e., RHEL 5) includes the service utility to help you manage your start up scripts and save you a lot of typing. This is handy when you're managing the already existing services (e.g., OATS services). /sbin/service is just a shell script (see Appendix) that comes as part of Red Hat's initscripts package.[4]

OATS Services

There are three OATS services running on Linux.[5] To restart them, you use the following commands:

# /sbin/service OracleATSAgent [start|stop]
# /sbin/service OracleATSServer [start|stop]
# /sbin/service OracleATSHelper [start|stop]

There is also a command option (i.e., "status") that allows you to check their current status:

#/sbin/service OracleATSHelper status
OATS Helper Service is running

Note that you need to be the superuser to run the above commands. Also, if you install OATS database on the same server, you need to start your DB first before you restart OATS services.


cat /sbin/service

. /etc/init.d/functions

VERSION="`basename $0` ver. 0.91"
USAGE="Usage: `basename $0` < option > | --status-all | \
[ service_name [ command | --full-restart ] ]"

if [ $# -eq 0 ]; then
   echo "${USAGE}" >&2
   exit 1

cd /
while [ $# -gt 0 ]; do
  case "${1}" in
    --help | -h | --h* )
       echo "${USAGE}" >&2
       exit 0
    --version | -V )
       echo "${VERSION}" >&2
       exit 0
       if [ -z "${SERVICE}" -a $# -eq 1 -a "${1}" = "--status-all" ]; then
          cd ${SERVICEDIR}
          for SERVICE in * ; do
            case "${SERVICE}" in
              functions | halt | killall | single| linuxconf| kudzu)
                if ! is_ignored_file "${SERVICE}" \
                    && [ -x "${SERVICEDIR}/${SERVICE}" ]; then
                  env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" status
          exit 0
       elif [ $# -eq 2 -a "${2}" = "--full-restart" ]; then
          if [ -x "${SERVICEDIR}/${SERVICE}" ]; then
            env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" stop
            env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" start
            exit $?
       elif [ -z "${SERVICE}" ]; then
         OPTIONS="${OPTIONS} ${1}"

if [ -x "${SERVICEDIR}/${SERVICE}" ]; then
   echo $"${SERVICE}: unrecognized service" >&2
   exit 1


  1. Oracle Application Testing Suite
  2. Oracle Linux
    • Oracle Linux, formerly known as Oracle Enterprise Linux, is a Linux distribution based on Red Hat Enterprise Linux (RHEL), repackaged and freely distributed by Oracle, available under the GNU General Public License (GPL) since late 2006
  3. How To Install Services on Linux
  4. Understanding your (Red Hat Enterprise Linux) daemons
  5. OpenScript Functional Testing Introduction
    • Services installed by ATS Setup on Linux:
      • Oracle ATS Agent: Remote Agent Service used for Load testing
      • Oracle ATS Helper: OpenScript Helper Service.
      • Oracle ATS Server: Oracle Load Testing & Oracle Test Manager Console. (Weblogic server)
  6. OATS: Tie All Processes Together — from OpenScript to Scenario (Xml and More)

Thursday, November 7, 2013

Java Throwable: ClassNotFoundException vs. NoClassDefFoundError

Many times we have confused ourselves with the following two Java Throwable messages:

Although both of them are related to Java Classpath,[7] they are different.[1] In a nutshell, they differ in this way:
  • ClassNotFoundException
    • Thrown when an application tries to load a class at run-time and name was provided during runtime not at compile time
  • NoClassDefFoundError[11,12]
    • When JVM or a ClassLoader instance is not able to find a particular class at runtime which was available during compile time


ClassNotFoundException is thrown when an application tries to load in a class through its string name using:
  • The forName method in class Class.
  • The findSystemClass method in class ClassLoader .
  • The loadClass method in class ClassLoader.
but no definition for the class with the specified name could be found. See How-To section below for solutions.


The searched-for class definition existed when the currently executing class was compiled, but the definition can no longer be found. One way to debug NoClassDefFoundError is going back to the design-time environment. Using an IDE, you might be able to find where the class is coming from at compile time. Then use that as a clue to find why that class cannot be found at runtime. See How-To section for more details.

What Could Go Wrong?

Classpath[7] in Java is path to directory or list of directory which is used by ClassLoaders[3] to find and load class in Java program. If a class cannot be found at runtime, it may be due to:
  • Classloaders are not set up correctly[3]
  • Class is corrupted
    • Java compiler is not backwards compatible. For example, bytecode generated with JDK 7 won't run in Java 1.6 JVM.[2]
  • Jar file could be renamed in the runtime environment
  • Your startup script may have overridden Classpath environment variable
  • You might be running your program using jar command and class was not defined in manifest file's ClassPath attribute.

How to Resolve it?

The application that triggered the request to load a class receives a ClassNotFoundException or NoClassDefFoundError if neither the classloader nor any of its ancestors can locate the class.[3] In that case, you can take the following actions:
  • You can use System.getproperty("java.class.path") to get the class path used by your Java application at runtime.[4]
  • Try to run with -classpath option using the classpath you think would work: if it works, then it's a sign that some one is overriding java classpath.
  • Check the permission of your jar files.  Your application may not be able to access them.
  • Enable class loading traces at JVM level. For example, you can specify -verbose:class for both JRockit and HotSpot.[5]
  • If your application is deployed in WebLogic server, read [3] and enable classloader debugging. 
    •  For example, you may want to set:
      • -Dweblogic.utils.classloaders.GenericClassLoader.Verbose=true 
      • -Dweblogic.utils.classloaders.ChangeAwareClassLoader.Verbose=true
    • You can also use Classloader Analysis Tool (http://localhost:port/wls-cat/) which is deployed by default on admin servers of domains in development mode.[6]
    • If your application runs in one environment and not in another,
      • Try adding the CLASSPATH explicitly pointing to your jars in setDomainEnv script 
      • You can also set "EXT_PRE_CLASSPATH=...." or "EXT_POST_CLASSPATH=..." where "..." are your jar files. The will pick up these and add to CLASSPATH. The above environment variables can be set when you log on or somewhere at the top of


  1. 3 ways to solve java.lang.NoClassDefFoundError in Java J2EE
  2. Is JDK “upward” or “backward” compatible?
  3. WebLogic's Classloading Framework (Xml and More)
  4. System Properties
  5. -verbose:class Option
  6. Using the new WebLogic Classloader Analysis Tool (CAT)
    • Note that I'm not sure if this is still available in newer WLS releases.
  7. How to Set Classpath for Java on Windows Unix and Linux
    • Main difference between PATH and CLASSPATH is that former is used to locate Java commands while later is used to locate Java class files.
  8. java.lang.UnsatisfiedLinkError: Setting Environment Variable (Xml and More)
  9. Using the Classloader Analysis Tool (CAT)
  10. WebLogic Server (WLS) Support Pattern: Investigating Different Classloading Issues (Doc ID 1572862.1)
  11. If you use JPA 2.1 with WLS 12.1.1 or 12.1.2, then you may see this (because JPA 2.1 only supported starting in 12.1.3):
    • java.lang.NoClassDefFoundError: javax/persistence/StoredProcedureQuery
  12. java.lang.NoClassDefFoundError: sun/io/CharacterEncoding (Xml and More)

Wednesday, November 6, 2013

ORA-01691: unable to extend lob segment

This article is one of the Oracle Application Test Suite (OATS)[1] series published on Xml and More, which includes the following:
Today, I have seen the following message:
ORA-1691: unable to extend lobsegment OLT.SYS_LOB0000079598C00002$ by 128 in tablespace USERS
from alert/log.xml, which was reported by the Oracle DB of OATS.[1]

In this article, we will show:
  1. How to investigate
  2. How to provide the solution


If you see "ORA-01691" error, normally this means that the tablespace is full or that there is not an extent (aka chunk) available that fits the size needed.[2]

To investigate further, you can query the DB with the following SQL commands:[2]

SQL> set long 100000
SQL> select dbms_metadata.get_ddl('TABLESPACE','USERS') from dual;


  '/scratch/aime1/app/oracle11.' SIZE 5242880

  '/scratch/aime1/app/oracle11.' RESIZE 34359

When I checked the used% of tablespace USERS, it showed that it's almost 100% full. So, the error is due to the tablespace being full.

SQL>  select * from dba_tablespace_usage_metrics order by used_percent desc;

------------------------------ ---------- --------------- ------------
USERS                             4194128         4194302   99.9958515

The Solution

You can find out what schema (or user) uses tablespace USERS by issuing the following SQL command:

SQL> select owner, table_name,tablespace_name from dba_tables where tablespace_name='USERS';

From the output, we know tablespace USERS is shared by the following users:
In our environment, we have used OLT to run load tests. For OLT, typically it is the session data that is taking most of the space in the DB. As an immediate workaround, you can try removing sessions from OLT Controller UI. However, it may run very slowly because tablespace USERS has already run out of space. To facilitate the session-removing task from OLT Controller UI, I have added 100MB to the tablespace USERS by giving it a second datafile:

SQL> ALTER TABLESPACE users ADD DATAFILE '/scratch/aime1/app/oracle11.' size 100m;

Tablespace altered.

By adding some space to the tablespace, it helped smooth out the task of session removing. Admittedly, it was still a slow process. But, the solution worked.


  1. Oracle Application Testing Suite
  2. ORA-01691: unable to extend lob segment (OTN)
  3. Administering Tablespaces
  4. SQL Purge command
  5. Folllow @OracleAskTom
  6. Oracle Load Testing Overview
  7. Oracle Test Manager Overview

Tuesday, November 5, 2013

WebLogic Server Cluster Messaging Protocols—Unicast vs. Multicast

WebLogic Server clusters form a loosely-federated group of managed servers that provide a model for applications to leverage for achieving scalability, load balancing, and failover.[1]

To support the above-said functionality, the cluster uses a messaging model for members of the cluster to exchange the information required to keep the cluster in sync.  WebLogic Server supports two cluster messaging protocols:[1,2]
  • Multicast – This protocol, which relies on UDP Multicast, has been around since WebLogic Server introduced clustering back in WebLogic Server version 4.0.
  • Unicast – This protocol, which relies on point-to-point TCP/IP sockets, was added in WebLogic Server 10.0.
In this article, we will look at an example cluster (i.e, CustomerCluster) which is composed of two cluster members:
  • CustomerServer_1
  • CustomerServer_2
Note that, in our configuration, we have both servers installed on the same machine.  But, the best practice is to install them on different machines in case that one of them crashes.


CustomerCluster mentioned here is configured in the CRM Domain of CRM Fusion Application.  There are two members in the cluster.  To sync up each other, unicast cluster messaging mode was chosen as shown below:

It is important to note that although unicast is the default protocol, Oracle fully supports both protocols equally. As stated in [1], parts of the WLS documentation suggest or imply that multicast is only supported for backwards compatibility (see [4]). This suggestion or implication is incorrect.  For example, if you are using WebLogic Server 12c, the choice of protocols should not be influenced by this wording in the WLS documentation. Read [1] for a good comparison for clusters using either unicast or multicast protocols.

Group Leader Strategy

Unicast protocol relies on point-to-point TCP/IP sockets.  So, WebLogic Server’s unicast implementation uses a group leader strategy to limit the growth in the number of sockets required as the cluster size grows.  The cluster is split into one or more groups; each group has a group leader.  Group members communicate with the group leader; group leaders also communicate with other group leaders in the cluster. If a group leader dies, the group elects another group leader.  In the example CustomerCluster, CustomerServer_1 is the group leader (see Figure).

Final Words

When configuring WebLogic Server clusters for unicast communications, if the servers are running on different machines, you must explicitly specify their listen addresses or DNS names.

To find out more information, read the following articles:


  1. WebLogic Server Cluster Messaging Protocols
  2. WebLogic Server
    Version: is used in the demonstration of this article.
  3. Interview Question - How to persist session across Weblogic?
  4. Communications In a Cluster

Monday, November 4, 2013

Java EE and GlassFish Server Roadmap Update

On 11/04/2013, Oracle has just announced a roadmap update on Java EE and GlassFish.  The major changes include, but not limited to:
  • Oracle will no longer release future major releases of Oracle GlassFish Server with commercial support – specifically Oracle GlassFish Server 4.x with commercial Java EE 7 support will not be released.
  • Commercial Java EE 7 support will be provided from WebLogic Server.
  • Oracle GlassFish Server will not be releasing a 4.x commercial version

Sunday, November 3, 2013

Career, Work, and Health (7)

Flu season is coming. Some experts advise not to take flu vaccines while some, especially drug companies, are promoting flu vaccines in earnest.

Based on personal research, I have posted two articles:
Similar to provide cares for your computer bugs, I hope to provide some guidance for your health issues too. Since 2005, I have read many books (most of them written by physicians) and research articles. The results are approximately one hundred health-related articles which were posted on two web sites:
Wish your computer and health are both in excellent conditions! As the author of best-sellers John Robbins said in his book:
"May all be fed. May all be healed. May all be loved."

Saturday, November 2, 2013

How to Create Load Testing Scripts Using OpenScript

This article is one of the Oracle Application Test Suite (OATS)[1] series published on Xml and More, which includes the following:

In this article, we will show:
  • How to create load testing scripts using OpenScript[2]

Introduction to OpenScript

Application Testing Suite (OATS) is comprised of several tightly integrated products.[1] The script designer —OpenScript—only runs on Windows, but all the runtime components are available for both Linux and Windows. OpenScript is a scripting platform for creating automated test scripts in Java.

You can use OpenScript to create scripts for different testings. For example, OATS supports
  • Functional Testing
  • Load Testing
In this article, we will show you how to create load testing scripts in OpenScript.

The Platform

Scripting platform is based upon the Eclipse open source development environment. Initial OpenScript product provides access to a limited set of the Eclipse development environment.

The workbench is the base layer of software and code that provide the foundation on which the OpenScript Modules and Application Programming Interfaces (APIs) operate. Each Workbench window contains one or more perspectives. OpenScript Workbench provides the following perspectives:
  • Tester perspective
  • Developer perspective
  • Reset perspective
Workspaces are created in Oracle OpenScript. Workspaces store project-related script files and Results Log files. You can use them to organize your various testing projects. Three levels of management are provided:
  • Scripts (lowest)
  • Folders
  • Repositories (highest)
You can download OATS from [3]. The version used in this demonstration is
Version: Build 376

Cheat Sheet

Like every recording task, you need to rehearse and make sure all glitches are resolved before the final recording. If you have decided the click path, prepare a cheat sheet like below:

[1] Bring_up_FUSE_URL
[2] Login_SALESREPUSER00001_Welcome1
[3] Click_the_Opportunities_Card
[4] Select_Quarter_2_2013
[5] Drilldown_on_Pinnacle_Server
[6] Click_on_Sales_Account_Picker
[7] Search_for_CUSTOMER_101328336
[8] Click_Cancel
[9] Click_Add_Revenue_Item
[10] Select_Type_Item
[11] Click_Product_LOV_and_Search
[12] Search_for_Elite_Pro_DG_452
[13] Select_Product_and_click_Ok
[14] Click_Cancel
[15] Logout

The numbering of step is for human readers and can help the recording. Each row on the list corresponds to a click in your click path and will become the title of step group in OpenScript recording.

OpenScript Preferences

Before recording, there are some preferences you want to set. To set them, you click on View and then OpenScript Preferences. For example, we would like to control the grouping, naming and numbering of step groups by ourselves (see "Cheat Sheet"). So, set your "ADF Load" preferences as below:

Creating a New Project

In this demonstration, we will create an open script for load testing (File > New...). Our web application is CRM FUSE. So, we have selected "Oracle Fusion/ADF" wizard from the New Project (see above).
  • Oracle Fusion/ADF
    • This option lets you create a new script for load testing of Oracle Application Development Framework (ADF)-based applications and other applications that utilize HTTP and ADF protocols at the protocol level.

At the next step, you are asked to provide a Script Name. So, we set them as follows:
  • Create script as a Function Library (unchecked)
  • Script Name: FUSE_Saleopty_oct07_wrk
Finally click Finish to create a new script. The resulting script will contain the Initialize, Run, and Finish nodes. The Run node will contain recorded HTTP protocol navigations based upon the defined Step Group preferences and the navigations and ADF protocol for actions performed during recording. You can edit the script tree or Java code to customize the script.[4]

In the following sections, we will demonstrate how to create:
  • First step group
  • Remaining step groups

Creating First Step Group

Creating the first step group is a bit different from the rest. So, we describe it separately. Note that we have NOT clicked on the record button yet.

Before we click on the record button (i.e., red circle), create the first message group as shown below:
Open your notepad and copy the first row into the title field as shown below. Then click OK.

Note that we have chosen "No delay" for the first step. But, for other steps, we will specify "Delay 44 secs."

Start Recording

Now click the record button. Your chosen browser (for our demonstration, it's Firefox) will be brought up. Copy your URL:
into the address field and hit Enter. This finishes the recording of the first step group.

Next repeat the following subtasks for the remaining groups until it finishes:
  1. Creating a new step group in OpenScript
    • Right select previous Step Group to bring up context menu and select New > Step Group
  2. Copying the next row of click path from Notepad
  3. Clicking next step in your Browser

Finally, don't forget to stop the recorder.

Exporting Script

If your runtime environment is Linux, you need to export script created in OpenScript as follows:
File > Export Script...

For example, a new zip file was created in our default repository:
You can then copy it to your Linux box:
scp aime1@mylinuxserver:/scratch/aime1/work


  1. Oracle Application Testing Suite
  2. OpenScript for Load Testing Script Troubleshooting (Tutorial)
    • Version: Build 376 was used in this article.
    • This version requires Firefox 10.0 ESR (Windows download).
  3. Oracle Application Testing Suite Downloads
  4. Oracle Application Testing Suite 12.x: Oracle Load Testing Overview
  5. OATS: Tie All Processes Together — from OpenScript to Scenario (Xml and More)

Tuesday, October 8, 2013

How to Configure Scripts to use Databanks in OATS

This article is one of the Oracle Application Test Suite (OATS)[1] series on Xml and More, which began with the following articles:
In this article, we will show you:
  • How to set up databanks in OpenScript[4]
  • How to configure scripts to use databanks


Databanks are used to hold input data that can be automatically fed into your Web application during Load Testing[5].  In this article, we have created a new script in OpenScript using ADF Load Test Module for the load testing of CRM FUSE (i.e., Fusion CRM with FUSE interface).
At the login step (i.e., step 2), we have used the following credentials:
  • SALESREPUSER00001 /Welcome1
during script recording in OpenScript.  To do load testing, we need to use different users to drive our Web application .  In the following, we will show you how to achieve that.

Substitute Variables

After clicking the Post Data node, list of parameters used in HTTP Post is displayed.  As you can see that the value of userid was recorded as SALESREPUSER00001.  So, we need to configure it to use a databank.
After clicking on Edit..., it brings up the Parameter message box.  Let's click on the second Substitute button next to the Value field.

This brings up Substitue Variable window.  Next we need to add new databank before we can use it for variable substitution.

Note that before we started OpenScript, we have copied an existing databank named UserName.csv in the the subfolder of our workspace (i.e., FUSE_Saleopty_oct07_wrk).  Therefore, you have seen it appearing in the right panel.  For our script, we have chosen to store our databank "Relative to current script".

If you open the databank, you can see different user names are stored in it.  Note that all users will use the same password in our case.

After adding new databank, the value of userid has been updated to:

Finishing variable substitution, a new node (i.e., GetNextDatabankRecord) was shown next to "Run" node.

You can click on the new node to edit it.  But, the default (i..e, Next Record) works for us.

After we have configured our script to use databank, we can test it using "Iterate" playback.

Clicking on the Iterate button, this allows you to specify how many iterations you want to test.  In the following example, we have set it to be three times.

Next Steps

After testing our script running OK in the OpenScript (which runs on Windows only), we proceed with the following steps:
  • Export script to a ZIP archive (i.e.,
  • Copy the ZIP archive file to the Oracle Load Testing (OLT) server (which runs on Linux)
  • In OLT, we have imported the OpenScript ZIP to a scenario.
  • After setting up Autopilot, we have run load testings using the databank to feed CRM FUSE with different users.
A new article describing the above steps will be posted in the future given time permitted.

Monday, October 7, 2013

Removing "Compatibility setting is not supported" Warning before You Do OpenScript Recording

This article is one of the Oracle Application Test Suite (OATS)[1] series on Xml and More, which began with the following article:
In this article, we will show one of the warnings that you may want to remove before recording your scripts using OpenScript [3]in OATS[1]:

Compatibility Setting

You can use either Firefox or Internet Explorer to record scripts. In this example, we used IE8. The application we are recording with is Fusion CRM with FUSE interface.

After provided home page URL and logged in, we have seen the above warning. Don't know exactly how OATS records this Message box. But, this is not the regular step of the application task flows.[5] So, you should remove this issue before any recording.

The Solution

In IE8, you select:
  • Tools > Compatibility Setting

Then the following Compatibility View Settings will be shown:
To fix the compatibility warning, you just uncheck the following item:
  • Display intranet sites in Compatibility View


  1. Oracle Application Testing Suite (a comparable product to LoadRunner)
  2. Auto-Correlating Session IDs in Oracle Application Test Suite (OATS)
  3. OpenScript for Load Testing Script Troubleshooting (Tutorial)
  4. Turn off Compatibility View
  5. Oracle ADF Task Flow in a Nutshell (Xml and More)
  6. How to Configure Scripts to use Databanks in OATS (Xml and More)

Friday, September 27, 2013

Configuring Diagnostic Framework (DFW) Settings

A while ago, I've posted an article titled
Understanding WebLogic Incident and the Diagnostic Framework behind It[1]

This article is a follow-up of that one.  In this article, we will discuss how to configure Diagnostic Framework Settings.

Diagnostic Framework (DFW)

A quick recap what Diagnostic Framework (DFW) is.  Oracle Fusion Middleware includes a Diagnostic Framework (DFW). DFW is available with all FMW 11g installations that run on WebLogic Server. It aids in detecting, diagnosing, and resolving problems, which are targeted in particular critical errors.

There are two ways that you can modify DFW settings:
  1. Modifying the configuration file named dfw_config.xml
  2. Making updates via Fusion Middleware Control (FMW Console)[2]
In this article, we will show how to modify the following setting:
  • maxTotalIncidentSize 
which configures the maximum total disk space allocated to incidents.

Configuration File

DFW's configuration file is named dfw_config.xml.  There is one dfw_config.xml file for each server.  For example, there is one for  CRMCommonServer_1:
  • config/fmwconfig/servers/CRMCommonServer_1/dfw_config.xml
in the CRMDomain.

Here is the sample contents of dfw_config.xml:
<?xml version="1.0" encoding="UTF-8"?>
<diagnosticsConfiguration xmlns="<snipped>" 
  <!-- maxTotalIncidentSize configures the maximum total disk space
       allocated to incidents, in megabytes. -->
  <incidentCleanup maxTotalIncidentSize="500"/>
  <threadDump useExternalCommands="true"/>
  <dumpSampling enabled="true">
      <dumpArgument name="timing" value="true"/>
      <dumpArgument name="context" value="true"/>
In our case, we would like to update
  • maxTotalIncidentSize
to 150 MB from 500 MB.

FMW Console

To start Oracle Enterprise Manager 11g, I have used the following URL:

The following diagrams show how to configure
  • maxTotalIncidentSize 
using the Fusion Middleware Control System MBean Browser:
  1. From the target navigation pane, expand the farm, then WebLogic Domain.
  2. Select the domain.
  3. From the WebLogic Domain menu, choose System MBean Browser.
  4. The System MBean Browser page is displayed.
  5. Expand Application Defined Beans, then oracle.dfw, then domain.domain_name, then dfw.jmx.DiagnosticsConfigMBean.
  6. Select one of the DiagnosticConfig entries. There is one DiagnosticConfig entry for each server.
  7. In the Application Defined MBean pane, expand Show MBean Information to see the server name.