Wednesday, December 24, 2014

OATS: How to Fix Quarter-Crossing Issue for a Benchmark

In [1], we have described how to record a load testing script using OpenScript for CRM Fuse. In the click path, it includes one of the following step:
  • [4] Select_Quarter_3_2013
After setting up a benchmark, we tend to use it for a long time. Step 4 actually causes an issue which we call it "Quarter-Crossing" issue. What happens is that CRM Fuse is live and advance its list by adding current quarter to the list and removing the last entry from the list.

When that happens (i.e., at end of each fiscal quarter), our Oracle Load Testing (OLT) scenario will fail with validation error as:
ERROR [1] Iteration 1 failed at line ( Failed in finding component with xpath=//AdfRichTable[@fullId='atkfr1:0:r22:1:r1:0:pt1:t1']//AdfRichCommandLink[@partialSubmit='true'], no matching pathSegment=/
Basically, ADF internal state[2] will change and the xpath used to retrieve the list item "Quarter 3, 2013" will fail.

In this article, we will discuss how to work around this if it happens.

Quarter Crossing Events

At the end of each quarter, we need to maintain our benchmark CRM Fuse because of the quarter-crossing event. As shown above, "Quarter 3, 2013" is the only list item with seeded data for CRM testing. In other words, "Pinnacle Server Sales per test0000001" will be shown only when "Quarter 3, 2013" is selected. However, at the end of each quarter, the position of it will be shifted down one place on the list. This will cause the validation step in OLT scenario to fail. In addition, "Quarter 3, 2013" eventually will be moved out of the list.

SQL Statements

Whenever scuh events happen, it means you need to recreate OLT scenario by following the steps described in [3]. In addition, you need to update Fusion table (i.e., moo_opty) with new effective date. So, here we show the SQL statements needed for that purpose.

$ORACLE_HOME/bin/sqlplus fusion/fusion

select effective_date from moo_opty where name like 'Pinnacle Server Sales Test%';


As shown above, "Quarter 3, 2013" is associated with an EFFECTIVE_DATE of "03-JUN-13". On a future date, say, "Quarter 3, 2014", "Quarter 3, 2013" may be shifted out of the list and no longer available for selection. That's why you want to update the above moo_opty table in addition to recreate OLT scenario at each quarter-crossing event.

For demonstration, we have updated the above EFFECTIVE_DATE (of type DATE) column with a new value 01-JUN-14:

update moo_opty set effective_date = to_date('06/01/2014', 'MM/DD/YYYY') where name like 'Pinnacle Server Sales Test%';

1,000 rows updated.

After the above update, the seed data with "Pinnacle Server Sales per test0000001" will be associated to a new quarter (e.g., Quarter 2, 2014).  But, eventually you may need to update EFFECTIVE_DATE to a later date than 01-JUN-14 if time moves on further.


  1. How to Create Load Testing Scripts Using OpenScript
  2. Oracle Application Development Framework - Oracle ADF
  3. OATS: Tie All Processes Together — from OpenScript to Scenario

OATS: All Things about OLT Sessions

Oracle Load Testing(OLT)[1] is a component of Oracle Application Test Suite (OATS), which can simulate hundreds of virtual users accessing the application simultaneously and measures the effect of the load on application performance.

The results of load tests are grouped by sessions which are stored in Oracle database.  Oracle Load Testing then allows you to create customized post-run reports on the results of your load tests.


After you've run many load tests, eventually you could run into:
ORA-01654: unable to extend index OLT.COUNTERRUN_IDX by 1024 in tablespace USERS
OLT comes with a Session Manager which allows you to manage session data in the database:
  • Edit
    • Displays the Edit Session dialog box for changing the name and description of the session.
  • Delete
    • Deletes the entire selected session data from the database.
  • Delete Virtual User Logs
    • Deletes only the VU log data for the selected sessions. 
The easiest solution to the ORA-01654 is to delete old session data as shown below:

On the other hand, if you are using Oracle EE[2,3] and have enough disk space, all you need to do is:
Use ALTER TABLESPACE ADD DATAFILE statement to add one or more files to the tablespace indicated.

Note that if you are currently using XE, we would recommend you use a dedicated machine for database and use Oracle EE.

Attach to Session

Read [4] for how to set up autopilot.  After setting up the parameters of autopilot, you can run scenarios from the command line.[4]  Assuming you have done that, then you can monitor load test session by attaching to it in OLT.  This enables a tester to share real-time test results and to collaborate with team members during testing.

From menu Session, select Attach.... The current running session as below should be listed.

You can then select how you want to attach to the session:

Monday, December 22, 2014

Java Security Manager: Troubshooting Security

In [1], we have discussed how to fix security violations when Java Security Manager is enabled at WebLogic Server start-up.  In this article, we will look at how to troubleshoot security policy-related issues by using the option.

To troubleshoot security policy-related issues, you can configure Java Security Manager with the level of security-related information to be reported using:  
To find out what the debug option offers, do:
If using JDK 7, you can find more information from [2].  As for JDK 8, we have included the list of debugging options in the Addendum for your convenience.  Note that you need to separate multiple options with a comma.  For example, we will look at the output from the following combination of debug options:,access

policy Option

Setting "policy" option will allow Security Manager to report the details of
Loading and granting permissions with policy file
For example, from the WebLogic Server log file, you can find the following entries:

policy: Adding policy entry: 
policy:   signedBy null
policy:   codeBase file:/home/myusername/wls1221141208/wlserver/server/lib/-
policy:   ("" "" "")
policy: Adding policy entry: 
policy:   signedBy null
policy:   codeBase file:/home/myusername/wls1221141208/wlserver/modules/-
policy:   ("" "" "")

which correspond to grant entries from the weblogic.policy:[5]

grant codeBase "file:/home/myusername/wls1221141208/wlserver/server/lib/-" {

grant codeBase "file:/home/myusername/wls1221141208/wlserver/modules/-" {

Beside of policy loading entries, you could also find other entries (e.g. permission granting entries) such as:

policy:   granting ("java.util.PropertyPermission" "java.version" "read")
policy:   granting ("java.util.PropertyPermission" "java.vendor" "read")

access Option

Enabling access option, allows Security Manager to print all results from the AccessController.checkPermission method.

You can use the following options with the access option:
  • stack: Include stack trace
  • domain: Dump all domains (i.e., protection domain) in context
  • failure:[3] Before throwing exception, dump stack and domain that do not have permission
You can also use the following options with the stack and domain options:
  • permission=<classname>[4]
    • Only dump output if specified permission is being checked 
  • codebase=<URL>
    • This option would be useful when customer desires to trace the permissions impact of only the code in a given code souce, such as jar file.
    • URL is the location of the specified code base. 
      • Note that because the comma (',") is used as multi options separator, if the URL contains comma, the security debugger would not work properly as expected, it is recommended that the URL should not include character comma (','), semicolon (';'), and  space.


$ jdk-hs/bin/java

all           turn on all debugging
access        print all checkPermission results
certpath      PKIX CertPathBuilder and
              CertPathValidator debugging
combiner      SubjectDomainCombiner debugging
              GSS LoginConfigImpl debugging
configfile    JAAS ConfigFile loading
configparser  JAAS ConfigFile parsing
jar           jar verification
logincontext  login context results
jca           JCA engine class debugging
policy        loading and granting
provider      security provider debugging
pkcs11        PKCS11 session manager debugging
              PKCS11 KeyStore debugging
sunpkcs11     SunPKCS11 provider debugging
scl           permissions SecureClassLoader assigns
ts            timestamping

The following can be used with access:

stack         include stack trace
domain        dump all domains in context
failure       before throwing exception, dump stack
              and domain that didn't have permission

The following can be used with stack and domain:

              only dump output if specified permission
              is being checked
              only dump output if specified codebase
              is being checked

The following can be used with provider:

              only dump output for the specified list
              of JCA engines. Supported values:
              Cipher, KeyAgreement, KeyGenerator,
              KeyPairGenerator, KeyStore, Mac,
              MessageDigest, SecureRandom, Signature.


  1. Java Security Manager: access denied (Xml and More)
  2. Troubshooting Security
  3. Debugging Security Policy Issues
  4. Fine granularity diagnosis on security
  5. Java SecurityManager: "" and "java.policy" Files (Xml and More)

Saturday, December 13, 2014

Java Security Manager: access denied

The JVM has security mechanisms built into it that allow you to define restrictions to code through Java security policy files (i.e., java.policy file). The Java Security Manager[1] uses these policies to enforce a set of permissions granted to classes. The permissions allow specified classes running in that instance of the JVM to allow or not allow certain runtime operations.

In this article, we will cover how to resolve security violations such as access denied
when you start WebLogic Server with Java Security checking enabled (see [1] for how to enable Java Security Manager).

AccessController is the main security enforcer in Java Security Manager.  It helps decide whether an access to a critical system resource is to be allowed or denied, based on the security policy currently in effect,  For example, an will be thrown when a permission was not granted to read property as below:

Caused by: access denied ("java.util.PropertyPermission" "" "read")
at java.lang.SecurityManager.checkPermission(
at java.lang.SecurityManager.checkPropertyAccess(
at java.lang.System.getProperty(
at java.lang.Boolean.getBoolean(

If the security violations in your application are only a few, you may consider to add an entry like below in the default section of your policy file:

// default permissions granted to all protection domains
grant {
    permission java.util.PropertyPermission "", "read";

"weblogic.policy" and "ojdbc.policy"

When you run applications in WebLogic server, Oracle has provided an OOTB policy file at:
If you specify it in the JVM option:${WL_HOME}/server/lib/weblogic.policy
then the specified policy file will be loaded in addition to all the java policy files (see [1] for details). However,  OOTB weblogic.policy is not intended to cover all deployed applications.  For example, if you use Oracle JDBC Driver in your applications, you may need to add the grant entries specified in ojdbc.policy[2] onto the ones specified in weblogic.policy file.

Grant Entries

If you look at either weblogic.policy or ojdbc.policy file, there are many grant entries. The basic format of a grant entry is the following:[4]

  grant signedBy "signer_names", codeBase "URL",
        principal principal_class_name "principal_name",
        principal principal_class_name "principal_name",
        ... {
      permission permission_class_name "target_name", "action", 
          signedBy "signer_names";
      permission permission_class_name "target_name", "action", 
          signedBy "signer_names";

Each grant entry includes one or more "permission entries" preceded by optional codeBase, signedBy, and principal name/value pairs that specify which code you want to grant the permissions.

Sometimes it will be too much to enumerate all permissions individually.  This is the time for granting permissions by any combination of the following entries:
  • SignedBy
  • Principal
  • CodeBase

Grant Permission by SignedBy

An example of granting permission(s) based on signer is shown below:[3]
keystore "kim.keystore";

// Here is the permission ExampleGame needs.
// It grants code signed by "terry" the
// HighScorePermission, if the
// HighScorePermission was signed by "chris"
grant SignedBy "terry" {
      "ExampleGame", signedBy "chris";
A signedBy value indicates the alias for a certificate stored in the keystore. The public key within that certificate is used to verify the digital signature on the code; you grant the permission(s) to code signed by the private key corresponding to the public key in the keystore entry specified by the alias.

The signedBy value can be a comma-separated list of multiple aliases. An example is "Adam,Eve,Charles", which means "signed by Adam and Eve and Charles"; the relationship is AND, not OR. To be more exact, a statement like "Code signed by Adam" means "Code in a class file contained in a JAR which is signed using the private key corresponding to the public key certificate in the keystore whose entry is aliased by Adam".

Grant Permission by Principal

An example of granting permission(s) based on principal is shown below:
// Grant notification listener actions to standard roles

grant principal "Administrators" {
    permission "*", "addNotificationListener";
    permission "*", "removeNotificationListener";

A principal value specifies a class_name/principal_name pair which must be present within the executing thread's principal set. The principal set is associated with the executing code by way of a Subject.

The principal_class_name may be set to the wildcard value, *, which allows it to match any Principal class. In addition, the principal_name may also be set to the wildcard value, *, allowing it to match any Principal name. When setting the principal_class_name or principal_name to *, do not surround the * with quotes. Also, if you specify a wildcard principal class, you must also specify a wildcard principal name.

Grant Permission by CodeBase

A codeBase value indicates the code source location; you grant the permission(s) to code from that location. An example of granting permission by CodeBase is shown below:
// Grant for internal applications when using WebLogic startup scripts
grant codeBase "file:${user.dir}/servers/${weblogic.Name}/tmp/_WL_internal/-" {

BNF Grammar

An informal BNF grammer for the Policy file format is given below, where non-capitalized terms are terminals:[12]

PolicyFile -> PolicyEntry | PolicyEntry; PolicyFile
PolicyEntry -> grant {PermissionEntry}; |
           grant SignerEntry {PermissionEntry} |
           grant CodebaseEntry {PermissionEntry} |
           grant PrincipalEntry {PermissionEntry} |
           grant SignerEntry, CodebaseEntry {PermissionEntry} |
           grant CodebaseEntry, SignerEntry {PermissionEntry} |
           grant SignerEntry, PrincipalEntry {PermissionEntry} |
           grant PrincipalEntry, SignerEntry {PermissionEntry} |
           grant CodebaseEntry, PrincipalEntry {PermissionEntry} |
           grant PrincipalEntry, CodebaseEntry {PermissionEntry} |
           grant SignerEntry, CodebaseEntry, PrincipalEntry {PermissionEntry} |
           grant CodebaseEntry, SignerEntry, PrincipalEntry {PermissionEntry} |
           grant SignerEntry, PrincipalEntry, CodebaseEntry {PermissionEntry} |
           grant CodebaseEntry, PrincipalEntry, SignerEntry {PermissionEntry} |
           grant PrincipalEntry, CodebaseEntry, SignerEntry {PermissionEntry} |
           grant PrincipalEntry, SignerEntry, CodebaseEntry {PermissionEntry} |
           keystore "url"
SignerEntry -> signedby (a comma-separated list of strings)
CodebaseEntry -> codebase (a string representation of a URL)
PrincipalEntry -> OnePrincipal | OnePrincipal, PrincipalEntry
OnePrincipal -> principal [ principal_class_name ] "principal_name" (a principal)
PermissionEntry -> OnePermission | OnePermission PermissionEntry
OnePermission -> permission permission_class_name
                 [ "target_name" ] [, "action_list"]
                 [, SignerEntry];

Some entries in the grammar are optional, If they are omitted, it signifies:

When Omitted
It Means
CodebaseEntry "any code base" (it doesn't matter where the code originates from)
SignerEntry "any signer" (it doesn't matter whether the code is signed or not or by whom)
PrincipalEntry "any principals"

The "target_name"is the name of the permission is aimed.  For, the targets of this class can be specified as:

directory (same as directory/)
directory/* (all files in this directory)
* (all files in the current directory)
directory/- (all files in the file system under this directory)
- (all files in the file system under the current directory)
"<<ALL FILES>>" (all files in the file system)

As an example, you may grant read permission to all files in the file system as below:
permission "<<ALL FILES>>", "read"; 

Finally, a set of actions can be specified together as a comma-separated composite string as below:
permission "WEBLOGIC-APPLICATION-ROOT${/}-", "read, write, delete, execute";


Sunday, December 7, 2014

Linux: How to Load Balance with DNS Round Robin

If you are running multiple application server instances, you will need to use something to balance the HTTP traffic from the network. For example, you can use a DNS server or a hardware load balancer (e.g. F5) to load balance HTTP traffic.

In this article, we will cover how to configure a DNS server to round-robin network traffic across all application servers.

Load Balancing with DNS Round Robin

Domain Name Service (DNS) is an Internet service that map human-memorable domain names and hostnames into the corresponding numeric Internet Protocol (IP) addresses which are used to identify and locate computer systems and resources on the Internet. In this way, DNS alleviates the need to remember IP addresses. Computers that run DNS are called name servers. Both Red Hat and Ubuntu ship with BIND (Berkley Internet Naming Daemon),[12] the most widely deployed DNS server.

Using a DNS server, even with one application server, makes it very easy if you need to change to different machines — you just need to update the IP address on the DNS server instead of on every other machine.

Load balancing allows organizations to distribute inbound traffic across multiple back-end nodes or servers (represented as the IP address of a physical server). Round robin is a local balancing mechanism used by DNS servers to share and distribute network resource loads. Compared to hardware load balancer, DNS round robin is a poor man’s load balancing solution.[1]

Configuring Name Servers

Managing BIND's named daemon is easy to do, but the procedure differs between Linux distributions. Here are some things to keep in mind:
  • Firstly, different Linux distributions use different daemon management systems. Each system has its own set of commands to do similar operations.
  • Secondly, the daemon name needs to be known. In this case the name of the daemon is named.
Armed with this information you can know how to:
  • Start your daemons automatically on booting
  • Stop, start and restart them later on during troubleshooting or when a configuration file change needs to be applied.
For more details, read [11] and [12]. The Linux release used in this article is:

$ cat /etc/*-release
Enterprise Linux Enterprise Linux Server release 5.4 (Carthage)
Red Hat Enterprise Linux Server release 5.4 (Tikanga)

Assume that your named is up and running—To enable DNS services on the name server, the /etc/host.conf file should look like this:

# Lookup names via /etc/hosts first, then by DNS query
order hosts, bind

Next you need to configure DNS tables for DNS services. If you use the BIND 8.x package with the Red Hat distribution, the configuration of DNS tables can be done in:
  • /etc/named.conf
For example, the /etc/named.config could look like this:

options {
// DNS tables are located in the /var/named directory
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
listen-on { any; };
allow-query { any; };
rrset-order {
order cyclic;

controls {
inet allow {

zone "." {
type forward;
forward first;
forwarders {;

// All our DNS information is stored in /var/named/foobar
zone "" {
type master;
notify no;
file "";

The rrset-order substatement accepts fixed, random and cyclic as arguments. By specifying cyclic as the argument, you can configure the name server to return matching records in cyclic (round robin) order.[7]

The DNS table named /var/named/ translates host names (such as jassut) to IP addresses. could have the contents like this:[8]

; Zone file for
@ IN SOA (
2005061504 ; serial # (today's date + increment)
1S ; refresh
1S ; retry
1S ; expire
1S) ; minumum ttl
; sorted alphnumerically by hostname (first column) then by IP address

nameserver A

jassut A
jasapp1dom1inst1nic1 A
jassut A

jasapp1dom1inst1nic2 A
jassut A

jasapp1dom1inst1nic3 A
jassut A

jasapp1dom1inst1nic4 A

Configuring DNS Clients

There are three main client configuration files associated with DNS:[9]
  • /etc/hosts
  • /etc/nsswitch.conf
  • /etc/resolv.conf
When your computer looks for another computer on a network such as the Internet, it typically looks in two places: /etc/hosts and any DNS servers that you've set up for your network.


The /etc/hosts file keeps a local name database. This file helps in local name resolution if your local DNS server is not functioning. Network adinistrators should manually populate entries in this file. A sample /etc/hosts file is copied below.

$ cat /etc/hosts

# Do not remove the following line, or various programs
# that require network functionality will fail. localhost.localdomain localhost clientserver1


The order of name resolution process is determined by a single line in /etc/nsswitch.conf:

hosts: files dns

In this case, the name resolution process will begin with checking /etc/hosts file, and if the name cannot be resolved, the name resolution will happen with the DNS server.


The /etc/resolv.conf file contains directives with the IP addresses of nameservers available to a host. A sample /etc/resolv.conf file is copied below.

$ cat /etc/resolv.conf
options attempts: 5
options timeout: 15


How to Test?

nslookup is a network administration command-line tool available for many computer operating systems for querying the DNS server(s) to obtain domain name or IP address mapping or for any other specific DNS record.

For example, our sample setup uses four server instances in one WebLogic domain, which listens on four Network Interface Cards.

On the client computer, you can query host name jassut by typing:
$ nslookup jassut






Other commands for debugging DNS besides nslookup are host and dig. Note that ping is not useful for debugging DNS, as it will use whatever is configured in /etc/nsswitch.conf to do the name-lookup. Also, noted that our host (i.e., has multiple network interface controllers (NIC).[16]

DNS Caching

The Internet's Domain Name System (DNS) involves caching on both DNS servers and on the client computers that contact DNS servers. To minimize the load on the actual DNS server, we want to use a local DNS cache. There are two levels of local name caching on DNS clients:
  • OS
    • To enable DNS caching on the client computers, you can install nscd and run it.
      • A daemon that provides a cache for the most common name service requests.
      • To launch the daemon, do:
        • /etc/init.d/nscd start
  • JVM
    • By default, the JVM will cache DNS information.
However, if client-side DNS caching is enabled, it can lessen the effectiveness of the round-robin DNS load balancing. In other words, due to the client-side DNS query caching, your round-robin DNS load balancing is not likely to distribute the load on your remote servers on a per-request basis. It will be more like a per-session basis.

To effectively support DNS round robin, you want to:
  • Disable JVM local caching by setting
    • -Dnetworkaddress.cache.ttl=0
  • Decrease the local DNS cache (i.e., nscd) default time-to-live to the minimum of 1 second (see instructions below).

On Windows, under HKEY_LOCAL_MACHINE\System\CurrentControlSet set Services\Dnscache\Parameters\MaxCacheEntryTtlLimit to 1

For UNIX machines with a name service cache daemon (nscd),[5] ensure that /etc/nscd.conf contains

enable-cache hosts yes
positive-time-to-live hosts 1
negative-time-to-live hosts 0


  1. Configuring DNS round robin on Windows
  2. Load Balancing With Round Robin DNS
  3. port 53
    • Default port listened by DNS
  4. Domain Name Server (DNS) Configuration and Administration
    • BIND 8.x
      • /etc/named.conf file should be configured to point to your DNS tables
    • BIND9 (Berkley Internet Naming Daemon)
      • BIND9 Configuration files are stored in /etc/bind
  5. nscd (Name Service Cache Daemon)The default configuration file, /etc/nscd.conf, determines the behavior of the cache daemon
    • See also: nsswitch.conf - System Databases and Name Service Switch configuration file
  6. The Domain Name System (good)
  7. Round Robin Load Distribution
  8. Zone files (Red Hat Linux 5)
  9. Linux Domain Name System (DNS) client configuration files
  10. Linux / Unix: Dig Command
    • The chief difference between nslookup and dig is that dig has no interactive mode: you specify everything at the command line.
  11. Quick HOWTO : Ch18 : Configuring DNS (good)
  12. Berkeley Internet Name Domain (BIND) (Red Hat Linux 5)
  13. Why 192.168.*.* for local addresses?
  14. Load Balancing T3 InitialContext Retrieval for WebLogic using Oracle Traffic Director
    • For bootstrapping the InitialContext, you can choose two approaches:
      • DNS round robin
      • External load blancer such as OTD
  15. DNS and BIND, 5th edition, by Cricket Liu and Paul Albitz
  16. Multihoming

Monday, December 1, 2014

JDK 8: When Softly Referenced Objects Are Flushed?

There are four different degrees of reference strength in Java language:[1]
  1. Strong
    • StringBuffer strongRef = new StringBuffer();
  2. Soft
    • SoftReference softRef = new SoftReference(obj);
  3. Weak
    • WeakReference weakRef = new WeakReference(widget);
  4. Phantom
    • PhantomReference phantomRef =new PhantomReference(bool,rq);
in order from strongest to weakest.

In this article, we will focus on softly referenced objects and when they will be flushed by Garbage Collector (GC) in HotSpot.

SoftReference vs WeakReference

A soft reference is exactly like a weak reference, except that it is less eager to throw away the object to which it refers. An object which is only weakly reachable (the strongest references to it are WeakReferences) will be discarded at the next garbage collection cycle, but an object which is softly reachable will generally stick around for a while.

SoftReferences aren't required to behave any differently than WeakReferences, but in practice softly reachable objects are generally retained as long as memory is in plentiful supply. This makes them an excellent foundation for a cache since you can let the garbage collector worry about both how reachable the objects are and how badly it needs the memory they are consuming.

When Softly Referenced Objects Are Flushed?

Prior to version 1.3.1, the Java HotSpot VMs cleared soft references whenever it found them. Starting with 1.3.1, softly reachable objects will remain alive for some amount of time after the last time they were referenced. The default value is one second of lifetime per free megabyte in the heap. This value can be adjusted using the -XX:SoftRefLRUPolicyMSPerMB flag, which accepts integer values representing milliseconds. For example, to change the value from one second to 2.5 seconds, use this flag:

When, exactly, is a soft reference freed?  Here are the factors to be considered:
  • First the referent must not be strongly referenced elsewhere
  • If the soft reference is the only remaining reference to its referent, the referent is freed during the next GC cycle only if the time it was last accessed is greater than (AmountOfFreeMemoryInMB * SoftRefLRUPolicyMSPerMB).
    • AmountOfFreeMemoryInMB is calculated differently for server and client VMs:
      • Server VM
        • Uses the maximum possible heap size (as set with the -Xmx option) to calculate free space remaining
          • -Xmx therefore has a significant effect on when soft references are garbage collected.
        • The general tendency is for the Server VM to grow the heap rather than flush soft references
      • Client VM
        • Uses the current heap size to calculate the free space
        • The Client VM will have a greater tendency to flush soft references rather than grow the heap


  • When to use SoftReference?
    • Soft references are used when the object in question has a good chance of being reused in the future,
      • For example, as a cache implementation
        • Soft references are essentially one large, least-recently-used cache
  • Performance consideration
    • The key to getting good performance from that cache is to make sure that it is cleared on a timely basis.
    • Eagerly vs. lazily reclaimation of softly referenced objects
      • You can tune how eagerly they are reclaimed by setting:
        • -XX:SoftRefLRUPolicyMSPerMB flag
          • Default is 1000 ms. Optimal setting depends on the applications and which GC is used.
          • This behavior is not part of the VM specification, however, and is subject to change in future releases. Likewise the -XX:SoftRefLRUPolicyMSPerMB flag is not guaranteed to be present in any given release.


  1. Understanding Weak References (good)
  2. Frequently Asked Questions About the Java HotSpot VM
  3. HotSpot Virtual Machine Garbage Collection Tuning Guide
  4. Garbage-First Garbage Collector Tuning
  5. Other JDK 8 articles on Xml and More
  6. Tuning that was great in old JRockit versions might not be so good anymore
    • Trying to bring over each and every tuning option from a JR configuration to an HS one is probably a bad idea.
    • Even when moving between major versions of the same JVM, we usually recommend going back to the default (just pick a collector and heap size) and then redoing any tuning work from scratch (if even necessary).

Saturday, November 29, 2014

WebLogic Server 12.1.3: weblogic.jar Archive File

Prior to the  release of WebLogic 10.0, the weblogic.jar file could be bundled with a client application to provide WebLogic Server specific value added features, such as:[1]
  • Enhanced JDBC and WLS specific JMX interfaces
  • WLS T3 Client
  • WLS-IIOP Client


To boost modularity, WebLogic Server’s file structure have been reorganized for greater flexibility since WLS 10.0. Many WebLogic Server components that were formerly included in the weblogic.jar archive file are now included in separate modules. The weblogic.jar archive now refers to these components in the modules directory from its manifest classpath.

For instance, the weblogic.jar in WLS 12.1.3 includes relative manifest classpath references to:
  • $WL_HOME/modules
  • $(dirname $WL_HOME)/oracle_common/modules

Because of this, weblogic.jar can no longer simply be moved to any new location without complications.


As a consequence of the weblogic.jar reorganization, weblogic.jar is simply a container for the manifest file (i..e, META-INF/MENIFEST.MF).

Here is the contents of manifest file in weblogic.jar of WLS 12.1.3:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.9.2
Created-By: 1.7.0_15-b33 (Oracle Corporation)
Main-Class: weblogic.Server
Implementation-Vendor: BEA Systems
Implementation-Title: WebLogic Server  Fri Dec 27 03:55:01
 PST 2013 1570920
Class-Path: weblogic-classes.jar ../../../oracle_common/modules/featur
 es/ ../../modules/features/weblogic.ser
 ver.merged.jar ../../modules/features/weblogic.server.merge.modules_1 ../../../oracle_common/common/lib/config-launch.jar schem
 a/weblogic-domain-binding.jar schema/weblogic-domain-binding-compatib
 ility.jar schema/diagnostics-binding.jar schema/diagnostics-image-bin
 ding.jar aqapi.jar ../../../oracle_common/modules/mysql-connector-jav
 a-commercial-5.1.22/mysql-connector-java-commercial-5.1.22-bin.jar cr
 yptoj.jar  ../../../oracle_common/modules/datadirect/wlsqlserver.jar
 ../../../oracle_common/modules/datadirect/wldb2.jar ../../../oracle_c
 ommon/modules/datadirect/wlsybase.jar ../../../oracle_common/modules/
 datadirect/wlinformix.jar ../../../oracle_common/modules/datadirect/f
 mwgenerictoken.jar osgi.jar wlw-langx.jar jcom.jar weblogic-L10N.jar
 wljaccutil.jar   ../../modules/features/weblogic.server.modules_12.1.

Thursday, November 27, 2014

WebLogic: Important Deployment Concepts

WebLogic Server implements the Java EE 5 specification. Java EE 5 includes a deployment specification, JSR-88,[1] that describes a standard API used by deployment tools and application server providers to configure and deploy applications to an application server.

Although the Java EE 5 deployment API provides a simple, standardized way to configure applications and modules for use with a Java EE 5-compliant application server, the specification does not address many deployment features that were available in previous WebLogic Server releases. For this reason, WebLogic Server provides important extensions to the Java EE 5 deployment API specification to support capabilities described in [2].

In this article, we will discuss some important concepts for understanding the deployment process in WebLogic Server.

Deployment Tools

The term application deployment refers to the process of making an application or module available for processing client requests in a WebLogic Server domain.  You have several options to deploy a web application onto WebLogic Server:
  • weblogic.Deployer
    • A packaged deployment tool which provides deployment services for WebLogic Server. 
    • Any deployment operation that can be implemented using the WebLogic Deployment API[19] is implemented, either in part or in full, by weblogic.Deployer.
    • weblogic.Deployer is the recommended deployment tool for the WebLogic Server environment. 
  • WLDeploy
  • WLST[4,5,20]
    • WLST functionality includes the capabilities of the following WebLogic Server command-line utilities:
      • weblogic.Deployer
      • WLDeploy
  • JSR-88 API for deployment
    • WebLogic Deployment API,[19] which implements and extends the Java EE Deployment API specification (JSR-88)
    • The WebLogic Deployment API[19] classes and interfaces extend and implement the Java EE Deployment API specification (JSR-88) interfaces, which are described in the javax.enterprise.deploy sub-packages.[6]
  • JMX Deployment API[7]
    • Supports all of the common functionality available in the Java EE Deployment API specification (JSR-88). 
    • You can use the JMX API as an alternative to JSR-88 to perform deployment tasks on specified target servers
  • WLS console (through JSR-88 API)[8]
  • FMWC (through JSR-88 API and JMX Deployment API)[9]

Deployment Concepts

To help you understand the deployment process in WebLogic Server, we have listed the most important concepts below:
  • Deployable objects (or artifacts)
    • Which can be deployed on a weblogic domain
      • EJBs
      • Modules
        • Supported module types include:[11,12]
          • JMS, JDBC, Interception, Config, and WLDF[10]
      • Web Services
      • Applications[13]
    • Physical package types:
      •  EARs, standalone J2EE and non-J2EE modules
  • Deployment operations (or tasks)
    • Such as distributing, starting, stopping, deploying, redeploying, undeploying, etc.
      • deploying
        • if successful, an AppDeploymentMBean[14] for the application will be created 
  • Deployment targets
    • Target types available to WebLogic Server include:
      • Cluster
      • JMS Server
      • SAF Agent
      • Server
      • Virtual host 
  • Deployment order
    • An integer value that indicates when this artifact is deployed, relative to other deployable artifacts on a server, during startup.
    • Artifacts with lower values are deployed before those with higher values.
  • Deployment plan
    • You can use either Weblogic Console or weblogic.PlanGenerator to export portions of a WebLogic Server deployment configuration into a deployment plan file.[16,17]
    • Describes the application structure, identifies all deployment descriptors, and exports a subset of the application's configurable properties.  For example, it includes:
      • Resource dependencies
      • Resource declarations
      • Non-resource oriented configurable properties
      • Properties that may be changed in a running application, etc
    • A deployment plan is an XML document used to define an application's deployment configuration for a specific WebLogic Server environment, such as development, test, or production. 
    • A deployment plan resides outside of an application's archive file and contains deployment properties that override an application's existing Java Enterprise Edition and WebLogic Server deployment descriptors. 
    • Use deployment plans to easily change an application's WebLogic Server configuration for a specific environment, without modifying existing deployment descriptors.
  • Deployment configuration
    • Refers to the process of preparing an application or deployable resource for deployment to a WebLogic Server instance.
    • Modification of individual WebLogic Server configuration values based on user inputs and the selected WebLogic Server targets
  • Deployment staging
    • The staging mode of an application or deployment plan can be provided to affect its deployment behavior.
      • An application's deployment plan can be staged independently of the application archive, allowing you to stage a deployment plan when the application is not staged.
      • If you do not specify a staging mode, the deployment plan uses the value specified for application staging as the default.
    • Staging mode includes:
      • STAGE—Application files (or deployment plans) are copied to target servers
      • NO_STAGE—Application files (or deployment plans) are not copied to target servers
      • EXTERNAL_STAGE—Application files (or deployment plans) are copied manually to target servers

"Install Root" Abstraction

The SessionHelper[18] in WebLogic Deployment API[19] views an application and deployment plan artifacts using an "install root" abstraction, which ideally is the actual organization of the application. The install root appears as follows:

install-root (eg myapp) 
-- app 
----- archive (eg myapp.ear) 
-- plan
----- deployment plan (eg plan.xml) 
----- external descriptors (eg META-INF/weblogic-application.xml...) 

Optionally, you may consider using this layout to organize your applications and deployment plans.


  1. JSR 88: JavaTM EE Application Deployment
  2. WebLogic Server Deployment Features
  3. Understanding the WebLogic Deployment API (WebLogic 12.1.2)
  4. WLST FAQs
  5. WLST Command and Variable Reference
  6. Java EE 5 API Summary
  7. The JMX API for Deployment Operations
  8. Administration Console Online Help
  9. Using Fusion Middleware Control to Manage WebLogic Server
  10. The Configuration File in WebLogic Server Domain — config.xml (Xml and More)
  11. Module Types
  12. Example Module Deployment
  13. Oracle® Fusion Middleware Deploying Applications to Oracle WebLogic Server
  14. Interface AppDeploymentMBean
  15. MBean Reference for Oracle WebLogic Server
  16. Create a deployment plan using Administration Console
  17. Oracle WebLogic Server 12c: Creating and Using a Deployment Plan
  18. SessionHelper
  19. WebLogic Deployment API
  20. WLST: deploy command (WLS 12.1.2)
  21. Pack and Unpack Commands
    • Provide a simple, one-step method for creating WebLogic domains and templates from the command line.
    • You cannot, however, use these commands to customize the contents of your WebLogic domain or template in the same way as with the other tools (i.e., Configuration Wizard or WLST).
  22. Deployment Tools for Developers (WLS 12.1.1)

Sunday, November 23, 2014

G1 GC: Humongous Objects and Humongous Allocations

For Garbage First Garbage Collector (G1 GC), any object that is more than half a region size is considered a Humongous Object.   A humongous object is directly allocated into Humongous Regions.[1]

In this article, we will discuss the following topics:
  • Humongous regions and humongous allocations
  • How humongous allocations impact G1's performance?
  • How to detect humongous allocations?
    • Basic Investigation
    • Advanced Investigation

Humongous Regions and Humongous Allocations

A humongous object is allocated directly in the humongous regions. These humongous regions are a contiguous set of regions. StartsHumongous marks the start of the contiguous set and ContinuesHumongous marks the continuation of the set.

Before allocating any humongous region, the marking threshold is checked, initiating a concurrent cycle, if necessary. Dead humongous objects are freed at the end of the marking cycle during the cleanup phase also during a full garbage collection cycle (but, a new implementation has changed this; see the next section).

Since each individual set of StartsHumongous and ContinuesHumongous regions contains just one humongous object, the space between the end of the humongous object and the end of the last region spanned by the object is unused. For objects that are just slightly larger than a multiple of the heap region size, this unused space can cause the heap to become fragmented.

How Humongous Allocations Impact G1's Performance?

In the old implementation, humongous objects are not released until after a full concurrent marking cycle.[4]  This is far from ideal for many transaction-based enterprise applications that create short-lived (i.e., in the transaction scope) humongous objects such as ResultSet(s) generated from JDBC queries.  This results in the heap filling up relatively quickly and leads to unnecessary marking cycles to reclaim them.

A new implementation since:
java version "1.8.0_40-ea"
Java(TM) SE Runtime Environment (build 1.8.0_40-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b05, mixed mode)
handles humongous regions differently and can reclaim them earlier if they are short-lived (see [4] for details).

For investigating humongous allocations in G1, you should begin with the following minimal set of VM options:
  • -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintReferenceGC (-XX:+UseG1GC)

Basic Humongous Allocation Investigation

For basic humongous allocations investigation, you simply add a new option:
  • -XX:+PrintAdaptiveSizePolicy
This option will print out something like below:
12025.832: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 3660578816 bytes, allocation request: 1048592 bytes, threshold: 3650722120 bytes (85.00 %), source: concurrent humongous allocation]
From the above, we saw a humongous allocation is initiated with a request size of 1048592 bytes. Note that our region size is 1MBytes in this test. So, the request object is considered as humongous (i.e., 1048592 Bytes > half of 1 MBytes).

In the output, you can also find the following entries:
      [Humongous Reclaim: 0.0 ms]
         [Humongous Total: 54]
         [Humongous Candidate: 0]
         [Humongous Reclaimed: 0]
This shows that there are 54 humongous regions found in this GC event.   Also, none of them were reclaimed in this GC event. The reason is that the current algorithm is relatively conservative and it skips humongous objects for reclamation if there are sparse references into them.  In the case that those references belongs to sparse table entries, G1 may improve the reclamation rate by discovering if they are actually not referenced at all after iterating the table.  A performance enhancement is pending for this effort now.[5]

Advanced Humongous Allocation Investigation

To further investigate humongous allocation in more details, you can add:
-XX:+PrintAdaptiveSizePolicy -XX:+UnlockExperimentalVMOptions -XX:+G1ReclaimDeadHumongousObjectsAtYoungGC -XX:G1LogLevel=finest -XX:+G1TraceReclaimDeadHumongousObjectsAtYoungGC
Note that the following VM options are experimental:
  • G1TraceReclaimDeadHumongousObjectsAtYoungGC[4]
  • G1ReclaimDeadHumongousObjectsAtYoungGC[3]
  • G1LogLevel (i.e., [fine|finer|finest])
So, that's why we added the unlock option;
With extra printing options, you will find detailed description of humongous allocations as shown below:
3708.669: [SoftReference, 0 refs, 0.0000340 secs]3708.669: [WeakReference, 1023 refs, 0.0001790 secs]3708.669: [FinalReference, 295 refs, 0.0007020 secs]3708.670: [PhantomReference, 0 refs, 0.0000060 secs]3708.670: [JNI Weak Reference, 0.0000140 secs]Live humongous 1 region 9 size 65538 with remset 1 code roots 0 is marked 0 live-other 0 obj array 0

Live humongous 1 region 10 size 88066 with remset 1 code roots 0 is marked 0 live-other 0 obj array 0
Live humongous 1 region 15 size 262146 with remset 1 code roots 0 is marked 0 live-other 0 obj array 1
Finally, you can also use Java Flight Recorder (JFR) and Java Mission Control (JMC) to get information about humongous objects by showing their allocation sources with stack traces.[7]


  1. Garbage First Garbage Collector Tuning
  2. G1GC: Migration to, Expectations and Advanced Tuning
  3. Thread local allocation buffers (TLAB's)
  4. Early reclamation of large objects in G1
    • Introduced since 8u40 b02 and 8u45
    • Was not back ported to 8u20
  5. Early reclaim of large objects that are referenced by a few objects
  6. G1TraceReclaimDeadHumongousObjectsAtYoungGC
    • JDK-8058801
      • Prints information about live and dead humongous objects
        • (Before bug fix) prints that information when there is at least one humongous candidate.
        • (After bug fix) prints that information even when there is no humongous candidates; it also prints humongous object size in the output
  7. Java Mission Control (JMC) and Java Flight Recorder (JFR)
  8. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)
  9. Other JDK 8 articles on Xml and More

Saturday, November 22, 2014

Java SecurityManager: "" and "java.policy" Files

In many cases, where the threat model does not include malicious code being run in the JVM, the Java Security Manager[1] is unnecessary. However, when untrusted third-parties use WebLogic Server[2] and untrusted classes are being run, the Java Security Manager may be useful.

In this article, we will discuss the following topics:
  1. What is Java Security Manager?
  2. What is "" file?
  3. What is "java.policy" file?

Java Security Manager

The JVM has security mechanisms built into it that allow you to define restrictions to code through Java security policy files (i.e., java.policy file). The Java Security Manager uses these policies to enforce a set of permissions granted to classes. The permissions allow specified classes running in that instance of the JVM to allow or not allow certain runtime operations.

Therefore Java Security Manager can be used with WebLogic Server to provide additional protection for resources running in a Java Virtual Machine (JVM).[1] However, using a Java Security Manager is an optional security step.

To use the Java Security Manager with WebLogic Server, just specify:${WL_HOME}/server/lib/weblogic.policy
arguments when starting WebLogic Server. The argument specifies a filename (using a relative or fully-qualified pathname) that contains Java 2 security policies.[5] If weblogic.policy file is provided, then the specified policy file will be loaded in addition to all the policy files that are specified in the security properties file (see "java.policy" section).  If you instead type the following command options, using a double equals, then just the specified policy file will be used; all others will be ignored:${WL_HOME}/server/lib/weblogic.policy

A security model can be as simple as the one supported in Linux or Windows file system—file permissions are granted to principals based on user IDs. Or it can be as sophisticated as the one implemented in Adobe Flash Player (see [6] for details). Java Security Manager supports a security model sits in between the two in terms of complexity and capability.

Sometimes a Java application (like, say, a Web browser) needs to run untrusted code within itself. In this case, Java system libraries need some way of distinguishing between calls originating in untrusted code and calls originating from the trusted application itself. Clearly, the calls originating in untrusted code need to be restricted to prevent hostile activities. By contrast, calls originating in the application itself should be allowed to proceed (as long as they follow any security rules that the operating system mandates).

Security is all about separation by creating protection domains. Access control in Java Security Manager is enforced with the help of:[10]
  • Executable Code Identity
    • Decisions are based on the identity of the executable code
      • AccessController in package consults policy and uses stack inspection (see below) to decide whether to allow or disallow a call.
    • Executable code is categorized based on its URL of origin and the private keys used to sign the code:
      • Code source
        • Note that permission objects are associated with each code source
      • Signature
    • Trusted code vs untrusted code
  • Protected System Resource
    • File access
    • Network access (via sockets)
    • Listening ports, etc.
  • Permission (or privilege)
    • Permission is configure by:
      • Grant Target
        • Such as the name of a file
      • Grant Action
        • Such as "read" action on a file
    • Collection of Permissions
      • Homogeneous vs heterogeneous
  • Logical groupings
    • Separating groups from each other and granting groups particular permissions.

Stack Inspection

Java implements its security system by allowing security-checking code to examine the runtime stack for frames executing untrusted code. This is called stack inspection.[7] Java Security Manager inspects the stack in order to make access control decisions.

Each thread of execution has its own runtime stack. The stack grows and shrinks during typical program operation. Each stack frame includes both a method call and a trust label (T for trusted, U for untrusted).

Security Properties File Location

The security properties file is located in the file named

java.home/lib/security/ (Solaris/Linux/Mac OS X)
java.home\lib\security\ (Windows)


Java security can be dictated by security policies. A policy file can be composed via a simple text editor, or via the graphical Policy Tool utility.[8] There is by default a single system-wide policy file, and a single (optional) user policy file. Policy can be set by the user (usually a bad idea) or by the system administrator, and is represented in the class

The configuration file(s) specify what permissions are allowed for code from a specified code source, and executed by a specified principal. Each configuration file must be encoded in UTF-8.[4]

Policy File Location

Policy file locations are specified in the security properties file (i.e.,  For example, the default system and user policy files are defined using URL specification in the security properties file as:


System Policy File

The system policy file is meant to grant system-wide code permissions. It is by default located at

java.home/lib/security/java.policy (Solaris/Linux/Mac OS X) 
java.home\lib\security\java.policy (Windows)

The java.policy file installed with the JDK grants all permissions to standard extensions, allows anyone to listen on un-privileged ports, and allows any code to read certain "standard" properties that are not security-sensitive, such as the "" and "file.separator" properties.

Executable code is categorized based on its URL of origin and the private keys used to sign the code. The security policy maps a set of access permissions to code characterized by particular origin/signature information. Protection domains can be created on demand and are tied to code with particular CodeBase and SignedBy properties.

Code can be signed with multiple keys and can potentially match multiple policy entries. In this case, permissions are granted in an additive fashion


Tuesday, November 18, 2014

WLST: Offline and Online Modes

In this article, we will cover the following topics:
  • How to invoke WLST?
  • Offline and Online WLST
  • How to switch from Offline to Online mode?

How to invoke WLST?

In a Windows environment, you can start WLST from the Start program by simply selecting:
Start | Programs | Oracle | Oracle Home | WebLogic Server 12c | Tools | WebLogic Scripting Tool
You can also invoke WLST from the command line by using:[1]
  • The java weblogic.WLST command or 
  • The command script wlst.cmd ( in UNIX)

java weblogic.WLST Command

D:\>cd Oracle\wls1036tx\user_projects\domains\base1_domain\bin
D:\Oracle\wls1036tx\user_projects\domains\base1_domain>java weblogic.WLST

Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands


Command Script wlst.cmd

D:\>cd Oracle\wls1036tx\oracle_common\common\bin
D:\Oracle\wls1036tx\user_projects\domains\base1_domain\bin> wlst.comd

Initializing WebLogic Scripting tool (WLST)...
Type help() for help on available commands


Note that you use the wlst.cmd script from the ORACLE_HOME\oracle_common\common\bin directory and not a directory specific to any particular WebLogic Server domain.[1]

Offline and Online WLST

As shown above, both WLST invocations bring you to the offline mode.  In the next section, we will show how to switch from offline to online mode and vice versa.  To begin with, here are what offline and online mode can and can't do:[5]

WLST Offline
WLST Online
Can Do
  • Create/modify domain templates
  • Create domains
  • Extend an existing domains by access and modify the configuration for an offline domain
  • Change configuration when servers are in RUNNING state
  • View runtime data for monitoring various runtime MBeans performances
  • Deploy applications targeting to Servers or clusters
  • Start and stop servers or cluster members.
Can't Do
  • View runtime performance data
  • Modify security data
  • Create a new domain (must be offline mode)
    • You can only update an existing domain configuration.

Online Mode

In online mode, WLST needs to connect to an active Admin or Managed Server.  The WLST online commands are analogous to the Administration Console changes after connecting to Admin Server.  WLST in online mode acts as a Java Management Extensions (JMX) client that manages the domain's resources by modifying the server's in-memory runtime Management Beans (MBeans). Thus, WLST offers you all the domain management configuration capabilities as the Administration Console.

Offline Mode

In offline mode, WLST helps you extend a domain, create domain templates, and create domains with those templates. Since you aren't connected to an active Admin Server, you won't be able to modify domain configuration in offline mode.

The WLST offline configuration commands are analogous to the Configuration Wizard. You just need to know the navigation on the configuration MBean trees, modify the MBean attribute values. Internally WLST offline commands work on WebLogic domain Configuration Framework, which Configuration Wizard also uses.  Using either tool, modified configuration data are persisted consistently in  the domain’s config directory or in a domain template JAR.

You can list a summary of all online and offline commands from the command-line using the following commands, respectively:[3,4]

How to Switch from Offline to Online mode?

You can switch from offline to online mode and vice versa by using connect() and disconnect() commands. When you use connect() command at offline it will transfer your prompt to ServerConfig that indicates online command mode.  The following example shows you how:
wls:/offline> connect('weblogic', 'welcome1', 't3://myserver:7001')

Connecting to t3://myserver:7001 with userid weblogic ...
Successfully connected to Admin Server "myserver" that belongs to domain "mydomain".

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

wls:/mydomain/serverConfig> disconnect()

Disconnected from weblogic server: myserver

In order to...
Use this command...
Use with WLST...
Connect to and disconnect from a WebLogic Server instance
Connect WLST to a WebLogic Server instance.
Online or Offline
Disconnect WLST from a WebLogic Server instance.