Saturday, December 13, 2014

Java Security Manager: access denied

The JVM has security mechanisms built into it that allow you to define restrictions to code through Java security policy files (i.e., java.policy file). The Java Security Manager[1] uses these policies to enforce a set of permissions granted to classes. The permissions allow specified classes running in that instance of the JVM to allow or not allow certain runtime operations.

In this article, we will cover how to resolve security violations such as access denied
when you start WebLogic Server with Java Security checking enabled (see [1] for how to enable Java Security Manager).

AccessController is the main security enforcer in Java Security Manager.  It helps decide whether an access to a critical system resource is to be allowed or denied, based on the security policy currently in effect,  For example, an will be thrown when a permission was not granted to read property as below:

Caused by: access denied ("java.util.PropertyPermission" "" "read")
at java.lang.SecurityManager.checkPermission(
at java.lang.SecurityManager.checkPropertyAccess(
at java.lang.System.getProperty(
at java.lang.Boolean.getBoolean(

If the security violations in your application are only a few, you may consider to add an entry like below in the default section of your policy file:

// default permissions granted to all protection domains
grant {
    permission java.util.PropertyPermission "", "read";

"weblogic.policy" and "ojdbc.policy"

When you run applications in WebLogic server, Oracle has provided an OOTB policy file at:
If you specify it in the JVM option:${WL_HOME}/server/lib/weblogic.policy
then the specified policy file will be loaded in addition to all the java policy files (see [1] for details). However,  OOTB weblogic.policy is not intended to cover all deployed applications.  For example, if you use Oracle JDBC Driver in your applications, you may need to add the grant entries specified in ojdbc.policy[2] onto the ones specified in weblogic.policy file.

Grant Entries

If you look at either weblogic.policy or ojdbc.policy file, there are many grant entries. The basic format of a grant entry is the following:[4]

  grant signedBy "signer_names", codeBase "URL",
        principal principal_class_name "principal_name",
        principal principal_class_name "principal_name",
        ... {
      permission permission_class_name "target_name", "action", 
          signedBy "signer_names";
      permission permission_class_name "target_name", "action", 
          signedBy "signer_names";

Each grant entry includes one or more "permission entries" preceded by optional codeBase, signedBy, and principal name/value pairs that specify which code you want to grant the permissions.

Sometimes it will be too much to enumerate all permissions individually.  This is the time for granting permissions by any combination of the following entries:
  • SignedBy
  • Principal
  • CodeBase

Grant Permission by SignedBy

An example of granting permission(s) based on signer is shown below:[3]
keystore "kim.keystore";

// Here is the permission ExampleGame needs.
// It grants code signed by "terry" the
// HighScorePermission, if the
// HighScorePermission was signed by "chris"
grant SignedBy "terry" {
      "ExampleGame", signedBy "chris";
A signedBy value indicates the alias for a certificate stored in the keystore. The public key within that certificate is used to verify the digital signature on the code; you grant the permission(s) to code signed by the private key corresponding to the public key in the keystore entry specified by the alias.

The signedBy value can be a comma-separated list of multiple aliases. An example is "Adam,Eve,Charles", which means "signed by Adam and Eve and Charles"; the relationship is AND, not OR. To be more exact, a statement like "Code signed by Adam" means "Code in a class file contained in a JAR which is signed using the private key corresponding to the public key certificate in the keystore whose entry is aliased by Adam".

Grant Permission by Principal

An example of granting permission(s) based on principal is shown below:
// Grant notification listener actions to standard roles

grant principal "Administrators" {
    permission "*", "addNotificationListener";
    permission "*", "removeNotificationListener";

A principal value specifies a class_name/principal_name pair which must be present within the executing thread's principal set. The principal set is associated with the executing code by way of a Subject.

The principal_class_name may be set to the wildcard value, *, which allows it to match any Principal class. In addition, the principal_name may also be set to the wildcard value, *, allowing it to match any Principal name. When setting the principal_class_name or principal_name to *, do not surround the * with quotes. Also, if you specify a wildcard principal class, you must also specify a wildcard principal name.

Grant Permission by CodeBase

A codeBase value indicates the code source location; you grant the permission(s) to code from that location. An example of granting permission by CodeBase is shown below:
// Grant for internal applications when using WebLogic startup scripts
grant codeBase "file:${user.dir}/servers/${weblogic.Name}/tmp/_WL_internal/-" {

BNF Grammar

An informal BNF grammer for the Policy file format is given below, where non-capitalized terms are terminals:[12]

PolicyFile -> PolicyEntry | PolicyEntry; PolicyFile
PolicyEntry -> grant {PermissionEntry}; |
           grant SignerEntry {PermissionEntry} |
           grant CodebaseEntry {PermissionEntry} |
           grant PrincipalEntry {PermissionEntry} |
           grant SignerEntry, CodebaseEntry {PermissionEntry} |
           grant CodebaseEntry, SignerEntry {PermissionEntry} |
           grant SignerEntry, PrincipalEntry {PermissionEntry} |
           grant PrincipalEntry, SignerEntry {PermissionEntry} |
           grant CodebaseEntry, PrincipalEntry {PermissionEntry} |
           grant PrincipalEntry, CodebaseEntry {PermissionEntry} |
           grant SignerEntry, CodebaseEntry, PrincipalEntry {PermissionEntry} |
           grant CodebaseEntry, SignerEntry, PrincipalEntry {PermissionEntry} |
           grant SignerEntry, PrincipalEntry, CodebaseEntry {PermissionEntry} |
           grant CodebaseEntry, PrincipalEntry, SignerEntry {PermissionEntry} |
           grant PrincipalEntry, CodebaseEntry, SignerEntry {PermissionEntry} |
           grant PrincipalEntry, SignerEntry, CodebaseEntry {PermissionEntry} |
           keystore "url"
SignerEntry -> signedby (a comma-separated list of strings)
CodebaseEntry -> codebase (a string representation of a URL)
PrincipalEntry -> OnePrincipal | OnePrincipal, PrincipalEntry
OnePrincipal -> principal [ principal_class_name ] "principal_name" (a principal)
PermissionEntry -> OnePermission | OnePermission PermissionEntry
OnePermission -> permission permission_class_name
                 [ "target_name" ] [, "action_list"]
                 [, SignerEntry];

Some entries in the grammar are optional, If they are omitted, it signifies:

When Omitted
It Means
CodebaseEntry "any code base" (it doesn't matter where the code originates from)
SignerEntry "any signer" (it doesn't matter whether the code is signed or not or by whom)
PrincipalEntry "any principals"

The "target_name"is the name of the permission is aimed.  For, the targets of this class can be specified as:

directory (same as directory/)
directory/* (all files in this directory)
* (all files in the current directory)
directory/- (all files in the file system under this directory)
- (all files in the file system under the current directory)
"<<ALL FILES>>" (all files in the file system)

As an example, you may grant read permission to all files in the file system as below:
permission "<<ALL FILES>>", "read"; 

Finally, a set of actions can be specified together as a comma-separated composite string as below:
permission "WEBLOGIC-APPLICATION-ROOT${/}-", "read, write, delete, execute";


Sunday, December 7, 2014

Linux: How to Load Balance with DNS Round Robin

If you are running multiple application server instances, you will need to use something to balance the HTTP traffic from the network. For example, you can use a DNS server or a hardware load balancer (e.g. F5) to load balance HTTP traffic.

In this article, we will cover how to configure a DNS server to round-robin network traffic across all application servers.

Load Balancing with DNS Round Robin

Domain Name Service (DNS) is an Internet service that map human-memorable domain names and hostnames into the corresponding numeric Internet Protocol (IP) addresses which are used to identify and locate computer systems and resources on the Internet.  In this way, DNS alleviates the need to remember IP addresses. Computers that run DNS are called name servers. Both Red Hat and Ubuntu ship with BIND (Berkley Internet Naming Daemon),[12] the most widely deployed DNS server.

Using a DNS server, even with one application server, makes it very easy if you need to change to different machines — you just need to update the IP address on the DNS server instead of on every other machine.

Load balancing allows organizations to distribute inbound traffic across multiple back-end nodes or servers (represented as the IP address of a physical server). Round robin is a local balancing mechanism used by DNS servers to share and distribute network resource loads.  Compared to hardware load balancer, DNS round robin is a poor man’s load balancing solution.[1]

Configuring Name Servers

Managing BIND's named daemon is easy to do, but the procedure differs between Linux distributions. Here are some things to keep in mind:
  • Firstly, different Linux distributions use different daemon management systems. Each system has its own set of commands to do similar operations. 
  • Secondly, the daemon name needs to be known. In this case the name of the daemon is named.
Armed with this information you can know how to:
  • Start your daemons automatically on booting 
  • Stop, start and restart them later on during troubleshooting or when a configuration file change needs to be applied.
For more details, read [11] and [12].  The Linux release used in this article is:

$ cat /etc/*-release
Enterprise Linux Enterprise Linux Server release 5.4 (Carthage)
Red Hat Enterprise Linux Server release 5.4 (Tikanga)

Assume that your named is up and running—To enable DNS services on the name server, the /etc/host.conf file should look like this:

# Lookup names via /etc/hosts first, then by DNS query
order hosts, bind

Next you need to configure DNS tables for DNS services. If you use the BIND 8.x package with the Red Hat distribution, the configuration of DNS tables can be done in:
  • /etc/named.conf
For example, the /etc/named.config could look like this:

options {
  // DNS tables are located in the /var/named directory
  directory "/var/named";
  dump-file "/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  listen-on { any; };
  allow-query { any; };
  rrset-order {
    order cyclic;

controls {
  inet allow {

zone "." {
  type forward;
  forward first;
  forwarders {;

// All our DNS information is stored in /var/named/foobar
zone "" {
  type master;
  notify no;
  file "";

The rrset-order substatement accepts fixed, random and cyclic as arguments. By specifying cyclic as the argument, you can configure the name server to return matching records in cyclic (round robin) order.[7]

The DNS table named /var/named/ translates host names (such as jassut) to IP addresses. could have the contents like this:[8]

; Zone file for
@ IN SOA (
   2005061504 ; serial # (today's date + increment)
   1S ; refresh
   1S ; retry
   1S ; expire
   1S) ; minumum ttl
; sorted alphnumerically by hostname (first column) then by IP address

nameserver A

jassut A
jasapp1dom1inst1nic1 A
jassut A

jasapp1dom1inst1nic2 A
jassut A

jasapp1dom1inst1nic3 A
jassut A

jasapp1dom1inst1nic4 A

Configuring DNS Clients

There are three main client configuration files associated with DNS:[9]
  • /etc/hosts
  • /etc/nsswitch.conf
  • /etc/resolv.conf
When your computer looks for another computer on a network such as the Internet, it typically looks in two places: /etc/hosts and any DNS servers that you've set up for your network.


The /etc/hosts file keeps a local name database. This file helps in local name resolution if your local DNS server is not functioning. Network adinistrators should manually populate entries in this file. A sample /etc/hosts file is copied below.

$ cat /etc/hosts

# Do not remove the following line, or various programs
# that require network functionality will fail. localhost.localdomain localhost clientserver1


The order of name resolution process is determined by a single line in /etc/nsswitch.conf:

hosts: files,dns

In this case, the name resolution process will begin with checking /etc/hosts file, and if the name cannot be resolved, the name resolution will happen with the DNS server.


The /etc/resolv.conf file contains directives with the IP addresses of nameservers available to a host. A sample /etc/resolv.conf file is copied below.

$ cat /etc/resolv.conf
options attempts: 5
options timeout: 15


How to Test?

nslookup is a network administration command-line tool available for many computer operating systems for querying the DNS server(s) to obtain domain name or IP address mapping or for any other specific DNS record.

For example, our sample setup uses four server instances in one WebLogic domain, which listens on four Network Interface Cards.

On the client computer, you can query host name jassut by typing:
$ nslookup jassut






Other commands for debugging DNS besides nslookup are host and dig. Note that ping is not useful for debugging DNS, as it will use whatever is configured in /etc/nsswitch.conf to do the name-lookup.

DNS Caching

The Internet's Domain Name System (DNS) involves caching on both DNS servers and on the client computers that contact DNS servers.  To minimize the load on the actual DNS server, we want to use a local DNS cache. There are two levels of local name caching on DNS clients:
  • OS
    •  To enable DNS caching on the client computers, you can install nscd and run it.
      • A daemon that provides a cache for the most common name service requests.
      • To launch the daemon, do:
        • /etc/init.d/nscd start
  • JVM
    • By default, the JVM will cache DNS information.
However, if client-side DNS caching is enabled, it can lessen the effectiveness of the round-robin DNS load balancing.  In other words, due to the client-side DNS query caching, your round-robin DNS load balancing is not likely to distribute the load on your remote servers on a per-request basis. It will be more like a per-session basis.

To effectively support DNS round robin, you want to:
  • Disable JVM local caching by setting
    • -Dnetworkaddress.cache.ttl=0
  • Decrease the local DNS cache (i.e., nscd) default time-to-live to the minimum of 1 second (see instructions below).

On Windows, under HKEY_LOCAL_MACHINE\System\CurrentControlSet set Services\Dnscache\Parameters\MaxCacheEntryTtlLimit to 1

For UNIX machines with a name service cache daemon (nscd),[5] ensure that /etc/nscd.conf contains

enable-cache hosts yes
positive-time-to-live hosts 1
negative-time-to-live hosts 0


  1. Configuring DNS round robin on Windows
  2. Load Balancing With Round Robin DNS
  3. port 53
    • Default port listened by DNS
  4. Domain Name Server (DNS) Configuration and Administration
    • BIND 8.x
      • /etc/named.conf file should be configured to point to your DNS tables
    • BIND9 (Berkley Internet Naming Daemon)
      • BIND9 Configuration files are stored in /etc/bind
  5. nscd (Name Service Cache Daemon)The default configuration file, /etc/nscd.conf, determines the behavior of the cache daemon
    • See also: nsswitch.conf - System Databases and Name Service Switch configuration file
  6. The Domain Name System (good)
  7. Round Robin Load Distribution
  8. Zone files (Red Hat Linux 5)
  9. Linux Domain Name System (DNS) client configuration files
  10. Linux / Unix: Dig Command
  11. Quick HOWTO : Ch18 : Configuring DNS (good)
  12. Berkeley Internet Name Domain (BIND) (Red Hat Linux 5)
  13. Why 192.168.*.* for local addresses?
  14. Load Balancing T3 InitialContext Retrieval for WebLogic using Oracle Traffic Director
    • For bootstrapping the InitialContext, you can choose two approaches:
      • DNS round robin
      • External load blancer such as OTD