Tuesday, January 29, 2013

java.lang.NoClassDefFoundError: sun/io/CharacterEncoding

sun.io.CharacterEncoding will be removed in JDK 8.  In our Java test using JDK 8, we have found the following message:
  • java.lang.NoClassDefFoundError: sun/io/CharacterEncoding
when WebLogic Server (WLS) started.

In this article, we will discuss how to trouble shoot and resolve this issue.

JVM Option -verbose


When it comes to java.lang.NoClassDefFoundError exception, one thing we need to find out this who has thrown that exception.  To trouble shoot class loading issues, the following JVM option comes in handy:
  • -verbose
After we add -verbose option, we have found the following messages from the WLS output:

[Loaded oracle.webservices.annotations.PortableWebService from file:/scratch/perfgrp/mt/rup1/fusionapps/oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar]
[Loaded oracle.j2ee.ws.common.util.TestPageUtils from file:/scratch/perfgrp/mt/rup1/fusionapps/oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar]


[Loaded java.lang.UnsatisfiedLinkError from /scratch/perfgrp/JVMs/jdk1.8.0/jre/lib/rt.jar]
[Loaded java.lang.VerifyError from /scratch/perfgrp/JVMs/jdk1.8.0/jre/lib/rt.jar]
[Loaded com.bea.logging.ThrowableWrapper from file:/scratch/perfgrp/mt/rup1/fusionapps/modules/com.bea.core.logging_1.9.0.0.jar]

java.lang.NoClassDefFoundError: sun/io/CharacterEncoding
        at oracle.j2ee.ws.common.util.TestPageUtils.encode(TestPageUtils.java:216)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.createTestPagePath(WebServiceOperation.java:276)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.initialize(WebServiceOperation.java:253)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.(WebServiceOperation.java:244)
        at oracle.j2ee.ws.server.provider.GenericProviderInterceptorPipeline.registerWebServiceOperationMBean(GenericProviderInterceptorPipeline.java:175)
        Truncated. see log file for complete stack trace

After some investigation, we finally identify that TestPageUtils class from wsclient-rt.jar is the culprit. In JDK 8, the following class:
  • sun.io.CharacterEncoding 
will be removed.  It should be rewritten using:
  • java.nio.charset.Charset.forName()
For example, here are our changes:
//String encoding = sun.io.CharacterEncoding.aliasName(givenEncoding.toUpperCase());
String encoding = java.nio.charset.Charset.forName(givenEncoding.toUpperCase()).name();

How to Patch wsclient-rt.jar with New Class?


After we got a fix for TestPageUtils class, we loaded the jar with new class.  Note that from the -verbose output, we also know where TestPageUtils was loaded from.  For instance, it was loaded from:

  • /.../oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar

Here are the steps that we have taken to apply the patch:
  1. cd /.../oracle_common/modules/oracle.webservices_11.1.1
  2. mkdir tmpdir
  3. cd tmpdir
  4. cp ../wsclient-rt.jar .
  5. unzip wsclient-rt.jar
  6. cp <location of new patch>/TestPageUtils.class oracle/j2ee/ws/common/util
  7. jar uf /.../oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar oracle/j2ee/ws/common/util/TestPageUtils.class
  8. cd ..
  9. rm -rf tmpdir

References

Sunday, January 27, 2013

Creating the Required Schemas for Oracle Fusion Middleware Using RCU

To create required schema for Oracle Fusion Middleware[1], you can use Repository Creation Utility (RCU)[2]. You can also use RCU to drop component schemas.  In this article, we will show you how to create schema for the component named OID and discuss the significance of schema prefix.

Repository Creation Utility


Repository Creation Utility (RCU) is a Java tool to create database schema for component schemas that are part of Oracle Fusion Middleware.  RCU is available only on 32-bit x86 Linux and 32-bit Microsoft Windows Operating System platforms. You can run RCU from these machines to connect to any certifed versions of Oracle, or Microsoft SQL Server database, in order to create the schemas required by Fusion Middleware components.

Preparations and Prerequisites


If you are creating schemas on an Oracle database, you must use a user with SYSDBA privileges such as SYS[3].

Here is the outline of instructions of creating new schemas:
  1. Start up your database instance (i.e., ATGOID)
  2. Start tns listener
  3. Download the RCU and unzip it
  4. Open the RCU by invoking rcu in the bin folder
  5. Create component schemas as shown in the next section


Steps


The following sequence takes place when a schema is created with RCU (note that highlighted portion is the option that we have chosen for our installation):
  1. Welcome
  2. Create Repository
    • Create
      • Create and load component schemas into a database
    • Drop
      • Remove component schemas from a database
  3. Database Connection Detail
    • See Figure 1
    • Make sure prerequisite step 1 &2 were executed first.  Otherwise, you will see the following messages:
      • Unable to connect to the database using the provided details.
        Please enter a valid hostname and port or check if the listener is up and running.
    • When you click on Next, Checking Prerequisites window will be displayed (see Figure 1)
      • Prior to the schema being created, RCU performs global and component level prerequisite checks to ensure that certain minimum requirements are met.
    • You may see warning such as:
      • The database you are connecting is with non-AL32UTF8 character set. Oracle strongly recommends using AL32UTF8 as the database character set.
  4. Select Components
    • See Figure 2
      • Select an existing Prefix
      • Create a new Prefix: (Leave it empty; see why in next section)
    • When you click on NExt, Checking Component Prerequisites window will be popped up
  5. Schema Password
    • Enter the passwords for the main and auxiliary schema users.
      • Use same passwords for all schemas
      • Use main schema pssswords for auxiliary schemas
      • Specify different passwords for all schemas
    • Component
      • Oracle Internet Directory (Owner: ODS)
      • Auxiliary Schema (Owner: ODSSM)
  6. Map Tablespaces
    • Choose tablespaces for selected components (Default)
  7. Summary
    • Database details
      • Host Name: myserver
      • Port: 1521
      • Service Name: ATGOID
      • Connected As: sys
      • Operation: Create
      • Prefix for (non-prefixable) Schema Owners : DEFAULT_PREFIX
      • Component
        • Name: Oracle Internet Directory
        • Schema Owner: ODS
        • Tablespace Name:
          • Default: OLTS_DEFAULT
          • Temp: IAS_TEMP
          • Additional:
            • OLTS_ATTRSTORE
            • OLTS_BATTRSTORE
            • OLTS_CT_STORE
            • OLTS_SVRMGSTORE
  8. Completion Summary
    • RCU Logfile: $ORACLE_BASE/logdir.2013-01-15_11-03/rcu.log
    • Component Log Directory: $ORACLE_BASE/logdir.2013-01-15_11-03
    • Status: Success

Schema Prefix


You can use RCU to create multiple schemas of each component using custom prefixes. The prefix is prepended to and separated from the schema name with an underscore (_) character, as shown below:
  • prefix_schemaname
However, the Oracle Internet Directory (ODS) component cannot be prepended with a custom prefix because there can only be one repository for this component per database.  That's why we said that leave the new prefix empty at step 4 above.

The default prefix used by RCU is DEV.  If DEV has already been used, then RCU will default to DEV1, then DEV2, and so on. Prefixes are used to create and organize logical groups of schemas. For example, you may want to create a test version of the Metadata Services (schema name MDS) called TEST_MDS.  Then, when are ready for your production version, you can create a second version of the schema called PROD_MDS. Both TEST_MDS and PROD_MDS may reside on the same or separate databases.

You are only allowed to use a prefix once per schema within a single database. For example, if you had a version of the Metadata Services schema called DEV_MDS, then you can not use the DEV prefix again to create another version of the Metadata Services schema (for example, DEV_MDS2).  If you want to create another version of the schema using the same prefix, you must first drop the existing schema and then create the schema again.

Finally, the mapping between the prefixes and schemas is maintained in schema_version_registry.


SQL> select comp_id,mrc_name from SCHEMA_VERSION_REGISTRY;
COMP_ID                        MRC_NAME                       
------------------------------ ------------------------------ 
OID                            DEFAULT_PREFIX                 
ORASDPM                        DEV                            
SOAINFRA                       DEV                            

3 rows selected

References


  1. Oracle Fusion Middleware Articles
  2. Repository Creation Utility Overview
  3. What’s the Difference between the SYS and SYSTEM Schemas?
  4. Using Custom Prefixes
  5. Oracle Identity and Access manager 11g for Administrators

Saturday, January 26, 2013

Memlock limit too small: 32768 to accommodate segment size: 4194304

In our database alert_<sid>.log, we have found the following warning message:
  • Memlock limit too small: 32768 to accommodate segment size: 4194304
For Oracle to lock shared memory for the shared pool, memlock limit (i.e., locked-in-memory address space) must be large enough.  For example, our system has the following maximum locked-in-memory address space:

# -l     The maximum size that may be locked into memory
$ ulimit -l    
32

Note that the "memlock" figure is specified in Kb.  From the warning, we know our memlock limit needs to be larger than 4194304 bytes.  In reality, it is better to oversize it a little.


How to increase memlock limit on Linux?


Unix operating system can enforce "Limits" the resources a process/user can consume.  Memlock is one of the resource.  On Linux systems, you can adjust the "memlock" parameter in the 

  • /etc/security/limits.conf


For example, this is what we have specified to correct the "memlock-limit-too-small" issue:

#vi /etc/security/limits.conf

@perfgrp    soft    memlock         14680064
@perfgrp    hard    memlock         14680064
*           soft    nofile          65535
*           hard    nofile          65535


In general, individual limits have priority over group limits, so if you impose no limits for admin group, but one of the members in this group have a limits line, the user will have its limits set according to this line. In our case, we specify the group limit for "perfgrp" to be 14680064 bytes. For other users, default values will be applied, which is 65535 bytes. Finally, you need to reboot to make new settings effective.

Consideration for Java Applications


Similar to database software, you also have the same tuning requirement for application running on HotSpot. For example,
  • For all users who will run HotSpot with large pages, you need to set their memlock limit to a value higher than the maximum heap they will run.  This ensures user running the Java application can lock the correct amount of memory.
Of course, the memlock limit to be considered should be based on how much physical memory you have on your systems.  See [1] for details.

References

  1. HugePages Configuration and Monitoring
  2. Large SGA On Linux
  3. How I Simplified Oracle Database Installation on Oracle Linux (Good)
  4. How to Test Large Page Support on Your Linux System

Friday, January 25, 2013

UnicastUdpSocket failed to set receive buffer size to 1428 packets

The following messages have been found from the WebLogic Server output:

<Jan 24, 2013 2:32:04 PM PST> <Warning> <Coherence> <BEA-000000> <2013-01-24 14:32:04.325/37.402 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): UnicastUdpSocket failed to set receive buffer size to 1428 packets (2096304 bytes); actual size is 357 packets (524288 bytes). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.>

As warned: if you proceed with the actual receive buffer value, it may cause sub-optimal performance for your application. In this case, it's our OAM server.

What's OAM Server?


Oracle Access Manager Server[2] is a runtime engine used to provide shared services for access such as Authentication/Authorization service, session management, token processing, and single sign-on. OAM Server runs on one of WebLogic's Managed servers, such as oam_server1.

How to Fix the Issue?


To fix the issue, you can increase TCP max buffer size.  For example, our original values were:

net.core.rmem_max=524288
net.core.wmem_max=524288

After increasing both read and write buffer to bigger size (note that they are in bytes) as follows:

net.core.rmem_max = 4194304
net.core.wmem_max = 4194304

We don't see the above warning anymore.  You can find the above TCP buffer sizes in:

  • /etc/sysctl.config

After login as a root, you can modify them and issue
  • sysctl -p 
to enable sysctl data immediately.


Similarly you can issue the following Linux commands to achieve the same effects:
  • sysctl -w net.core.rmem_max=4194304
  • sysctl -w net.core.wmem_max=4194304

As suggest in [3], you may want to tune other TCP parameters for better networking performance.

References

  1. 'UdpSocket failed to set receive buffer size to 1428 packets' Error Due to Coherence Tuning
  2. Oracle Identity and Access Manager 11g for Administrators
  3. Linux Tuning








Sunday, January 20, 2013

ORA-03113: end-of-file on communication channel

This article is a follow-up to the previous article:
As pointed out in that article, there are limitations and issues with cloning a database. This article describes one of the issues (see also [234]):
  • ORA-03113: end-of-file on communication channel

ORA-03113


When we tried to startup a cloned database, we saw the following message:

SQL> startup;
ORACLE instance started.

Total System Global Area 1.2827E+10 bytes
Fixed Size                  2240344 bytes
Variable Size            1811939496 bytes
Database Buffers         1.0972E+10 bytes
Redo Buffers               40890368 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 9809
Session ID: 5339 Serial number: 3

It turns out that database has been mounted OK.  But, database failed when it was opened.

ALTER DATABASE OPEN Failed


Oerr is an Oracle utility that extracts error messages with suggested actions from the standard Oracle message files.  Here is the suggested actions from oerr:

$ oerr ORA 03113
03113, 00000, "end-of-file on communication channel"
// *Cause: The connection between Client and Server process was broken.
// *Action: There was a communication error that requires further investigation.
//          First, check for network problems and review the SQL*Net setup. 
//          Also, look in the alert.log file for any errors. Finally, test to 
//          see whether the server process is dead and whether a trace file
//          was generated at failure time.


One of the suggested action is to look in the alert.log file.  To find the location of alert.log, you do:
  1. Find the location of diagnostic destination from the initialization parameter file (i.e., dbs/init<sid>.ora ).  For example, we have this entry:
    • diagnostic_dest=/slot/fiz7865/log
  2. From there, you can find alert.log file in the following sub directory:
    • <diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace

In the alert_<sid>.log, we have found the following messages:

ALTER DATABASE OPEN
Errors in file /slot/fiz7865/log/diag/rdbms/fiz7865/fiz7865/trace/fiz7865_lgwr_9793.trc:
ORA-00338: log 2 of thread 1 is more recent than control file
ORA-00312: online log 2 thread 1: '/data1/rup3.redolog/log2.dbf'
Errors in file /slot/fiz7865/log/diag/rdbms/fiz7865/fiz7865/trace/fiz7865_lgwr_9793.trc:
ORA-00338: log 2 of thread 1 is more recent than control file
ORA-00312: online log 2 thread 1: '/data1/rup3.redolog/log2.dbf'
Errors in file /slot/fiz7865/log/diag/rdbms/fiz7865/fiz7865/trace/fiz7865_ora_9809.trc:
ORA-00338: log 1 of thread  is more recent than control file
ORA-00312: online log 2 thread 1: '/data1/rup3.redolog/log2.dbf'
USER (ospid: 9809): terminating the instance due to error 338

What Happened?


As stated in ORA-00338, our redo log file is more recent than control file.  This happened because we have cloned control files first and then cloned redo files only after we have found that they were missing.  Note that in between two cloning, we have also restarted our source database.  So, after we have synchronized the control files and the redo files, we were able to start up database successfully.  In the alert.log file, you should find the following messages:

ALTER DATABASE OPEN
Thread 1 opened at log sequence 371
  Current log# 2 seq# 371 mem# 0: /data1/rup3.redolog/log2.dbf
Successful open of redo thread 1

References

  1. Simplify Cloning by Using Hosts File
  2. ORA-00313: open failed for members of log group 1 of thread 1
  3. Using rsync to Clone Local and Remote Systems
  4. ORA-01031: insufficient privileges
  5. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
    • This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
    • Oracle Fusion Middleware provides a series of scripts for this task.
  6. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
    • For your Oracle production systems, follow official recommendations as shown in this article.

Tuesday, January 15, 2013

Using Database Configuration Assistant to Create Additional Databases

During Oracle Database installation, if you choose to create a starter database (see also [1]), Oracle Universal Installer automatically runs Database Configuration Assistant (DBCA) after software installation is complete. DBCA then guides you through the steps to create the database.

If you do not create a starter database and later want to create one, or if you want to create additional databases, then you use DBCA.  The Database Configuration Assistant enables you to
  • Create a database
  • Configure database options in an existing database
  • Delete a database
  • Manage database templates

There may be situations in which you want to create more than one Oracle database on the same host 
computer. When you do this with DBCA, the new databases typically  use the same Oracle home directory as the first database, but store database data files separately from those of the first database.

In [1], we describe the steps that we used to install a starter database with a Global Database Name (GDNSTAGE7.  In this article, we will show how to add a new database (GDN: ATGOID) in the same Oracle home directory.


Steps


Here are our steps to create a new database named ATGOID using DBCA.  Note that highlighted options are our selections.  For more detailed descriptions, read [2].
  1. Welcome
  2. Operations
    • Select the operation that you want to perform:
      • Create a Database
      • Configure Database Options
      • Delete a Database
      • Manage Tempaltes
    • ASM configuration operations must be performed using Automatic Storage Management Configuration Assistant (ASMCA) from Oracle Grid Infrastructure home.
  3. Database Templates
    • Select template to use for creation
      • General Purpose or Transaction Processing
      • Custom Database
      • Data Warehouse
  4. Database Identification
    • See Figure 1 (note that we didn't specify domain name here)
    • Global Database Name: ATGOID
    • SID: ATGOID
  5. Management Options
    • Uncheck "Configure Enterprise Manager"
    • We don't need this option for our benchmark
  6. Database Credentials
    • Specify passwords for the following user accounts in the new database:
      • SYS
      • SYSTEM
  7. Network Configuration
    • Select the listeners for which you want to register this database:
      • Register this database with all the listeners
      • Register this database with selected listeners only
    • Note that we have created a new listener ATGOID using Net Configuration Assistant[3]
  8. Database File Locations
    • Specify storage type and locations for database files:
      • Storage Type: File System
      • Storage Locations:
        • Use Database File Locations from Template
        • Use Common Location for All Database Files
        • Use Oracle-Managed Files
  9. Recovery Configuration
    • Specify Fast Recovery Area (Default)
  10. Database Content
    • Default
  11. Initialization Parameters
    • Typical (Default)
  12. Database Storage
    • You can specify storage parameters for database creation.  (Default)
  13. Creation Options
    • Select the database creation options
      • Create Database
      • Save as a Database Template
      • Generate Database Creation Scripts
    • Generate Database Creation Scripts

Log File Location


Log files for the current operation are located at:
  • $ORACLE_BASE/cfgtools/dbca/ATGOID

What to Expect?


After the creation of a new database, you should find the following information:
Database Information:
Global Database Name: ATGOID
System Identifier(SID): ATGOID
Server Parameter File name:  $ORACLE_HOME/dbs/spfileATGOID.ora

If you look into initATGOID.ora, you can also find the following two entries:
*.db_domain=''
*.db_name='ATGOID'
Note that the domain name is an empty string because we didn't provide domain name at step 4.

References

  1. Installing Oracle Database 11g Release 2
  2. Oracle® Database 2 Day DBA 11g Release 1 (11.1)
  3. Using Net Configuration Assistant to Configure a Listener

Monday, January 14, 2013

Using Net Configuration Assistant to Configure a Listener

Oracle Net Configuration Assistant (i.e., netca[1] is provided to configure basic network components during database installation[2], including:
  • Listener names and protocol addresses
  • Naming methods the client will use to resolve connect identifiers to connect descriptors
  • Net service names in a tnsnames.ora file 
  • Directory server usage
netca runs automatically during database software installation, as described in your Oracle installation guide.  It can also be run after installation in standalone mode to configure the above-described components in a similar way that is provided during installation.

In this article, we will show you how to configure a new listener named ATGOID using netca.

Running Net Configuration Assistant


Configuration files are typically created in
  • UNIX
    • $ORACLE_HOME/network/admin
  • Windows
    •  %ORACLE_HOME%\network\admin
However, configuration files can be created in a variety of places.  To make sure, the correct configuration files are updated, you can set TNS_ADMIN environment variable on Unix:
  • $ export TNS_ADMIN=/export/home/oracle/atg/Oracle11gR2/product/11.2.0/dbhome_1/network/admin
You also need to specify ORACLE_HOME:
  • $export ORACLE_HOME=/export/home/oracle/atg/Oracle11gR2/product/11.2.0/dbhome_1

Log on to your computer as a member of the administrative group that is authorized to install Oracle software and create and run the database.  To start Oracle Net Configuration Assistant:
  • UNIX
    • Run netca from $ORACLE_HOME/bin
  • Windows
    • Choose Start > Programs > Oracle - HOME_NAME > Configuration and Migration Tools > Net Configuration Assistant


Configuration Steps


Here are our steps to configure a new listener named ATGOID using netca.  Note that highlighted options are our selections.
  1. Welcome
    • Choose the configuration you would like to do:
      • Listener configuration
      • Naming Methods configuration
      • Local Net Service Name configuration
      • Directory Usage Configuration
  2. Listener Configuration, Listener
    • Select what you want to do:
      • Add
      • Reconfigure
      • Delete
      • Rename
  3. Listener Configuration, Listener Name
    • Listener name: ATGOID
  4. Listener Configuration, Select Protocols
    • See Figure 1
  5. Listener Configuration, TCP/IP Protocol
    • Which TCP/IP port number should the listener use? (Default)
  6. Listener Configuration, IPC Protocol
    • To communicate with the database using the IPC protocol, an IPC key value is required.
    • IPC Key value: EXTPROC1522
  7. Listener Configuration, More Listeners
    • Would you like to configure another listener? (No)
  8. Listener Configuration, Select Listener
    • Select a listener you want to start: ATGOID
  9. Listener Configuration Done

Log location


In case there is error found during configuration, you can find more details from the log files located here:
  • $ORACLE_HOME/cfgtoollogs/netca

For example, we have found two files in there:


$ORACLE_HOME/cfgtoollogs/netca
$ ls -lrt
total 60
-rw-r--r-- 1 oracle oinstall 52023 Jan 14 16:49 trace-13011512AM2431.log
-rw-r--r-- 1 oracle oinstall   371 Jan 14 16:49 netca-13011512AM2431.log

Console Ouput


If the configuration succeeded, you should find the following messages from the console:

$ ./netca 

Oracle Net Services Configuration:
Configuring Listener:ATGOID
Listener configuration complete.
Oracle Net Listener Startup:
    Running Listener Control: 
      /export/home/oracle/atg/Oracle11gR2/product/11.2.0/dbhome_1/bin/lsnrctl start ATGOID
    Listener Control complete.
    Listener started successfully.
Oracle Net Services configuration successful. The exit code is 0

listener.ora


After the configuration, configuration file listener.ora located in $TNS_ADMIN folder will be updated.  This configuration file for the  listener may include:
  • Protocol addresses it is accepting connection requests on
  • Database and nondatabase services it is listening for
  • Control parameters used by the listener
In our case, two entries were added in listener.ora:

ATGOID =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1522))
      (ADDRESS = (PROTOCOL = TCP)(HOST = myserver.oracle.com)(PORT = 1521))
    )
  )

ADR_BASE_ATGOID = /export/home/oracle/atg/Oracle11gR2


References

Saturday, January 12, 2013

Cloning Issue—What If Host Name(s) Are Stored in the Database

This article is a follow-up to the previous article:
As pointed out in that article, there are limitations and issues with cloning—either an application or a database. This article describes one of the issues (see also [23, 4]): when host names are stored in the database tables.

Cloning


In [1], we have outlined the cloning tasks step by step.  In this article, we will discuss one of the cloning issues:
  • If host name(s) are stored in the database tables
To do cloning (see [3]), we need to duplicate a software installation from a source to a destination.  As described in [1], you can use /etc/hosts file to simplify the cloning task by redirecting host name resolution on Linux. However, if the host name is stored in the database table and not yet listed in the hosts file, your application can still fail.

In the following section, we will describe one use case that showcases this issue.

ADF Task Flow


A user would like to change the host name of a Redhat Linux which hosts SOASuite and WebCenter.  After following the approach as described in [1] using hosts file.  He found out that his workflow still wasn't working correctly.

When he opened a task, the browser tried to connect to:

and he has found that localhost.localdomain still pointed to the previous host.  Now the server doesn't know how to resolve this name and asks browsers to connect to that host.

After some diggings, he has finally identified that there is a setting in the WFTASKDISPLAY table of SOAINFRA schema, which still keeps the previous host name.  Presumably, when a task was deployed, previous host name was stored there and caused the malfunction of his ADF task flow.

To be honest, this is a case reported by another user.  So, I might have described it wrong.  However, it does point out a potential problem—if you have renamed a server and some other servers may not recognize that old name anymore because the old host name is stored in the database.  So, you need to change hosts file on those machines too to redirect the mapping of the old name to the new IP address.

References

  1. Simplify Cloning by Using Hosts File
  2. ORA-00313: open failed for members of log group 1 of thread 1
  3. Using rsync to Clone Local and Remote Systems
  4. ORA-01031: insufficient privileges
  5. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
    • This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
    • Oracle Fusion Middleware provides a series of scripts for this task.
  6. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
    • For your Oracle production systems, follow official recommendations as shown in this article.



Wednesday, January 9, 2013

ORA-01031: insufficient privileges

This article is a follow-up to the previous article:
As pointed out in that article, there are limitations and issues with cloning (either an application or a database). This article describes one of the issues (see also [2, 3]) when you clone a database.

Cloning


In [1], we have outlined the cloning tasks step by step.  In this article, we will discuss:
  • How to resolve "ORA-01031: insufficient privileges"
  • What is config.c
To do cloning (see [3]), we need to duplicate a software installation from a source to a destination by preserving its path structure and as the same UNIX user (say "oracle").  However, the same UNIX user can have different effective group ID's in the source and destination servers and this can cause issue such as ORA-01031.

ORA-01031


When we invoked "sqlplus / as sysdba" from the cloned environment, we have seen:

ERROR:
ORA-01031: insufficient privileges


When you clone a database, there could be differences in UNIX user/group setup between source server and destination server.

For example, our source server has the following user/group settings for the sqlplus executable:

-rwxr-x--x 1 oracle oracle 9221 Dec  5 01:25 /export/home/oracle/atg/Oracle11gR2/product/11.2.0/dbhome_1/bin/sqlplus

but, our destination server has different user/group settings for it:

-rwxr-x--x 1 oracle oinstall 9221 Dec  5 01:25 /export/home/oracle/atg/Oracle11gR2/product/11.2.0/dbhome_1/bin/sqlplus

To look into this issue, you can  check the UNIX group ID defined for sqldba adminstrative access in:
  • $ORACLE_HOME/rdbms/lib/config.c

config.c


When you see the following error:
ORA-01031: insufficient privileges

you want to check config.c if your password file is ok and the group is correct for the Oracle account (i.e., "oracle").  This file tells you which UNIX group ID is assigned for sqldba administrative access by this Oracle Installation.


This is the content of config.c from source server:

/*  SS_DBA_GRP defines the UNIX group ID for sqldba adminstrative access.  */
/*  Refer to the Installation and User's Guide for further information.  */

/* IMPORTANT: this file needs to be in sync with
              rdbms/src/server/osds/config.c, specifically regarding the
              number of elements in the ss_dba_grp array.
 */

#define SS_DBA_GRP "oracle"
#define SS_OPER_GRP ""
#define SS_ASM_GRP ""

char *ss_dba_grp[] = {SS_DBA_GRP, SS_OPER_GRP, SS_ASM_GRP};

However, this is the content of config.c from another working Oracle instance on destination server:

/*  SS_DBA_GRP defines the UNIX group ID for sqldba adminstrative access.  */
/*  Refer to the Installation and User's Guide for further information.  */

/* IMPORTANT: this file needs to be in sync with
              rdbms/src/server/osds/config.c, specifically regarding the
              number of elements in the ss_dba_grp array.
 */

#define SS_DBA_GRP "dba"
#define SS_OPER_GRP "oper"
#define SS_ASM_GRP ""

char *ss_dba_grp[] = {SS_DBA_GRP, SS_OPER_GRP, SS_ASM_GRP};

From the differences, we know that our cloned Oracle binary expects "oracle"  UNIX group ID for sqldba adminstrative access.   However, our "oracle" UNIX user belongs to the following UNIX groups:

  • oinstall
  • dba
  • oper
but, not "oracle".

$id
uid=507(oracle) gid=507(oinstall) groups=507(oinstall),8500(dba),8501(oper)

Solution


One way to resolve this issue is to create a new group named "oracle" on our destination server.  So, we have created a new group named "oracle" and add our "oracle" user to the "oracle" group (as a "root" user):


#groupadd -g 8502 oracle
#usermod -G oinstall,dba,oper,oracle oracle


For more information, read [4, 5].

References

  1. Simplify Cloning by Using Hosts File
  2. ORA-00313: open failed for members of log group 1 of thread 1
  3. Using rsync to Clone Local and Remote Systems
  4. Thread: ORA-01031: insufficient privileges While trying / as sysdba
  5. Creating and Maintaining a Password File
  6. ORA-27101: shared memory realm does not exist tips
  7. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
    • This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
    • Oracle Fusion Middleware provides a series of scripts for this task.
  8. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
    • For your Oracle production systems, follow official recommendations as shown in this article.

Monday, January 7, 2013

Global Database Name in Oracle Database 11g

When you install a new Oracle Database (see [1]), you are asked to provide the Global database name as shown below:


In a distributed database system, each database should have a unique global database name. Global database names uniquely identify a database in the system. For example, if global naming is enforced by the remote database, you must use the remote database global database name as the name of the database link that you are creating.

In this article, we will examine the following topics:
  • How to check if global naming is enforced?
  • How to view different names?
  • How to change the domain name in a global database name?

How to check if global naming is enforced?


To determine whether global naming on a database is enforced on a database, you can either examine the database initialization parameter file or query the V$PARAMETER view. For example, to see whether global naming is enforced on the remote database, you could start a session on the remote database and then create and execute the following globalnames.sql script (sample output included):

COL NAME FORMAT A12
COL VALUE FORMAT A6
SELECT NAME, VALUE FROM V$PARAMETER
   WHERE NAME = 'global_names'
/

SQL> @globalnames

NAME         VALUE
------------ ------
global_names FALSE


Note that global naming is not enabled by default when you install a new Oracle database.  In other words, you won't find the parameter global_names in the new initialization parameter file.

Global Database Name


In our installation, we have used "orcl.us.oracle.com" as the global  database name, which is composed of two components:
  • DB_NAME
    • "orcl"
  • DB_DOMAIN
    • "us.oracle.com"  (must follow standard Internet conventions)
You can find this piece of information in your initialization parameter file:

*.db_domain='us.oracle.com'
*.db_name='orcl'

How to query different names?


If a database instance is created, you should be able to find out all different names from the data dictionary:
  • DB_NAME
    • SQL> select name from v$database;

      NAME
      ---------
      ORCL

  • DB_DOMAIN
    • SQL> SELECT NAME, VALUE FROM V$PARAMETER WHERE NAME = 'db_domain';

      NAME      VALUE
      --------- -----------
      db_domain us.oracle.com
  • GLOBAL_NAME
    • SQL>SELECT * FROM GLOBAL_NAME;

      GLOBAL_NAME
      -------------------
      ORCL.US.ORACLE.COM

How to change domain name?


The db_domain initialization parameter is only important at database creation time when it is used, together with the db_name parameter, to form the database global name. At this point, the database global name is stored in the data dictionary. You must change the global name using an ALTER DATABASE statement, not by altering the db_domain parameter in the initialization parameter file. It is good practice, however, to change the db_domain  arameter to reflect the change in the domain name before the next database startup.

For more details on how to change the domain in a global database name, you can read [2].

References

  1. Installing Oracle Database 11g Release 2
  2. Managing Global Names in a Distributed System
  3. How to Change DB_NAME & ORACLE_SID
  4. Using Database Configuration Assistant to Create Additional Databases
  5. Using Net Configuration Assistant to Configure a Listener

Saturday, January 5, 2013

What’s the Difference between the SYS and SYSTEM Schemas?

After Oracle Database installation (see [1]), all database accounts except:
  • SYS
  • SYSTEM 
  • and a couple of others
are locked.

Curious people like me would normally ask, “What’s the difference between the SYS and SYSTEM schemas?”  In this article, answers are summarized from references [2] and [3].

SYS vs. SYSTEM


The SYS schema is the superuser of the database, owns all internal data-dictionary objects, and is used for tasks such as creating a database, starting or stopping the instance, backup and recovery, and adding or moving data files. These types of tasks typically require the SYSDBA or SYSOPER role. Security for these roles is often controlled through access to the OS account owner of the Oracle software. Additionally, security for these roles can be administered via a password file, which allows remote client/server access.

In contrast, the SYSTEM schema isn't very special. It’s just a schema that has been granted the DBA role.
Many shops lock the SYSTEM schema after database creation and never use it because it’s often the first
schema a hacker will try to access when attempting to break into a database.

Rather than risking an easily guessable entry point to the database, you can create a separate schema
(named something other than SYSTEM) that has the DBA role granted to it. This DBA schema is used for
administrative tasks such as creating users, changing passwords, granting database privileges, and so on.
Having a separate DBA schema(s) for administrators provides more options for security and auditing.




SYS
SYSTEM
When was it created?
Automatically created when Oracle database is installed Automatically created when Oracle database is installed

Granted role
Automatically granted the DBA role Automatically granted the DBA role

Default password CHANGE_ON_INSTALL (see Note 1) MANAGER (see Note 1)

What is this used for?
Owns the base tables and views for the database data dictionary used to create additional tables and views that display administrative information used to create internal tables and views used by various Oracle database options and tools

Connect as ...
The default schema when you connect as SYSDBA (select SYSDBA from the Role drop-down list in SQL Developer)

Select Default from the Role drop-down list in SQL Developer
Notes
Tables in the SYS schema are manipulated only by the database. They should never be modified by any user or database administrator, and no one should create any tables in the schema of user SYS. Database users should not connect to the Oracle database using the SYS account. Never use the SYSTEM schema to store tables of interest to non-administrative users.

Notes

  1. In Oracle Database 11g Release 2, this should be the Administrative Password you have specified during installation.[1]

References

  1. Installing Oracle Database 11g Release 2
  2. Pro Oracle Database 11g Administration
  3. What is the difference between SYS and SYSTEM database user accounts?
  4. Privileges, Roles and Profiles

Thursday, January 3, 2013

Swap Size - This Is a Prerequisite Condition Tested during Oracle Installation

During the "Prerequisite Checks" of Oracle Database installation[1], one of three issues we have seen is:
  • Swap Size

Detailed Message


The detailed description looks like this:

Swap Size - This is a prerequisite condition to test whether sufficient total swap space is available on the system.

Expected Value:11.2438GB (1.1790008E7KB)
Actual Value:7.8125GB (8191968.0KB)
Details:
PRVF-7573 : Sufficient swap size is not available on node "perf-x4600-4" [Required = 11.2438GB (1.1790008E7KB) ; Found = 7.8125GB (8191968.0KB)] - Cause: The swap size found does not meet the minimum requirement. - Action: Increase swap size to at least meet the minimum swap space requirement.

Swap Space Recommendation


For systems with a certain amount of physical RAM, the following recommendation for Oracle 11g Release 2 is typical[4]

  • 1GB <= RAM <= 2GB
    • Swap space of 1.5 times the size of RAM
  • 2GB <= RAM <= 16GB
    • Swap space to equal the size of RAM
  • RAM >= 16 GB
    • 16GB of swap space

The performance of disk-based storage is orders of magnitude slower than memory. Oracle should always run in RAM for the best performance. However, swapping is necessary for two important reasons:
  • When the system requires more memory than is physically available, the kernel swaps out less used pages and gives memory to the current application (process) that needs the memory immediately
  • A significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.

Memory and Swap Space


There are multiple ways to find the amount of physical RAM and swap space on a Liunx server:

To confirm the size of physical RAM on a Linux server:

$ grep MemTotal /proc/meminfo
MemTotal:     11790008 kB


To verify the amount of memory and swap space, run the following command:

$ free -t
             total       used       free     shared    buffers     cached
Mem:      11790008   11194300     595708          0     207588    9868888
-/+ buffers/cache:    1117824   10672184
Swap:      8191968      11824   11768092
Total:    19981976   11206124   12363800

How to Increase Swap Size?


In [2], it describes two ways of increasing swap space:
  • Use a dedicated hard drive partition to add new swap space
  • Create a swap file on an existing filesystem and use it as swap space
We used the second approach and here were our steps. Before you start, login as "root".
  1. swapon command with option -s, displays the current swap space in KB
    • #swapon -s
      Filename              Type    Size  Used Priority
      /LOCAL_SWAP/swapfile1 file 2047992 11824       -1
      /LOCAL_SWAP/swapfile2 file 2047992 0           -2
      /LOCAL_SWAP/swapfile3 file 2047992 0           -3
      /LOCAL_SWAP/swapfile4 file 2047992 0           -4

  2. dd command creates two swap files with the names “swapfile5” and “swapfile6”  under /LOCAL_SWAP directory
    • #dd if=/dev/zero of=/LOCAL_SWAP/swapfile5 bs=1000 count=2097152
      #dd if=/dev/zero of=/LOCAL_SWAP/swapfile6 bs=1024 count=1539964
  3. mkswap sets up a Linux swap area in a file
    • #mkswap /LOCAL_SWAP/swapfile5
      #mkswap /LOCAL_SWAP/swapfile6
  4. swapon enables files for paging and swapping
    • #swapon /LOCAL_SWAP/swapfile5
      #swapon /LOCAL_SWAP/swapfile6
  5. free command displays the total amount of free and used physical and swap memory in the system, as well as the buffers  used  by  the  kernel
    • #free -t
                total       used     free  shared buffers   cached
      Mem:   11790008   11193432   596576       0  214136  9871196
      -/+ buffers/cache: 1108100 10681908
      Swap:  11779916      11824 11768092
      Total: 23569924   11205256 12364668
Note that our system has 11.2438GB physical RAM.  So, the installer recommends 11.2438GB swap space. Initially, we had only 7.8125GB swap space.  After adjustment, swap space finally reaches 11.5038GB.

To make this swap space partition available even after the reboot, add the following lines to the /etc/fstab file:
/LOCAL_SWAP/swapfile5           swap    swap    defaults        0 0
/LOCAL_SWAP/swapfile6           swap    swap    defaults        0 0

As a final note, you should not place /LOCAL_SWAP in root which may cause out-of-space-in-root problem later.

References

  1. Installing Oracle Database 11g Release 2
  2. UNIX / Linux: 2 Ways to Add Swap Space Using dd, mkswap and swapon
  3. All about Linux swap space
  4. Pro Oracle Database 11g Administration
  5. How I Simplified Oracle Database Installation on Oracle Linux (Good)
  6. Removing a swap file
  7. HugePages consideration
    • On Linux, the HugePages feature allocates non-swappable memory for large page tables using memory-mapped files. If you enable HugePages, then you should deduct the memory allocated to HugePages from the available RAM before calculating swap space.
  8. Oracle Cloud - Adding swap space to your Oracle Linux compute instance

Installing Oracle Database 11g Release 2

There is a need for us to upgrade our Fusion benchmarks.  The first step required is to install Oracle Database 11g Release 2.  You can choose to either upgrade an existing Oracle or create a new one.  In our case, we have chosen to create a new one.

In this article, we have documented the details of installation process.

Steps


The following configurations are based on our own need--using Oracle in our Fusion benchmarks.  This may be different from a real production Oracle that you need.
  1. Configure Security Updates
    • You can check the option if you wish to receive security updates via My Oracle Support.
  2. Download Software Updates
    • You can choose the option to update software by providing your credentials or skip it.
  3. Installation Option
    • For my purpose, I have selected "Create and configure a database."
    • For upgrade, you can select "Upgrade an existing database" option.
  4. System Class
    • Depends on your system, you can select either 
      • Desktop Class--for a starter database, which allows minimal configuration
      • Server Class--for an advanced database, which allows more advanced configuration options.
  5. Grid Installation Options
    • You select the type of database installation you want to perform.  For instance, I've chosen "Single instance database installation" here.  Other options include:
      • Oracle Real Application Clusters database installation
      • Oracle RAC One Node database installation
  6. Install Type
    • Two options are available: 
      • Typical 
      • Advanced
  7. Typical Installation Configuration
    • See Figure 1 for details
    • Administrative password
      • This is the password for "SYS" and "SYSTEM" database accounts[4].  Note that it is case sensitive since Oracle Database 11g Release 1[10].
    • Global Database Name (GDN)
      • In a distributed database system, each database should have a unique global database name. Global database names uniquely identify a database in the system. 
      • For example, we have used orcl.us.oracle.com as our GDN.  Note that "orcl" in the GDN will become system identifier (SID).
  8. Prerequisite Checks
    • We have seen three issues:
      • User with Same UID 
        • You can ignore this although it has some complications; see [2]
      • Swap Size
        • See [3] for how to fix it
      • OS Kernel Parameter shmmax
    • You should try to fix these issues based on suggestions.
  9. Summary
  10. Install Product
  11. Finish


At step 10, a "Data Configuration Assistant" window will be shown.  On it, there is a "Password Management" button.  At this moment, all database accounts except SYS[4]SYSTEM and DBSNMP are locked.  So, from the Password Management window, you can unlock all other accounts you want to use.

Then an information window "Execute Configuration Scripts" is shown to advise you to execute a script named "root.sh" as "root" user.

Here is the result from the execution of root.sh:

# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /disk1/home/oracle/atg/OracleDB_11.2.0.3/product/11.2.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin
Creating /usr/local/bin directory...
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

At step 11, you should finally see the following message:
  • The installtion of Oracle Database was successful.

References