Tuesday, December 25, 2012

How to Download WebLogic Upgrade Installer?

In our old Oracle Fusion Middleware Installation, we have a 10.3.5.0 WebLogic Server. See [1] for how to find the version of WebLogic Server.  Because we are testing a new feature in HotSpot VM which requires 10.3.6.0 WebLogic Server to work with, we need to upgrade WebLogic Server from 10.3.5.0 to 10.3.6.0.

After some research, we have found [2] which provides the upgrade instructions and decided to give it a try.  In this article, we report the steps that we have used to download the WebLogic Upgrade Installer.

Types of Installers


In total, there are four types of WebLogic Server installers available:
  • OS-specific Package installer
  • Generic Package installer
  • Upgrade installer
  • Development-only and supplemental installers
You can read [4] for the detailed descriptions of these various types of installers.  For our purpose, what we need is an Upgrade installer.

Upgrade installers, as their names suggest, allow you to upgrade an existing WebLogic Server installation to a later patch release.  Note that if you have an existing WebLogic Server 10.3.0, 10.3.1, 10.3.2, or 10.3.3 installation that includes Workshop for WebLogic, and you want to use an Upgrade installer to upgrade that installation to WebLogic Server 10.3.6, you must uninstall Workshop for WebLogic before running the Upgrade installer.  See [5] for more information.

How to Download?







Our steps have followed the instructions in [3], however, with minor modifications.  To download an Upgrade installer:
  1. Enter the My Oracle Support URL (https://support.oracle.com/) in a Web browser.
  2. Click Sign In and enter your My Oracle Support username and password.
  3. Select the Patches and Updates tab.
  4. In the Patch Search pane, click Product or Family (Advanced Search).
  5. From the Product Is: drop-down list, select Oracle WebLogic Server.
  6. From the Release Is: drop-down list, click the arrow next to the Oracle WebLogic Server folder, select the release you want to download, and click Close. (To upgrade to WebLogic Server 10.3.6, select WLS 10.3.6.0 from the list.)
  7. From the Platform drop-down list, select your platform and click Close. You can select multiple platforms. Selected platforms are indicated by a check mark.
  8. Click + next to Platform drop-down list. A new filter row appears and + changes to -.
  9. From the left operand drop-down list, select Description.  From the operator drop-down list, select contains.  In the right operand field, specify INSTALLER.
  10. Click Search. The search results list displays all available Upgrade installers for the selected release for all of the selected platforms.
  11. Click the Download button on the right to begin the download.
  12. Click Save.
  13. Browse to the directory where you want to save the installer, and click Save again to start the file download. A compressed file downloads for the selected platform.
  14. After the download completes, extract the compressed file, which contains only the appropriate installer executable for the selected platform.
Note that the typical description for an upgrade installer patch includes the following text:
  • WEBLOGIC SERVER 11GR1(10.3.4) UPGRADE INSTALLER
For example, here is the description of the patch that we have found:
  • PLACEHOLDER BUG FOR WEBLOGIC SERVER 11GR1 (10.3.6) UPGRADE INSTALLER (Patch)




Finally, if you run into download issue (for example, the download sits at 99% and never completes), try using different browser (for example, Chrome instead of IE).

References

  1. How to find Oracle WebLogic Server Version?
  2. Oracle® Fusion Middleware Installation Guide for Oracle WebLogic Server 11g Release 1 (10.3.6)
  3. Downloading an Upgrade Installer From My Oracle Support
  4. Types of Installers
  5. Uninstalling the Software
  6. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston
  7. Oracle® Fusion Middleware Patching Guide 11g Release 1 (11.1.1.7.0)
    • Patching involves copying a small collection of files over an existing installation.
    • Upgrade involves moving from a previous major version to a new major version. For example, an upgrade would be required to move from Oracle Application Server 10g to Oracle Fusion Middleware 11g.
  8. Oracle Fusion Middleware 12c —Install, Patch, and Upgrade
  9. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More


Thursday, December 20, 2012

umount: /dev/sdd: device is busy

This article is a follow-up on a previous article [1].  When we tried to unmount one of the file system, it failed with the following messages:

# umount /dev/sdd
umount: /dev/sdd: device is busy
umount: /dev/sdd: device is busy

In the following sections, we will summarize what we have found.

What Was Keeping the Device Busy?


On Linux platforms, you can use fuser command to find what processes have been keeping your device busy. The fuser command lists the process numbers of local processes that use the local or remote files specified by the File parameter. For block special devices, the command lists the processes that use any file on that device.  For example, here is what we have found with fuser command:
# fuser -m /dev/sdd
/dev/sdd:            24381c 24393ce 24449ce

In the listing, each process number is followed by a letter indicating how the process uses the file[2]:
ItemDescription
cUses the file as the current directory.
eUses the file as a program's executable object.
rUses the file as the root directory.
sUses the file as a shared library (or other loadable object).

What Application It Is Given a Process Number?


To find which application it is given its process number, you can use:

#  ps auxw|grep  24381
oracle   24381  0.0  0.0  63868  1160 ?        S    Dec19   0:00 /bin/sh -f /home/oracle/atg/IDM_11gR1.PS4.RC4/Oracle_IDM1/bin/emctl asstart agent

# ps auxw|grep  24393
oracle   24393  0.0  0.0  88552 10904 ?        S    Dec19   0:01 /home/oracle/atg/IDM_11gR1.PS4.RC4/Oracle_IDM1/perl/bin/perl /home/oracle/atg/IDM_11gR1.PS4.RC4/Oracle_IDM1/bin/emwd.pl agent /home/oracle/atg/IDM_11gR1.PS4.RC4/Instances/asinst_1/EMAGENT/EMAGENT/sysman/log/emagent.nohup

# ps auxw|grep  24449
oracle   24449  0.0  0.3 318624 38424 ?        Sl   Dec19   0:09 /home/oracle/atg/IDM_11gR1.PS4.RC4/Oracle_IDM1/bin/emagent

In our case, the above orphaned processes still hang around after the following command:

  • $MW_HOME/asinst_1/bin/opmnctl  stopall

Normally, "opmnctl stopall"[5] should be able to stop opmn and all managed processes.  However, there are some mis-configured host names in our Management Agent property file:
  • $MW_HOME/asinst_1/EMAGENT/EMAGENT/sysman/config/emd.properties
which have caused orphaned processes unresponsive.

After stopping these orphaned processes by force, we are able to unmount /dev/sdd file system and have it checked by e2fsck command.

Tuesday, December 18, 2012

init: Id "co" respawning too fast: disabled for 5 minutes

From Linux syslog (i.e., /var/log/messages), we have found the following repeated messages:
  • Dec 17 04:21:41 myserver init: Id "co" respawning too fast: disabled for 5 minutes
As usual, we have taken action to investigate it.  Here is the report on what we have found.

The Culprit


The serial ports[2] in Linux are named ttyS0, ttyS1, etc. The /dev directory has a special file for each port. Type "ls /dev/ttyS*" to see them. Just because there may be (for example) a ttyS1 file, doesn't necessarily mean that there exists a physical serial port there.

For our issue, it turns out to be:
  • ttyS1 physical serial port is not present on our host myserer
To find out if a physical serial port is present or not, you can test it in two ways:
  1. Using setserial command
  2. Using dmesg command

Output from setserial Command


As shown below, we can tell that ttyS0 is present, but not ttyS1:
[aroot@myserver oracle]# setserial /dev/ttyS0
/dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
[aroot@myserver oracle]# setserial /dev/ttyS1
/dev/ttyS1, UART: unknown, Port: 0x02f8, IRQ: 3

Output from dmesg Command


As shown below, only ttyS0 is printed by dmesg:
[root@myserver ~]# dmesg | grep tty
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

The Solution


Because ttyS1 is absent, that is why the following line in /etc/inittab is failing and causes the re-spawning:
  • co:2345:respawn:/sbin/agetty 9600 ttyS1
When a process re-spawns too many times, it will be disabled for sometime by the kernel to keep the host stable. Following message is the result of that:
  • init: Id "co" respawning too fast: disabled for 5 minutes
As there is no ttyS1, the fix for our issue is:
  1. $su aroot
  2. #vi /etc/inittab
    • Commenting or removing the line for co
  3. #init q
You can verify that inittab is re-read by Linux kernel from the following line:
  • Dec 17 23:49:40 myserver init: Re-reading inittab
in Linux syslog.

Monday, December 17, 2012

EXT3-fs warning: checktime reached, running e2fsck is recommended

From Linux syslog[1] (i.e., /var/log/messages), we have found the following message:
  • EXT3-fs warning: checktime reached, running e2fsck is recommended 

And, from the file system disk space report (i.e., "df -k"), we find:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             61189192  31629588  26401228  55% /
/dev/sda1               101086     21650     74217  23% /boot
tmpfs                 16472072         0  16472072   0% /dev/shm
/dev/sdb              70440240  35203348  31658524  53% /data

So, our Linux platform has three different file systems:
  • /dev/sda1
  • /dev/sda3
  • /dev/sdb

What is /dev/sda1[2,5]?


For Linix device naming,

  • hd
    • It designates IDE devices
  • sd
    • It designates SCSI devices including kernel-level emulation of SCSI devices, like USB devices or, in some cases, CD-RW drives.

The a and the b's, etc .. (i.e., sda, sdb)are the equivalent to the concept of hda, and hdb, etc.  When different devices are plugged in, they are mapped to sda, sdb, etc. For detailed information about these plug-in devices, we can find them in the boot message, which can be generated by:
  • $dmesg > boot.messages
From the boot message, we have found the following entries:


scsi0 : aacraid
  Vendor: Sun       Model: ssssss-xdddd-dd   Rev: V1.0
  Type:   Direct-Access                      ANSI SCSI revision: 02
SCSI device sda: 143134720 512-byte hdwr sectors (73285 MB)
sda: Write Protect is off
sda: Mode Sense: 06 00 10 00
SCSI device sda: drive cache: write through w/ FUA
SCSI device sda: 143134720 512-byte hdwr sectors (73285 MB)
sda: Write Protect is off
sda: Mode Sense: 06 00 10 00
SCSI device sda: drive cache: write through w/ FUA
 sda: sda1 sda2 sda3
sd 0:0:0:0: Attached scsi removable disk sda
  Vendor: Sun       Model: solaris root      Rev: V1.0
  Type:   Direct-Access                      ANSI SCSI revision: 02
 .
 .
 .

EXT3 FS on sda3, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3-fs warning: checktime reached, running e2fsck is recommended
EXT3 FS on sdb, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 8289532k swap on /dev/sda2.  Priority:-1 extents:1 across:8289532k

So, we know our devices use EXT3 FS.

What Is EXT3 FS[3]?


EXT3 FS, or third extended filesystem, is a journaled file system that is commonly used by the Linux kernel.  It is the default file system for many popular Linux distributions.

EXT3 adds the following features to EXT2:
  • A journal
  • Online file system growth
  • Htree indexing for larger directories
Without these features, any EXT3 file system is also a valid EXT2 file system. This situation has allowed well-tested and mature file system maintenance utilities for maintaining and repairing EXT2 file systems to also be used with EXT3 without major changes. The EXT2 and EXT3 file systems share the same standard set of utilities, e2fsprogs, which includes an fsck tool. The close relationship also makes conversion between the two file systems (both forward to EXT3 and backward to EXT2) straightforward.

With all these background information, now we are ready to run e2fsck as recommended.

Running e2fsck


In general it is not safe to run e2fsck on mounted filesystems. So, before we run e2fsck command, we have unmounted our file system.


[root@myserver bench]# umount -v /dev/sdb
/dev/sdb umounted
[root@myserver bench]# /sbin/e2fsck -v  /dev/sdb
e2fsck 1.39 (29-May-2006)
data has gone 652 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

  349958 inodes used (3.90%)
    2278 non-contiguous inodes (0.7%)
         # of inodes with ind/dind/tind blocks: 16016/1145/0
 9082105 blocks used (50.76%)
       0 bad blocks
       2 large files

  311040 regular files
   38016 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
     893 symbolic links (893 fast symbolic links)
       0 sockets
--------
  349949 files


/etc/fstab[4]


The 6th column of fstab is a fsck option. fsck looks at the number in the 6th column to determine in which order the filesystems should be checked. If it's zero, fsck won't check the filesystem.

In our system, we find that both "/" and "/boot" filesystems are automatically checked whenever the system is rebooted:
LABEL=/                 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
Therefore, we don't need to fsck either /dev/sda3 or /dev/sda1.

References

  1. Troubleshooting Linux with syslog
  2. What is /dev/sda1?
  3. ext3 (Wikipedia)
  4. How to edit and understand /etc/fstab - 1.1
  5. linux sda vs hda ?

Friday, December 14, 2012

How to Configure Logging Using Weblogic Scripting Tool

Weblogic Scripting Tool (WLST) is a command-line tool that runs on the same machine as the Weblogic server and allows the user to browse the configuration and state of the server through a tree of mbeans (managememnt beans).  It is based on the Java scripting interpreter, Jython.

Using WLST, you can configure a server instance’s logging and message output.  To determine which log attributes can be configured, see LogMBean and LogFileMBean in the WebLogic Server MBean Reference[1].

In this article, we first show you how to configure log attributes from WebLogic Server Administration Console and then show you how to set attributes of LogMBean using WLST.

Modifying Attribute from WLS Console


To bring WLS Administration Console up, you type the following address into your browser's address field:
  • http://<myserver>:7001/console
and log in with your credentials (say, "weblogic/weblogic1").  To modify "Rotation file size" of the log attribute, you do:
  • Click "Lock & Edit"
  • Select and Click:
    • Environment > Servers > CRMDemo_server1 > Logging
You modify "Rotation file size" to be a different value and then activate your change.  



Next to the "Rotation file size" field, you can click on "More Info..." to see its detailed description.

As you can see, "Rotation file size" field is linked to the following MBean Attibute:
  • LogMBean.FileMinSize

Modifying Attribute from WLST


Instead of modifying log attributes from the console, you can also achieve it by using WLST.

$cd $MW_HOME/wlserver_10.3/common/bin
$./wlst.sh
wls:/offline> connect("weblogic","weblogic1", "t3://localhost:7001")  
wls:/atgdomain/serverConfig> cd("Servers/CRMDemo_server1/Log/CRMDemo_server1")
wls:/atgdomain/serverConfig/Servers/CRMDemo_server1/Log/CRMDemo_server1> ls()
dr--   DomainLogBroadcastFilter
dr--   LogFileFilter
-r--   FileMinSize                                  5000
-r--   FileName                                     logs/CRMDemo_server1.log
 .
 .
 .
-r-x   unSet                                        Void : String(propertyName)

As shown above, this is what happened:
  1. We connected to the server
  2. We set Current Management Object (CMO) to the server log config
  3. We listed all LogMBean attributes
  4. Our log attribute FileMinSize was shown on the list
To modify "FileMinSize" log attribute, you can create a script (i.e., setFileMinSize.py) such as:

# Connect to the server
connect("weblogic","weblogic1","t3://localhost:7001")
edit()
startEdit()
# set CMO to the server log config
cd("Servers/CRMDemo_server1/Log/CRMDemo_server1")
ls()
# change LogMBean attributes
set("FileMinSize", 400)
# list the current directory to confirm the new attribute values
ls()
# save and activate the changes
save()
activate(block="true")
# all done...
exit()

Thursday, December 13, 2012

How to find Oracle WebLogic Server Version?

In [1], it describes 3 ways of finding Oracle WebLogic Server Version.  In this article, we will focus on using registry.xml[2] which contains record of all Oracle products (including weblogic) installed .

MiddleWare Home and Oracle Home


Middleware home is a top-level directory created during the Weblogic Installation and contains all the Oracle Homes (or optional Oracle product homes), Weblogic server , Coherence Server and optionally Weblogic domain (under user_projects which will be created when you create the first domain).  A middleware home can reside on a local file system or on a remote shared disk that is accessible through NFS. The Oracle Fusion Middleware home is represented in path names as MW_HOME.  An Oracle home contains installed files necessary to host a specific product.  The Oracle home is represented in path names as ORACLE_HOME.



Before you install any Fusion Web Application, you need to install an Application Server such as WebLogic Server and other applications that this application depends on.  At "Installation Location" step, you will be asked for:
  • Oracle Middleware Home
  • Oracle Home Directory
In our case, we have specified:
  • Oracle Middleware Home
    • /export/home/bench/ATG/PS6ST7
  • Oracle Home Directory
    • Oracle_WC1
When the installation completes, it displays where Middleware Home Location and Oracle Home Location are (see the diagram).  


registry.xml


This registry file contains record of all weblogic products installed along with product-related information such as version number, patch set level, patch level and location of the production installation directories.

To find Oracle WebLogic Server Version, do:
  1. Go to Middleware Home under which WebLogic is installed 
  2. Look for file registry.xml
    • For example, it's under:
      • $MW_HOME/registry.xml
  3. Open registry.xml and search for “component name=”WebLogic Server”” 
    • Variable "version" next to "component name" will tell you WebLogic version. For example, this is what I've found from my FA installation:
              <component name="WebLogic Server" version="10.3.6.0" ...>
                <component name="Core Application Server"/>
                <component name="Administration Console"/>
                ...
                <component name="Evaluation Database"/>
                <component name="Workshop Code Completion Support"/>
              </component>
To learn more on the following topics

  • WebLogic Server Version Numbers
  • WebLogic Version Compatibility
  • etc.
read [5].


Photo Credit


References

  1. Using the BEA Registry API
  2. Oracle® Fusion Middleware Fusion Developer's Guide for Oracle Application Development Framework 11g Release 1 (11.1.1)
  3. How to find Oracle WebLogic Server Version ?
  4. Using Oracle Fusion Middleware
  5. WebLogic Server Compatibility (WLS 12c)
    • Within a WebLogic domain, the Administration Server, Managed Server instances, and the domain itself each have a WebLogic Server version number. The version number contains five decimal places, for example WebLogic Server 12.1.2.0.0.

Tuesday, December 11, 2012

Understanding WebLogic Incident and the Diagnostic Framework behind It

From the WebLogic Server log, I have found the following incident:

[INCIDENT_ERROR] [_createIncident] An incident was created. The details are: 
Incident details are: IncidentID=359
ProductID=fusionapps
InstanceID=ApplicationCoreCRMDemoUI
ProductType=ofm
DomainName=atgdomain
ServerName=CRMDemo_server1
. (FND-10102). (FND-10000)
<Dec 8, 2012 9:22:57 PM PST> <Alert> <Diagnostics> 
<BEA-320016> <Creating diagnostic image in /data/ATG/MLRRC4/user_projects/domains/...
   /incident/incdir_359 with a lockout minute period of 1.>
[INCIDENT_ERROR] [JUFormBinding] java.lang.Exception: An application error occurred.  
See the incident log for more information.
  at oracle.apps.fnd.applcore.messages.ExceptionHandlerUtil...

In this article, we will examine what WebLogic Incident and Diagnostic Framework are.

Diagnostic Framework[1]


Oracle Fusion Middleware includes a Diagnostic Framework (DFW).  DFW is available with all FMW 11g installations that run on WebLogic Server.  It aids in detecting, diagnosing, and resolving problems. The problems that are targeted in particular are critical errors such as those caused by:
  • Code bugs
  • Metadata corruption
  • Customer data corruption
  • Deadlocked threads
  • Inconsistent state
When a critical error occurs, it is assigned an incident number, and diagnostic data for the error (such as log files) are immediately captured and tagged with this number. The data is then stored in the Automatic Diagnostic Repository (ADR).

The Automatic Diagnostic Repository (ADR) is a file-based repository for storing diagnostics data associated with incidents. It consists of metadata that describes each Problem and Incident, along with the set of diagnostic dump output generated for each incident.

ADR Directory Structure



The ADR root directory is known as ADR base. By default, the ADR base is located in the following directory:
  • DOMAIN_HOME/servers/server_name/adr
Within ADR base, there can be multiple ADR homes, where each ADR home is the root directory for all incident data for a particular instance of Oracle WebLogic Server or a Java application.

For example, the following path shows the locations of ADR home for
  • Oracle WebLogic Server instance
    • ADR_BASE/diag/ofm/domain_name/server_name
  • Fusion Application
    • ADR_BASE/diag/ofm/fusionapps/app_name

ADRCI


The ADR Command Interpreter (ADRCI) is a utility that enables you to investigate problems, and package and upload first-failure diagnostic data to Oracle Support, all within a command-line environment. ADRCI also enables you to view the names of the dump files in the ADR, and to view the alert log with XML tags stripped, with and without content filtering.

ADRCI is installed in the following directory:

(UNIX) MW_HOME/wlserver_10.3/server/adr
(Windows) MW_HOME\wlserver_10.3\server\adr

$ find -name wlserver_10.3
$ cd ./MLRRC4/wlserver_10.3
$ cd server/adr
$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/export/home/bench/ATG/MLRRC4/wlserver_10.3/server/adr
$ export PATH=${PATH}:/export/home/bench/ATG/MLRRC4/wlserver_10.3/server/adr
$ adrci
ADRCI: Release 11.2.0.1.0 - Production on Tue Dec 11 19:47:41 2012

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

No ADR base is set
adrci> set base MW_HOME/user_projects/domains/atgdomain/servers/CRMDemo_server1/adr
adrci> show homes
ADR Homes:
diag/ofm/fusionapps/ApplicationCoreCRMDemoUI
diag/ofm/atgdomain/CRMDemo_server1

adrci> set homepath diag/ofm/fusionapps/ApplicationCoreCRMDemoUI
adrci> show incident
adrci> show incident -mode DETAIL -p "incident_id=359"

After setting LD_LIBRARY_PATH and PATH environment variables, we invoked adrci utility directly from the command line.  The first thing we did after entering ADRCI is setting ADR base and ADR home.  Within our specified ADR home, we listed all the incidents under it.  Then we listed a specific incident (i.e., 359) that we are interested in.  For further information, read [2,6].

Diagram Credit


  • Figure 13-1 ADR Directory Structure for Oracle Fusion Middleware

References

  1. Diagnosing Problems
    • Describes how to use the Oracle Fusion Middleware Diagnostic Framework to collect and manage information about a problem so that you can resolve it or send it to Oracle Support for resolution.
  2. ADRCI: ADR Command Interpreter
    • ADRCI is a command line utility that originated as a Oracle database software utility. Recent versions of Oracle Weblogic also include this utility.
  3. Introduction to FMW Diagnostic Framework
  4. How to View Java and PL/SQL Incidents
  5. Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests
  6. Using the ADRCI utility with Oracle Weblogic
  7. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston
  8. ADRCI-WebLogic
  9. Using the ADRCI utility with Oracle Weblogic
  10. Oracle Fusion Middleware Diagnostic Framework―How to Diagnosing Problems (XML and More)