Tuesday, November 27, 2012

Using rsync to Clone Local and Remote Systems

This article is a follow-up from the previous article:
As pointed out in that article, there are limitations and issues with cloning (either an application or a database). This article describes one of the issues (see also [4]).

Cloning


In [1], we have outlined the cloning tasks step by step.  In this article, we will discuss:
  • How to use rsync utility to transfer and synchronize local and remote systems
  • How to deal with symbolic links
To do cloning, we need to duplicate a software installation from a source to a destination by preserving its path structure.

Symbolic Links


One of the challenges in cloning is that not everything is self-contained in a source tree.  Very often, symbolic links are also involved.  There are two types of symbolic links:
  1. Symbolic links point outwards from the source tree
  2. External symbolic links point towards the source tree
For the cloning, we use rsync utility to do the job. Here are the options that we have used:
  • rsync -az
This command can copy the first type of links appropriately while it cannot handle the second type of links.  That means you need to create extra symbolic links in the destination after the cloning.  So, one of the pre-cloning tasks is to list all symbolic links and their locations in the source.[2]

Why It Happened?


Why the second type of symbolic links exist in the first place?  It depends on each application and the way cloning was done previously.  In our benchmark cloning, we usually clone one environment to multiple destinations in a chain.  For example, someone has set up a good benchmark on machine A.  Then we clone that to machine B followed by cloning it from machine B to C, etc.

On machine B, we often find there is a need to distribute resources on different file systems for load balancing. Because of that, new symbolic links were introduced. Then, when we clone the installation from machine B to C, we will find both types of symbolic links existing in the source.

Rsync Command[3]


One way of copying a directory is using rsync.  The rsync utility has an archive switch -a that allows it to perform a copy of a directory that includes dot files while maintaining all permissions, ownership, and modification times. However, the destination soft links have the modification time of when the copy was performed, but that shouldn't matter much.

When using the following commands, there is a very subtle syntax difference between the two (i.e., the trailing slash), which ends up with quite different results:
  • rsync -az /src/dir/ /dest/dir
    • The contents of /src/dir will be copied to /dest/dir
  • rsync -az /src/dir /dest/dir
    • The directory itself will be copied into /dest/dir. In other words, you’ll end up with /dest/dir/dir

In the command, we have also include a compression switch -z which can be used in the remote transfer to reduce network traffic.

To enable remote transfer, you append "<userLogin>@<serverName>:" to either src or dest path. For example,

  • rsync -az oracle@otherserver:/data/home/oracle/atg/OracleDB_11.2.0.2 /data/home/oracle/atg

will copy the directory named OracleDB_11.2.0.2 from a remote server into /data/home/oracle/atg.

References

  1. Simplify Cloning by Using Hosts File
  2. List symbolic links and location, pointing to a particular directory
  3. Expert Shell Scripting
  4. ORA-00313: open failed for members of log group 1 of thread 1
  5. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
    • This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
    • Oracle Fusion Middleware provides a series of scripts for this task.
  6. To check if the symbolic links are broken in the target system, do:
    • find . -type l -! -exec test -e {} \; -print
  7. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
    • For your Oracle production systems, follow official recommendations as shown in this article.

Saturday, November 24, 2012

Book Review: Oracle 11g Anti-hacker's Cookbook‏


The number of security threats related to operating systems and databases are increasing every day, and this trend is expected to continue. Therefore, effective countermeasures to reduce or eliminate these threats must be found and applied.

"Oracle 11g Anti-hacker's Cookbook" covers all the important security measures that can be deployed to protect hackers from attacking your Oracle database.  It provides many useful tips and tricks.  As such, you should add this book to your arsenals of Oracle security.

Connecting to the Database


There are different ways of connecting to an Oracle database (i.e., creating an Oracle session):
  • Programmers 
    • Use ODBC, JDBC and OCI 
  • Database Administrators
    • Use SQL*Plus and Oracle Enterprise Manager (OEM)
Although connection concepts apply to all utilities, we use SQL Command Line (SQL*Plus) for illustration, which is the principal DBA interface into Oracle.


The Weakest Link


As shown above, you can see there are multiple systems involved in the database connection.  Any system involved can have one or more vulnerabilities that can be exploited by hackers in a threat action.

Security practitioners[2] often point out that security is a chain; and just as a chain is only as strong as the weakest link, a database security system is only as secure as its weakest component.

Therefore, there are no short-cuts for Oracle protection.  In this book, it describes lots of tips and tricks which can be deployed to fortify components along this connection chain.

Types of attacks


There can exist many types of attacks on an Oracle session.  Here are some of them as covered in this book:
  • Man-in-the-middle-type (MITM) attack
    • Attack in which an interposed attacker hijacks a client connection
  • TCP and UDP protocol-level attack
    • Targeted towards the network traffic and the data in flight
  • TNS poison attack[3]
    • TNS poison attack is classified as a man-in-the-middle-type attack
  • Replay attack
    • An attack in which a valid data transmission is maliciously or fraudulently repeated or delayed
  • DoS attack
    • To fill up the the file systems on the disk with useless log messages
    • To send a succession of SYN requests
    • To send large numbers of IP packets with the source address faked to appear to be the address of the victim
  • IP Spoofing
    • To create Internet Protocol (IP) packets with a forged source IP address, with the purpose of concealing the identity of the sender or impersonating another computing system
  • Dictionary and pattern matching type attack
  • Password cracking
  • Other attacks 
    • Target the database, listener, and configuration files
And lastly and the most importantly, the weakest part of your Oracle system will be administrators, users or tech support people who fall prey to social engineering.

General vs. Oracle Specific Recipes


In this book, many recipes are provided to show how those security risks could be mitigated or reduced.  To sum up, recipes can be classified into general or Oracle specific security measures.  For example, to confront different interception-type attacks, you can use either Oracle Advanced Security encryption and integrity, or alternatives such as IPSEC, stunnel, and SSH tunneling.

For general measures, topics such as OS security and Securing the network and data in transit are covered in Chapter 1 and 2.  Starting from Chapter 3, security measures using Oracle products start emerging, which includes the following:
  • Oracle RMAN
  • Oracle Enterprise Manager
  • Oracle Virtual Private Database
  • Oracle Label Security
  • Oracle Database Vault
  • Oracle Audit
  • Oracle Cryptographic API
  • Oracle Wallets

Other Recommendations


In the book, it also make suggestions such as:
  • You should implement data audits to detect the origin of the attack or the source of the inappropriate data access or modification
  • You  should develop and implement appropriate alerting systems to proactively detect and prevent attacks on systems and data
  • You should test these security measures first before their final deployment
  • You should perform security assessments regularly on your system

Picture Credit


  • Figure 3-2 Remote Connection in Oracle® Database Express Edition 2 Day DBA 10g Release 2 (10.2)

References

    1. Oracle 11g Anti-hacker's Cookbook
    2. Viega, John & McGraw, Gary.  Building Secure Software: How to Avoid Security Problems the Right Way. Boston, MA: Addison-Wesley, 2002.
    3. Oracle Database TNS Listener Poison Attack
    4. Replay Attack (wikipedia)
    5. Oracle® Database Installation Guide 11g Release 2 (11.2) for Linux
    6. Securing the Weakest Link
    7. The Onion Model

    Wednesday, November 14, 2012

    ORA-00313: open failed for members of log group 1 of thread 1

    This article is a follow-up to the previous article:
    As pointed out in that article, there are limitations and issues with cloning (either an application or a database). This article describes one of the issues.

    ORA-00313


    After following the cloning steps as described in [1], we have run into this Oracle database exception when trying to bring up our cloned Oracle.  Obviously, this is our fault because we have not done a thorough planning before the cloning.

    What this exception tells us is:
    • The online log cannot be opened.

    What Is the Redo Log?


    The most crucial structure for Oracle recovery operations is the redo log, which consists of two or more preallocated files that store all changes made to the database as they occur. Every instance of an Oracle Database has an associated redo log to protect the database in case of an instance failure.

    Where Did We Find This ORA-00313?


    From the initialization parameter file (i.e., dbs/init<sid>.ora ) , we have traced down the location of diagnostic destination[3]:
    • diagnostic_dest=/slot/fiz7865/log
    From there, we went down to a folder named:
    • <diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
    In there, there is a file named:
    • alert_fiz7865.log
    From that file, we have found the following entries:
      Lost write protection disabled
      Completed: ALTER DATABASE   MOUNT
      Wed Nov 14 09:42:55 2012
      ALTER DATABASE OPEN
      Errors in file <diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace/fiz7865_lgwr_25410.trc:
      ORA-00313: open failed for members of log group 1 of thread 1
      ORA-00312: online log 1 thread 1: '/data1/rup3.redolog/log3.dbf'
    

    Note that Oracle will write the alert_<instname>.log file to the directory as specified by the BACKGROUND_DUMP_DEST parameter[4]. So, you can also find out its location by:
    SQL> show parameter BACKGROUND_DUMP_DEST
    
    NAME                   TYPE        VALUE
    ---------------------- ----------- ------------------------------
    background_dump_dest   string     /slot/fiz7865/log/diag/rdbms/fiz7865/fiz7865/trace
    

    What Happened?


    When we do the cloning, not everything is contained in a single source directory.  For example, redo log files have been reallocated to another file system (i.e., /data1) which is outside the source directory.  For the Oracle to be fully functional, original redo logs need to be reopened.  If they are not found, an ORA-00313 will be thrown.

    How to Find the Redo Log Location


    Before you do the cloning, keep the source database up and running.  Then query the logfile location by:
    • select * from V$LOGFILE;

    References

    1. Simplify Cloning by Using Hosts File
    2. Managing the Redo Log
    3. DIAGNOSTIC_DEST
    4. Alert Log
    5. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
      • This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
      • Oracle Fusion Middleware provides a series of scripts for this task.
    6. Oracle Products: What Patching, Migration, and Upgrade Mean?

    Wednesday, November 7, 2012

    Simplify Cloning by Using Hosts File

    Oftentimes, you will find there is a need to install a same application on different systems.  In our case, we have a multi-tier setup for benchmark:
    • Oracle Application Testing Suite (OATS)
    • Application Server
    • Database Server
    This means that individual server need to communicate with other servers using their domain names.

    In this article, we will discuss the simplest way of cloning an application from one environment to another.

    Hosts File


    The hosts file allows you to define which domain names (websites) are linked to which IP addresses. On some platforms, it takes precedence over your DNS servers.  However, the hosts file is under the direct control of the local computer's administrator unlike the DNS. So your DNS servers may say oracle.com is linked to a specific IP address, but you can have oracle.com go anywhere you want by using hosts file.

    In Microsoft Windows, hosts file is located at locations depending on your OS.  For example, for NT, 2000, XP (x86 & x64), 2003, Vista, 7and 8, it is located at:
    • %SystemRoot%\system32\drivers\etc\hosts
      • Need to have write permission on this file for the editing user
    In Linux, hosts file is located at:
    • /etc/hosts
    Hosts file is a plain text file and you can use any text editor to modify it as long as you are given the permission. After modification, it will take effect immediately without rebooting. So, you can restart your application to see the new changes right away.

    Redirection


    In its function of resolving host names, the hosts file may be used to define any host name or domain name for use in the local system. This may be used either beneficially or maliciously for various effects.  In this article, we will discuss using the mapping to redirect a website (i.e., source of cloning) to another website (i.e., destination of cloning) during the cloning of a multi-tier environment.  Because our multi-tier environment exists in a private network, there is no security concern for us.  However, it is possible for you to face serious security attacks if your hosts file is compromised[3].

    As we all know, to deploy and configure any web application is a non trivial task.  As performance engineers, we often need to create similar systems on different set of servers.  Instead of  deploying and configuring web applications from scratch, it will be easier to just do the cloning.

    After cloning the application from one system to another, you then need to fix platform-specific part in the cloned image.  For example, you need to change the domain names referenced in the URLs from old server's to new server's.  Domain names can also be embedded in configuration files, scripts, etc.  Either you can do a global search and replace them or you can modify hosts file to map same host names to the new IP addresses.  The latter is easier.

    Cloning[4]


    Cloning can be done in three stages and the most important stage is the preparation or pre-cloning.  In the following, we cover the tasks involved in these stages for Linux platforms.  For Windows, the steps are similar.
    • Pre-cloning Stage
      • nohup
        • Cloning can take hours to finish.  
          • If you use "putty" to access Linux box, remember to use "nohup" command
            • nohup is used to run a command that is immune to hangups
            • For example, you can prefix your cloning command with nohup and redirect the stderr and stdout to cloning.out file:
              • nohup {cloning command} &> cloning.out &
      • Be the right user that has the privilege to do the cloning.  
        • Sometimes, you may need to be the "root" user to do the cloning.  After the cloning, you can then reduce the accessibility to the correct level.
      • Create the same path structure on the destination as source's
        • Create symbolic links if needed
          • You may need to be root user to create the path.  But, reduce the accessibility to the correct level later.
      • Find the file system (or disk) that is big enough to hold the cloned image
        • Free space left should allow application data to grow after it starts running
      • Hosts file
        • Save the original hosts file
      • Shutdown server instances before cloning
    • Cloning Stage
      • Copy everthing needed from source machine to destination machine.  This can include
        • Server installation
        • Scripts
        • hosts file
          • Copy the new hosts file from source to destination machine and make appropriate changes
          • Validate the changes.  For example, you can use ping command to test 
      • Use rsync command to clone
        • Syntax:
          • rsync -az aroot@sourceServer:/export/home/bench/ATG/RUP3 /export/home/bench/ATG/
        • Don't forget to use nohup for the rsync
        • Try the command out with a small copy first
        • Be patient—the cloning could take hours
    • Post-cloning Stage
      • Verify that your cloned environment work as expected
        • You can test this by stages.  For example, you can
          1. Run your front end (or OATS) against original Application Server and Database Server first.  After verifying that your front end system is working correctly, move to next.
          2. Run your application server against original Databasse Server. After verifying that your middle tier is working correctly, move to next.
          3. Verifying your database server is working correctly.
          4. Run your application server against your new database server.
          5. And so on.
      • Document what you have done

    Warning


    Cloning applications seems to be straightforward.  But, there are limitations and caveats. You can read [5,6,9,10] for such details.  If you are cloning Oracle Fusion Middleware, read [7].  If you are moving from a test to a production environment, read [8].  Finally, you must pay attention to the license-violations-and-compliance issue when you plan a cloning.

    References

    1. Oracle Application Testing Suite
    2. 6 Surprising Uses For The Windows Hosts File
    3. Hosts (Wikipedia)
    4. Cloning Application Server Middle-Tier Instances
    5. General Considerations and Limitations for Cloning
    6. ORA-00313: open failed for members of log group 1 of thread 1
    7. Cloning Oracle Fusion Middleware (Chapter 20)
    8. Moving from a Test to a Production Environment (Chapter 21)
    9. Cloning Issue—What If Host Name(s) Are Stored in the Database
    10. ORA-01031: insufficient privileges
    11. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
      • This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
      • Oracle Fusion Middleware provides a series of scripts for this task.
    12. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
      • For your Oracle production systems, follow official recommendations as shown in this article.