Tuesday, November 27, 2012

Using rsync to Clone Local and Remote Systems

This article is a follow-up from the previous article:
As pointed out in that article, there are limitations and issues with cloning (either an application or a database). This article describes one of the issues (see also [4]).


In [1], we have outlined the cloning tasks step by step.  In this article, we will discuss:
  • How to use rsync utility to transfer and synchronize local and remote systems
  • How to deal with symbolic links
To do cloning, we need to duplicate a software installation from a source to a destination by preserving its path structure.

Symbolic Links

One of the challenges in cloning is that not everything is self-contained in a source tree.  Very often, symbolic links are also involved.  There are two types of symbolic links:
  1. Symbolic links point outwards from the source tree
  2. External symbolic links point towards the source tree
For the cloning, we use rsync utility to do the job. Here are the options that we have used:
  • rsync -az
This command can copy the first type of links appropriately while it cannot handle the second type of links.  That means you need to create extra symbolic links in the destination after the cloning.  So, one of the pre-cloning tasks is to list all symbolic links and their locations in the source.[2]

Why It Happened?

Why the second type of symbolic links exist in the first place?  It depends on each application and the way cloning was done previously.  In our benchmark cloning, we usually clone one environment to multiple destinations in a chain.  For example, someone has set up a good benchmark on machine A.  Then we clone that to machine B followed by cloning it from machine B to C, etc.

On machine B, we often find there is a need to distribute resources on different file systems for load balancing. Because of that, new symbolic links were introduced. Then, when we clone the installation from machine B to C, we will find both types of symbolic links existing in the source.

Rsync Command[3]

One way of copying a directory is using rsync.  The rsync utility has an archive switch -a that allows it to perform a copy of a directory that includes dot files while maintaining all permissions, ownership, and modification times. However, the destination soft links have the modification time of when the copy was performed, but that shouldn't matter much.

When using the following commands, there is a very subtle syntax difference between the two (i.e., the trailing slash), which ends up with quite different results:
  • rsync -az /src/dir/ /dest/dir
    • The contents of /src/dir will be copied to /dest/dir
  • rsync -az /src/dir /dest/dir
    • The directory itself will be copied into /dest/dir. In other words, you’ll end up with /dest/dir/dir

In the command, we have also include a compression switch -z which can be used in the remote transfer to reduce network traffic.

To enable remote transfer, you append "<userLogin>@<serverName>:" to either src or dest path. For example,

  • rsync -az oracle@otherserver:/data/home/oracle/atg/OracleDB_11.2.0.2 /data/home/oracle/atg

will copy the directory named OracleDB_11.2.0.2 from a remote server into /data/home/oracle/atg.


  1. Simplify Cloning by Using Hosts File
  2. List symbolic links and location, pointing to a particular directory
  3. Expert Shell Scripting
  4. ORA-00313: open failed for members of log group 1 of thread 1
  5. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
    • This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
    • Oracle Fusion Middleware provides a series of scripts for this task.
  6. To check if the symbolic links are broken in the target system, do:
    • find . -type l -! -exec test -e {} \; -print
  7. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
    • For your Oracle production systems, follow official recommendations as shown in this article.

1 comment:

Blogger said...

BlueHost is definitely one of the best hosting provider for any hosting services you might need.