Using rsync to Clone Local and Remote Systems

This article is a follow-up from the previous article:

• Simplify Cloning by Using Hosts File^[1]

As pointed out in that article, there are limitations and issues with cloning (either an application or a database). This article describes one of the issues (see also [4]).

Cloning

In [1], we have outlined the cloning tasks step by step.  In this article, we will discuss:

• How to use rsync utility to transfer and synchronize local and remote systems
• How to deal with symbolic links

To do cloning, we need to duplicate a software installation from a source to a destination by preserving its path structure.

Symbolic Links

One of the challenges in cloning is that not everything is self-contained in a source tree.  Very often, symbolic links are also involved.  There are two types of symbolic links:

1. Symbolic links point outwards from the source tree
2. External symbolic links point towards the source tree

For the cloning, we use rsync utility to do the job. Here are the options that we have used:

• rsync -az

This command can copy the first type of links appropriately while it cannot handle the second type of links.  That means you need to create extra symbolic links in the destination after the cloning.  So, one of the pre-cloning tasks is to list all symbolic links and their locations in the source.^[2]

Why It Happened?

Why the second type of symbolic links exist in the first place?  It depends on each application and the way cloning was done previously.  In our benchmark cloning, we usually clone one environment to multiple destinations in a chain.  For example, someone has set up a good benchmark on machine A.  Then we clone that to machine B followed by cloning it from machine B to C, etc.

On machine B, we often find there is a need to distribute resources on different file systems for load balancing. Because of that, new symbolic links were introduced. Then, when we clone the installation from machine B to C, we will find both types of symbolic links existing in the source.

Rsync Command^[3]

One way of copying a directory is using rsync.  The rsync utility has an archive switch -a that allows it to perform a copy of a directory that includes dot files while maintaining all permissions, ownership, and modification times. However, the destination soft links have the modification time of when the copy was performed, but that shouldn't matter much.

When using the following commands, there is a very subtle syntax difference between the two (i.e., the trailing slash), which ends up with quite different results:

• rsync -az /src/dir/ /dest/dir
• The contents of /src/dir will be copied to /dest/dir
• rsync -az /src/dir /dest/dir
• The directory itself will be copied into /dest/dir. In other words, you’ll end up with /dest/dir/dir

In the command, we have also include a compression switch -z which can be used in the remote transfer to reduce network traffic.

To enable remote transfer, you append "<userLogin>@<serverName>:" to either src or dest path. For example,

• rsync -az oracle@otherserver:/data/home/oracle/atg/OracleDB_11.2.0.2 /data/home/oracle/atg

will copy the directory named OracleDB_11.2.0.2 from a remote server into /data/home/oracle/atg.

References

1. Simplify Cloning by Using Hosts File
2. List symbolic links and location, pointing to a particular directory
3. Expert Shell Scripting
4. ORA-00313: open failed for members of log group 1 of thread 1
5. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
• This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
• Oracle Fusion Middleware provides a series of scripts for this task.
6. To check if the symbolic links are broken in the target system, do:

• find . -type l -! -exec test -e {} \; -print

7. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
• For your Oracle production systems, follow official recommendations as shown in this article.

Simplify Cloning by Using Hosts File

Oftentimes, you will find there is a need to install a same application on different systems.  In our case, we have a multi-tier setup for benchmark:

• Oracle Application Testing Suite (OATS)
• Application Server
• Database Server

This means that individual server need to communicate with other servers using their domain names.

In this article, we will discuss the simplest way of cloning an application from one environment to another.

Hosts File

The hosts file allows you to define which domain names (websites) are linked to which IP addresses. On some platforms, it takes precedence over your DNS servers.  However, the hosts file is under the direct control of the local computer's administrator unlike the DNS. So your DNS servers may say oracle.com is linked to a specific IP address, but you can have oracle.com go anywhere you want by using hosts file.

In Microsoft Windows, hosts file is located at locations depending on your OS.  For example, for NT, 2000, XP (x86 & x64), 2003, Vista, 7and 8, it is located at:

• %SystemRoot%\system32\drivers\etc\hosts
• Need to have write permission on this file for the editing user

In Linux, hosts file is located at:

• /etc/hosts

Hosts file is a plain text file and you can use any text editor to modify it as long as you are given the permission. After modification, it will take effect immediately without rebooting. So, you can restart your application to see the new changes right away.

Redirection

In its function of resolving host names, the hosts file may be used to define any host name or domain name for use in the local system. This may be used either beneficially or maliciously for various effects.  In this article, we will discuss using the mapping to redirect a website (i.e., source of cloning) to another website (i.e., destination of cloning) during the cloning of a multi-tier environment.  Because our multi-tier environment exists in a private network, there is no security concern for us.  However, it is possible for you to face serious security attacks if your hosts file is compromised^[3].

As we all know, to deploy and configure any web application is a non trivial task.  As performance engineers, we often need to create similar systems on different set of servers.  Instead of  deploying and configuring web applications from scratch, it will be easier to just do the cloning.

After cloning the application from one system to another, you then need to fix platform-specific part in the cloned image.  For example, you need to change the domain names referenced in the URLs from old server's to new server's.  Domain names can also be embedded in configuration files, scripts, etc.  Either you can do a global search and replace them or you can modify hosts file to map same host names to the new IP addresses.  The latter is easier.

Cloning^[4]

Cloning can be done in three stages and the most important stage is the preparation or pre-cloning.  In the following, we cover the tasks involved in these stages for Linux platforms.  For Windows, the steps are similar.

• Pre-cloning Stage
• nohup
• Cloning can take hours to finish.  
• If you use "putty" to access Linux box, remember to use "nohup" command
• nohup is used to run a command that is immune to hangups
• For example, you can prefix your cloning command with nohup and redirect the stderr and stdout to cloning.out file:
• nohup {cloning command} &> cloning.out &
• Be the right user that has the privilege to do the cloning.  
• Sometimes, you may need to be the "root" user to do the cloning.  After the cloning, you can then reduce the accessibility to the correct level.
• Create the same path structure on the destination as source's
• Create symbolic links if needed
• You may need to be root user to create the path.  But, reduce the accessibility to the correct level later.
• Find the file system (or disk) that is big enough to hold the cloned image
• Free space left should allow application data to grow after it starts running
• Hosts file
• Save the original hosts file
• Shutdown server instances before cloning
• Cloning Stage
  • Copy everthing needed from source machine to destination machine.  This can include
    • Server installation
    • Scripts
    • hosts file
    • Copy the new hosts file from source to destination machine and make appropriate changes
    • Validate the changes.  For example, you can use ping command to test 
  • Use rsync command to clone
  • Syntax:
  • rsync -az aroot@sourceServer:/export/home/bench/ATG/RUP3 /export/home/bench/ATG/
  • Don't forget to use nohup for the rsync
  • Try the command out with a small copy first
  • Be patient—the cloning could take hours
• Post-cloning Stage
• Verify that your cloned environment work as expected
• You can test this by stages.  For example, you can
1. Run your front end (or OATS) against original Application Server and Database Server first.  After verifying that your front end system is working correctly, move to next.
2. Run your application server against original Databasse Server. After verifying that your middle tier is working correctly, move to next.
3. Verifying your database server is working correctly.
4. Run your application server against your new database server.
5. And so on.
• Document what you have done

Warning

Cloning applications seems to be straightforward.  But, there are limitations and caveats. You can read [5,6,9,10] for such details.  If you are cloning Oracle Fusion Middleware, read [7].  If you are moving from a test to a production environment, read [8].  Finally, you must pay attention to the license-violations-and-compliance issue when you plan a cloning.

References

1. Oracle Application Testing Suite
2. 6 Surprising Uses For The Windows Hosts File
3. Hosts (Wikipedia)
4. Cloning Application Server Middle-Tier Instances
5. General Considerations and Limitations for Cloning
6. ORA-00313: open failed for members of log group 1 of thread 1
7. Cloning Oracle Fusion Middleware (Chapter 20)
8. Moving from a Test to a Production Environment (Chapter 21)
9. Cloning Issue—What If Host Name(s) Are Stored in the Database
10. ORA-01031: insufficient privileges
11. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
• This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
• Oracle Fusion Middleware provides a series of scripts for this task.
12. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
• For your Oracle production systems, follow official recommendations as shown in this article.