Linux rsync Command

Creating backups with rsync

rsync copying tool


rsync is a fast and powerful file copying tool. Rsync can copy locally, to and from another host over any remote shell, or to and from a remote rsync daemon. Rsync has a vast set of options that allow you to control virtually every aspect of its behaviour and the way files are copied. Rsync is famous for its "delta-transfer" algorithm which reduces the amount of data transferred over a network by sending only the differences between the source files and the existing files in the destination. This makes rsync extremely useful as a backup tool.

The rsync tool locates files that need to be transferred using a "quick check" algorithm that looks for any files that have changed in size or changes in the last-modified time.



Rsync is generally run from a command line or script which makes it ideally suited for making backups. Many tools are also available with a gui (Graphical User Interface) that allow you to configure backups quickly. An example of a GUI rsync tool is "Grsync". Grsync is basically a front end to the "rsync" command.

Below are examples of some basic uses of the "rsync" command for copying files:


Retrieving files from a remote host with rsync


The following rsync command retrieves files from a remote location to a specified directory. In the following example we are going to use our local machine "Ubuntu" and our remote machine "suse":

"Ubuntu" currently has no files in the directory below:



john@john-desktop:~/ubuntu$ ls -l
total 0

"suse" has the following files in its directory:



john@linux-pd5y:~/suse> pwd
/home/john/suse
john@linux-pd5y:~/suse> ls -l
total 40
-rw-r--r-- 1 john users  2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john users  5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john users  8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john users 14330 Apr 25 09:00 file4.txt

On our local machine "Ubuntu" we are going to issue the command:

rsync -v -e ssh root@192.168.0.11:/home/john/suse/* .

This command will copy the files from the remote "suse" machine to our local directory:



john@john-desktop:~/ubuntu$ rsync -v -e ssh root@192.168.0.11:/home/john/suse/* .
Password: 
file1.txt
file2.txt
file3.txt
file4.txt

sent 87 bytes  received 31778 bytes  4902.31 bytes/sec
total size is 31526  speedup is 0.99

First we will be asked to supply a password for the remote machine. Once authenticated, our files should be copied to our local machine in our current directory. If we now issue the "ls -l" command on our local machine we should see that the files have been transferred successfully.



john@john-desktop:~/ubuntu$ ls -l
total 40
-rw-r--r-- 1 john john  2866 Apr 25 09:09 file1.txt
-rw-r--r-- 1 john john  5732 Apr 25 09:09 file2.txt
-rw-r--r-- 1 john john  8598 Apr 25 09:09 file3.txt
-rw-r--r-- 1 john john 14330 Apr 25 09:09 file4.txt

Notice, the files that have been copied have a different time stamp to the original files on the remote "suse" machine.

If we wanted to preserve this file stamp, then we can modify our command to include the "-t" option to preserve our time stamp information:

rsync -vt -e ssh root@192.168.0.11:/home/john/suse/* .



john@john-desktop:~/ubuntu$ rsync -vt -e ssh root@192.168.0.11:/home/john/suse/* .
Password: 
file1.txt
file2.txt
file3.txt
file4.txt

sent 375 bytes  received 428 bytes  146.00 bytes/sec
total size is 31526  speedup is 39.26
john@john-desktop:~/ubuntu$ ls -l
total 40
-rw-r--r-- 1 john john  2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john john  5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john john  8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john john 14330 Apr 25 09:00 file4.txt

This time the "time stamp" information has been preserved and should match that of the original file from the remote "suse" machine.


Copying a local file to a remote server with rsync


In the next example we are going to copy a local file to our remote "suse" machine.

For this example, we have added a new file called ubuntu_file.txt into our current directory:



john@john-desktop:~/ubuntu$ ls -l
total 72
-rw-r--r-- 1 john john  2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john john  5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john john  8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john john 14330 Apr 25 09:00 file4.txt
-rw-rw-r-- 1 john john 31526 Apr 25 09:18 ubuntu_file.txt

Now on our local machine (Ubuntu) we issue the following command:

rsync -avz /home/john/ubuntu/ john@192.168.0.11:/home/john/suse/



john@john-desktop:~/ubuntu$ rsync -avz /home/john/ubuntu/ john@192.168.0.11:/home/john/suse/
Password: 
sending incremental file list
./
ubuntu_file.txt

sent 1066 bytes  received 46 bytes  202.18 bytes/sec
total size is 63052  speedup is 56.70

Now if we take a look at the remote "suse" machine:



john@linux-pd5y:~/suse> ls -l
total 72
-rw-r--r-- 1 john 1000  2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john 1000  5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john 1000  8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john 1000 14330 Apr 25 09:00 file4.txt
-rw-rw-r-- 1 john 1000 31526 Apr 25 09:18 ubuntu_file.txt

We can see that only the new file has been transferred.


rsync with ssh


In all of the previous examples so far we have had to supply our credentials (password) each time we connect to the remote "suse" machine. It is possible to create what is known as a public key which can then be copied to the remote server. This will then allow us to login without supplying a password. To achieve this we must carry out the following actions:

Define a ssh key on our local machine (Ubuntu):

This is achieved by issuing the "ssh-keygen" command. For more details on this visit our ssh page.

In the example below we did not provide any "passphrase", we simply pressed "enter".



john@john-desktop:~/.ssh$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/john/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/john/.ssh/id_rsa.
Your public key has been saved in /home/john/.ssh/id_rsa.pub.
The key fingerprint is:
d7:f5:6e:20:9b:fe:29:51:f3:d4:2f:bb:6c:c5:9a:31 john@john-desktop
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|              . .|
|           . .o.o|
|        S . o..=o|
|         .  .+Eo=|
|            o. Oo|
|           ...=o |
|            .+=. |
+-----------------+

From the above, we can see that we have generated what is known as a "public key" called "id_rsa.pub" in the following location: /home/john/.ssh

Next we need to copy the contents of our key "id_rsa.pub" to the remote machine.


Copying the Public Key to a remote location


We then copy the contents of our public key from our local server "/home/john/.ssh/id_rsa.pub" to the "authorized_keys" file on the "remote "suse" machine. To achieve this we can simply issue the following command. You will be prompted for your password to authorise this activity on the remote server.

cat $HOME/.ssh/id_rsa.pub | ssh 192.168.0.11 'cat >> .ssh/authorized_keys && echo "Contents Copied Successfully"'



john@john-desktop:~/ubuntu$ cat $HOME/.ssh/id_rsa.pub | ssh 192.168.0.11 'cat >> .ssh/authorized_keys && echo "Contents Copied Successfully"'
Password: 
Contents Copied Successfully

Now lets confirm that the file has been copied to the remote "suse" machine:



john@linux-pd5y:~/.ssh> ls -l
total 4
-rw-r--r-- 1 john users 399 Apr 25 09:34 authorized_keys
john@linux-pd5y:~/.ssh> cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpqMCtzPvRZD+pfnO8gHoJrtXF+JjOji/GXTK5FeioFHyZMzvT4CTXqAQIFRIkq1fIoZkMz8LYmVqBPP/vfS7SbTL81dR3Mpri6YzngjGka1ejH6+34PVgZT47eBi65kZddGWOpfpBYj/1x+7aNwvZTRwx6TLZcq0NQ5Ot8ZTJ/yTXJfRAaYKlLRN4cu6JZ4HYqVC0kBoVSGzeBdCmA1Yoa3ecWqGuAKXskb8sseRH2es6/7G+C3Wa205TUYf42DoUyD6X3BXhs0WvvsqXKrKOx+ZvUBgq0EsH58lL4tKu2aaKamg5mR8B6WUJZzlAkksaBukAW9sJqL6RxG4zJpNx john@john-desktop

Now on the remote machine we need to modify the permissions of the newly created "authorized_keys" file with the "chmod" command so that only the owner of the file can view its content:



john@linux-pd5y:~/.ssh> chmod 600 authorized_keys
john@linux-pd5y:~/.ssh> ls -l authorized_keys 
-rw------- 1 john users 399 Apr 25 09:34 authorized_keys

Test ssh key


To test that our key is working correctly, we can issue the command: ssh 192.168.0.11 from the local "ubuntu" machine.



john@john-desktop:~/ubuntu$ ssh 192.168.0.11
Last login: Thu Apr 25 09:39:34 2013 from 192.168.0.14
Have a lot of fun...

Success we now do not have to supply a password each time we use the "rsync" command on our remote "suse" machine.


rsync with ssh example


For our new test I have created an additional file called "landoflinux.com":



john@john-desktop:~/ubuntu$ ls -rtl
total 76
-rw-r--r-- 1 john john  2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john john  5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john john  8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john john 14330 Apr 25 09:00 file4.txt
-rw-rw-r-- 1 john john 31526 Apr 25 09:18 ubuntu_file.txt
-rw-rw-r-- 1 john john   333 Apr 25 09:42 landoflinux.com

Now lets issue our "rsync" command: rsync -avz /home/john/ubuntu/ john@192.168.0.11:/home/john/suse/



john@john-desktop:~/ubuntu$ rsync -avz /home/john/ubuntu/ john@192.168.0.11:/home/john/suse/
sending incremental file list
./
landoflinux.com

sent 333 bytes  received 34 bytes  244.67 bytes/sec
total size is 63385  speedup is 172.71

This time we didn't get asked for a password! The only file that was transferred was the "landoflinux.com" file. We can confirm this by looking at the remote "suse" server:



john@john-desktop:~/ubuntu$ ssh 192.168.0.11 ls -l /home/john/suse
total 76
-rw-r--r-- 1 john  1000  2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john  1000  5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john  1000  8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john  1000 14330 Apr 25 09:00 file4.txt
-rw-rw-r-- 1 john users   333 Apr 25 09:42 landoflinux.com
-rw-rw-r-- 1 john  1000 31526 Apr 25 09:18 ubuntu_file.txt

Notice now, we can remotely execute commands on the remote server directly from our local machine! In the above examples, we have seen only a very small part of what "rsync" is capable of. To get a full overview of the "rsync" command, I would recommend reading the "man pages" for rsync. An over view of some of the options that can be used with the rsync command follow. To view this option you can issue the command "rsync --help" or "man rsync"


rsync options



Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
 -v, --verbose               increase verbosity
 -q, --quiet                 suppress non-error messages
     --no-motd               suppress daemon-mode MOTD (see manpage caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
     --no-OPTION             turn off an implied OPTION (e.g. --no-D)
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
     --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup                make backups (see --suffix & --backup-dir)
     --backup-dir=DIR        make backups into hierarchy based in DIR
     --suffix=SUFFIX         set backup suffix (default ~ w/o --backup-dir)
 -u, --update                skip files that are newer on the receiver
     --inplace               update destination files in-place (SEE MAN PAGE)
     --append                append data onto shorter files
     --append-verify         like --append, but with old data in file checksum
 -d, --dirs                  transfer directories without recursing
 -l, --links                 copy symlinks as symlinks
 -L, --copy-links            transform symlink into referent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
     --safe-links            ignore symlinks that point outside the source tree
 -k, --copy-dirlinks         transform symlink to a dir into referent dir
 -K, --keep-dirlinks         treat symlinked dir on receiver as dir
 -H, --hard-links            preserve hard links
 -p, --perms                 preserve permissions
 -E, --executability         preserve the file's executability
     --chmod=CHMOD           affect file and/or directory permissions
 -A, --acls                  preserve ACLs (implies --perms)
 -X, --xattrs                preserve extended attributes
 -o, --owner                 preserve owner (super-user only)
 -g, --group                 preserve group
     --devices               preserve device files (super-user only)
     --specials              preserve special files
 -D                          same as --devices --specials
 -t, --times                 preserve modification times
 -O, --omit-dir-times        omit directories from --times
     --super                 receiver attempts super-user activities
     --fake-super            store/recover privileged attrs using xattrs
 -S, --sparse                handle sparse files efficiently
 -n, --dry-run               perform a trial run with no changes made
 -W, --whole-file            copy files whole (without delta-xfer algorithm)
 -x, --one-file-system       don't cross filesystem boundaries
 -B, --block-size=SIZE       force a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
     --rsync-path=PROGRAM    specify the rsync to run on the remote machine
     --existing              skip creating new files on receiver
     --ignore-existing       skip updating files that already exist on receiver
     --remove-source-files   sender removes synchronized files (non-dirs)
     --del                   an alias for --delete-during
     --delete                delete extraneous files from destination dirs
     --delete-before         receiver deletes before transfer, not during
     --delete-during         receiver deletes during the transfer
     --delete-delay          find deletions during, delete after
     --delete-after          receiver deletes after transfer, not during
     --delete-excluded       also delete excluded files from destination dirs
     --ignore-errors         delete even if there are I/O errors
     --force                 force deletion of directories even if not empty
     --max-delete=NUM        don't delete more than NUM files
     --max-size=SIZE         don't transfer any file larger than SIZE
     --min-size=SIZE         don't transfer any file smaller than SIZE
     --partial               keep partially transferred files
     --partial-dir=DIR       put a partially transferred file into DIR
     --delay-updates         put all updated files into place at transfer's end
 -m, --prune-empty-dirs      prune empty directory chains from the file-list
     --numeric-ids           don't map uid/gid values by user/group name
     --timeout=SECONDS       set I/O timeout in seconds
     --contimeout=SECONDS    set daemon connection timeout in seconds
 -I, --ignore-times          don't skip files that match in size and mod-time
     --size-only             skip files that match in size
     --modify-window=NUM     compare mod-times with reduced accuracy
 -T, --temp-dir=DIR          create temporary files in directory DIR
 -y, --fuzzy                 find similar file for basis if no dest file
     --compare-dest=DIR      also compare destination files relative to DIR
     --copy-dest=DIR         ... and include copies of unchanged files
     --link-dest=DIR         hardlink to files in DIR when unchanged
 -z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with a suffix in LIST
 -C, --cvs-exclude           auto-ignore files the same way CVS does
 -f, --filter=RULE           add a file-filtering RULE
 -F                          same as --filter='dir-merge /.rsync-filter'
                             repeated: --filter='- .rsync-filter'
     --exclude=PATTERN       exclude files matching PATTERN
     --exclude-from=FILE     read exclude patterns from FILE
     --include=PATTERN       don't exclude files matching PATTERN
     --include-from=FILE     read include patterns from FILE
     --files-from=FILE       read list of source-file names from FILE
 -0, --from0                 all *-from/filter files are delimited by 0s
 -s, --protect-args          no space-splitting; only wildcard special-chars
     --address=ADDRESS       bind address for outgoing socket to daemon
     --port=PORT             specify double-colon alternate port number
     --sockopts=OPTIONS      specify custom TCP options
     --blocking-io           use blocking I/O for the remote shell
     --stats                 give some file-transfer stats
 -8, --8-bit-output          leave high-bit chars unescaped in output
 -h, --human-readable        output numbers in a human-readable format
     --progress              show progress during transfer
 -P                          same as --partial --progress
 -i, --itemize-changes       output a change-summary for all updates
     --out-format=FORMAT     output updates using the specified FORMAT
     --log-file=FILE         log what we're doing to the specified FILE
     --log-file-format=FMT   log updates using the specified FMT
     --password-file=FILE    read daemon-access password from FILE
     --list-only             list the files instead of copying them
     --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
     --write-batch=FILE      write a batched update to FILE
     --only-write-batch=FILE like --write-batch but w/o updating destination
     --read-batch=FILE       read a batched update from FILE
     --protocol=NUM          force an older protocol version to be used
     --iconv=CONVERT_SPEC    request charset conversion of filenames
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (-h is --help only if used alone)