rsync copying tool

What is rsync?

Rsync is a fast and powerful file copying tool. Rsync can copy locally, to and from another host over any remote shell, or to and from a remote rsync daemon. Rsync has a vast set of options that allow you to control virtually every aspect of its behaviour and the way files are copied. Rsync is known for its "delta-transfer" algorithm which reduces the amount of data transferred over a network by sending only the differences between the source files and the existing files in the destination. This makes rsync extremely useful as a backup tool.

The rsync tool locates files that need to be transferred using a "quick check" algorithm that looks for any files that have changed in size or changes in the last-modified time.

Rsync is generally run from a command line or from within a script which makes it ideally suited for making backups. Many tools are also available with a GUI (Graphical User Interface) that allow you to configure backups quickly. An example of a GUI rsync tool is "Grsync". Grsync is basically a front end to the "rsync" command and is available within most repositories.

Below are examples of some basic uses of the "rsync" command for copying files:

Retrieving files from a remote host with rsync

The following rsync command retrieves files from a remote location to a specified directory. In the following example we are going to use our local machine "Mint" and our remote machine "CentOS":

"Mint" currently has no files in the directory below:


$ pwd
/home/tux/rsync_test

$ ls -l
total 0

"CentOS" has the following files in its directory:


$ pwd
/home/tux/data

$ ls -l
total 20
-rw-rw-r--. 1 tux tux 110 May 23 18:46 file1.img
-rw-rw-r--. 1 tux tux 158 May 23 18:46 file2.img
-rw-rw-r--. 1 tux tux 206 May 23 18:46 file3.img
-rw-rw-r--. 1 tux tux 254 May 23 18:46 file4.img
-rw-rw-r--. 1 tux tux 302 May 23 18:46 file5.img

On our local machine "Mint" we are going to issue the command:

rsync -v -e ssh tux@192.168.122.75:/home/tux/data/* .

This command will copy the files from the remote "CentOS" machine to our local directory. The "dot" "." after the remote directory location specifies to copy to the current directory on the local machine:


tux@mint01a:~/rsync_test$ pwd
/home/tux/rsync_test

tux@mint01a:~/rsync_test$ rsync -v -e ssh tux@192.168.122.75:/home/tux/data/* .
tux@192.168.122.75's password: 
file1.img
file2.img
file3.img
file4.img
file5.img

sent 119 bytes  received 1,357 bytes  328.00 bytes/sec
total size is 1,030  speedup is 0.70

First we will be asked to supply a password for the remote machine. Once authenticated, our files should be copied to our local machine in our current directory. If we now issue the "ls -l" command on our local machine we should see that the files have been transferred successfully.


tux@mint01a:~/rsync_test$ pwd
/home/tux/rsync_test

tux@mint01a:~/rsync_test$ ls -l
total 20
-rw-rw-r-- 1 tux tux 110 May 23 19:46 file1.img
-rw-rw-r-- 1 tux tux 158 May 23 19:46 file2.img
-rw-rw-r-- 1 tux tux 206 May 23 19:46 file3.img
-rw-rw-r-- 1 tux tux 254 May 23 19:46 file4.img
-rw-rw-r-- 1 tux tux 302 May 23 19:46 file5.img

Notice, the files that have been copied have a different time stamp to the original files on the remote "CentOS" machine. If we wanted to preserve this time stamp, then we can modify our command to include the "-t" option to preserve our time stamp information:

rsync -vt -e ssh tux@192.168.122.75:/home/tux/data/* .


tux@mint01a:~/rsync_test$ rsync -vt -e ssh tux@192.168.122.75:/home/tux/data/* .
tux@192.168.122.75's password: 
file1.img
file2.img
file3.img
file4.img
file5.img

sent 149 bytes  received 327 bytes  136.00 bytes/sec
total size is 1,030  speedup is 2.16

tux@mint01a:~/rsync_test$ ls -l
total 20
-rw-rw-r-- 1 tux tux 110 May 23 18:46 file1.img
-rw-rw-r-- 1 tux tux 158 May 23 18:46 file2.img
-rw-rw-r-- 1 tux tux 206 May 23 18:46 file3.img
-rw-rw-r-- 1 tux tux 254 May 23 18:46 file4.img
-rw-rw-r-- 1 tux tux 302 May 23 18:46 file5.img

This time the "time stamp" information has been preserved and should match that of the original files from the remote "CentOS" machine.

Copying a local file to a remote server with rsync

In the next example we are going to copy a local file to our remote "CentOS" machine.

For this example, we have added a new file called mint_file.txt into our current directory:


tux@mint01a:~/rsync_test$ ls -l
total 24
-rw-rw-r-- 1 tux tux  110 May 23 18:46 file1.img
-rw-rw-r-- 1 tux tux  158 May 23 18:46 file2.img
-rw-rw-r-- 1 tux tux  206 May 23 18:46 file3.img
-rw-rw-r-- 1 tux tux  254 May 23 18:46 file4.img
-rw-rw-r-- 1 tux tux  302 May 23 18:46 file5.img
-rw-rw-r-- 1 tux tux 2608 May 23 19:55 mint_file.txt

Now on our local machine (Mint) we issue the following command:

rsync -avz /home/tux/rsync_test/ tux@192.168.122.75:/home/tux/data/


tux@mint01a:~/rsync_test$ rsync -avz /home/tux/rsync_test/ tux@192.168.122.75:/home/tux/data/
tux@192.168.122.75's password: 
sending incremental file list
./
mint_file.txt

sent 1,526 bytes  received 38 bytes  347.56 bytes/sec
total size is 3,638  speedup is 2.33

Now if we take a look at the remote "CentOS" machine:


[tux@centos8a data]$ pwd
/home/tux/data

[tux@centos8a data]$ ls -l
total 24
-rw-rw-r--. 1 tux tux  110 May 23 18:46 file1.img
-rw-rw-r--. 1 tux tux  158 May 23 18:46 file2.img
-rw-rw-r--. 1 tux tux  206 May 23 18:46 file3.img
-rw-rw-r--. 1 tux tux  254 May 23 18:46 file4.img
-rw-rw-r--. 1 tux tux  302 May 23 18:46 file5.img
-rw-rw-r--. 1 tux tux 2608 May 23 19:55 mint_file.txt


We can see that the new file has been transferred.

rsync with ssh

In all of the previous examples so far we have had to supply our credentials (password) each time we connect to the remote "CentOS" machine. It is possible to create what is known as a public key which can then be copied to the remote server. This will then allow us to login without supplying a password. To achieve this we must carry out the following actions.

Creating a Public key

As with SSH, the "rsync" command can use "Public Key Authentication" to connect to a server without needing to enter a password. This functionality is useful if you need to transfer files from a script as you will not need to enter a password to authenticate.

Below is the process required for generating a public key

Generating a public key

Using an utility called "ssh-keygen", you can create a public key that can be distributed to servers that you require access to.

The following key has been generated on a Linux Mint system.


tux@mint01a:~$ ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/tux/.ssh/id_rsa): 
Created directory '/home/tux/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/tux/.ssh/id_rsa
Your public key has been saved in /home/tux/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:QdfQJfNmO2eixOQOrjKYKcmsVNbPdpJ+C8VGev10BhI tux@mint01a
The key's randomart image is:
+---[RSA 3072]----+
|        . oEo..  |
|       . .  ++   |
|        .. ...+  |
|    .   +..+.o.. |
|   o . .S=..+.+oo|
|  o   o =. +o.o= |
| + . + B .. o.   |
|. = + = +o       |
|.. .   +o..      |
+----[SHA256]-----+

In the above example we chose to use the "rsa" (Rivest, Shamir, Adleman) algorithm to create our digital signature. We could have chosen "dsa". Next we were prompted to supply a directory where we would like the key to be saved. In the example I pressed "Enter" to accept the default location of "/home/tux/.ssh/id_rsa". Next, We were asked to enter a passphrase. I chose not to enter a passphrase. All that is required now is to distribute our public key to our desired remote server.


tux@mint01a:~$ cd /home/tux/.ssh

tux@mint01a:~/.ssh$ ls -l
total 8
-rw------- 1 tux tux 2602 May 23 11:32 id_rsa
-rw-r--r-- 1 tux tux  565 May 23 11:32 id_rsa.pub

Copying Public keys to remote servers

In the above example of the "ssh-keygen -t rsa" command we can see that a public key was created.

My public key has been saved in /home/tux/.ssh/id_rsa.pub

To use the public key, we have to first copy the contents of the file "id_rsa.pub" to a file called "authorized_keys" on the remote server. Whenever copying a new key to a remote servers authorized_keys file, always use the append option. If you copy your files content, you could overwrite the file and destroy any other keys that may be located there. The easiest way to transfer the key to the remote server is to issue a command similar to the one below.

Note, the "authorized_keys" key file must already exist on the remote server under the account you are logging into.

In the example below, I will create the directory structure and authorized_keys file on the remote server as follows:


[tux@centos8a ~]$ pwd
/home/tux

[tux@centos8a ~]$ ls -al
total 12
drwx------. 2 tux  tux   62 May 23 11:31 .
drwxr-xr-x. 4 root root  29 May 23 11:31 ..
-rw-r--r--. 1 tux  tux   18 Jul 21  2020 .bash_logout
-rw-r--r--. 1 tux  tux  141 Jul 21  2020 .bash_profile
-rw-r--r--. 1 tux  tux  376 Jul 21  2020 .bashrc

[tux@centos8a ~]$ mkdir .ssh

[tux@centos8a ~]$ chmod 700 .ssh/

[tux@centos8a ~]$ cd .ssh
[tux@centos8a .ssh]$ touch authorized_keys

[tux@centos8a .ssh]$ chmod 600 authorized_keys 

[tux@centos8a .ssh]$ ls -l
total 0
-rw-------. 1 tux tux 0 May 23 11:46 authorized_keys

Note, I had to create the ".ssh directory first, set its permissions to "700" and then create the empty "authorized_keys" file using the "touch" command. The permissions on the "authorized_keys" file are also set to "600".

Once the above is in place, I can copy the contents of our public key to this file.


$ cat $HOME/.ssh/id_rsa.pub | ssh 192.168.122.75 'cat >> .ssh/authorized_keys && echo "Contents Copied Successfully"'
tux@192.168.122.75's password: 
Contents Copied Successfully

The above command simply appends the key on our local server to the "authorized_keys" file on the remote server. (Amend IP address to match your remote server)

Testing rsync Access with Public Key

Once the public key has successfully been copied to the correct location on the remote server, you should now be able to login to the remote server without specifying a password.

On our local machine "Mint", I have created a new file called "landoflinux.com.txt".


tux@mint01a:~/rsync_test$ pwd
/home/tux/rsync_test
tux@mint01a:~/rsync_test$ ls -rtl
total 28
-rw-rw-r-- 1 tux tux  110 May 23 18:46 file1.img
-rw-rw-r-- 1 tux tux  158 May 23 18:46 file2.img
-rw-rw-r-- 1 tux tux  254 May 23 18:46 file4.img
-rw-rw-r-- 1 tux tux  206 May 23 18:46 file3.img
-rw-rw-r-- 1 tux tux  302 May 23 18:46 file5.img
-rw-rw-r-- 1 tux tux 2608 May 23 19:55 mint_file.txt
-rw-rw-r-- 1 tux tux 2498 May 23 20:04 landoflinux.com.txt

Now lets issue the "rsync" command: rsync -avz /home/tux/rsync_test/ tux@192.168.122.75:/home/tux/data/


tux@mint01a:~/rsync_test$ rsync -avz /home/tux/rsync_test/ tux@192.168.122.75:/home/tux/data/
sending incremental file list
./
landoflinux.com.txt

sent 1,078 bytes  received 38 bytes  744.00 bytes/sec
total size is 6,136  speedup is 5.50

This time we didn't get asked for a password! The only file that was transferred was the "landoflinux.com.txt" file. We can confirm this by looking at the remote "CentOS" server:


[tux@centos8a data]$ pwd
/home/tux/data

[tux@centos8a data]$ ls -l
total 28
-rw-rw-r--. 1 tux tux  110 May 23 18:46 file1.img
-rw-rw-r--. 1 tux tux  158 May 23 18:46 file2.img
-rw-rw-r--. 1 tux tux  206 May 23 18:46 file3.img
-rw-rw-r--. 1 tux tux  254 May 23 18:46 file4.img
-rw-rw-r--. 1 tux tux  302 May 23 18:46 file5.img
-rw-rw-r--. 1 tux tux 2498 May 23 20:04 landoflinux.com.txt
-rw-rw-r--. 1 tux tux 2608 May 23 19:55 mint_file.txt

rsync options

To get a full overview of the "rsync" command, I would recommend reading the "man pages" for rsync. An overview of some of the options that can be used with the rsync command follow. To view this option you can issue the command "rsync --help" or "man rsync".


Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
 -v, --verbose               increase verbosity
     --info=FLAGS            fine-grained informational verbosity
     --debug=FLAGS           fine-grained debug verbosity
     --msgs2stderr           special output handling for debugging
 -q, --quiet                 suppress non-error messages
     --no-motd               suppress daemon-mode MOTD (see manpage caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
     --no-OPTION             turn off an implied OPTION (e.g. --no-D)
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
     --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup                make backups (see --suffix & --backup-dir)
     --backup-dir=DIR        make backups into hierarchy based in DIR
     --suffix=SUFFIX         set backup suffix (default ~ w/o --backup-dir)
 -u, --update                skip files that are newer on the receiver
     --inplace               update destination files in-place (SEE MAN PAGE)
     --append                append data onto shorter files
     --append-verify         like --append, but with old data in file checksum
 -d, --dirs                  transfer directories without recursing
 -l, --links                 copy symlinks as symlinks
 -L, --copy-links            transform symlink into referent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
     --safe-links            ignore symlinks that point outside the source tree
     --munge-links           munge symlinks to make them safer (but unusable)
 -k, --copy-dirlinks         transform symlink to a dir into referent dir
 -K, --keep-dirlinks         treat symlinked dir on receiver as dir
 -H, --hard-links            preserve hard links
 -p, --perms                 preserve permissions
 -E, --executability         preserve the file's executability
     --chmod=CHMOD           affect file and/or directory permissions
 -A, --acls                  preserve ACLs (implies --perms)
 -X, --xattrs                preserve extended attributes
 -o, --owner                 preserve owner (super-user only)
 -g, --group                 preserve group
     --devices               preserve device files (super-user only)
     --copy-devices          copy device contents as regular file
     --specials              preserve special files
 -D                          same as --devices --specials
 -t, --times                 preserve modification times
 -O, --omit-dir-times        omit directories from --times
 -J, --omit-link-times       omit symlinks from --times
     --super                 receiver attempts super-user activities
     --fake-super            store/recover privileged attrs using xattrs
 -S, --sparse                turn sequences of nulls into sparse blocks
     --preallocate           allocate dest files before writing them
 -n, --dry-run               perform a trial run with no changes made
 -W, --whole-file            copy files whole (without delta-xfer algorithm)
     --checksum-choice=STR   choose the checksum algorithms
 -x, --one-file-system       don't cross filesystem boundaries
 -B, --block-size=SIZE       force a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
     --rsync-path=PROGRAM    specify the rsync to run on the remote machine
     --existing              skip creating new files on receiver
     --ignore-existing       skip updating files that already exist on receiver
     --remove-source-files   sender removes synchronized files (non-dirs)
     --del                   an alias for --delete-during
     --delete                delete extraneous files from destination dirs
     --delete-before         receiver deletes before transfer, not during
     --delete-during         receiver deletes during the transfer
     --delete-delay          find deletions during, delete after
     --delete-after          receiver deletes after transfer, not during
     --delete-excluded       also delete excluded files from destination dirs
     --ignore-missing-args   ignore missing source args without error
     --delete-missing-args   delete missing source args from destination
     --ignore-errors         delete even if there are I/O errors
     --force                 force deletion of directories even if not empty
     --max-delete=NUM        don't delete more than NUM files
     --max-size=SIZE         don't transfer any file larger than SIZE
     --min-size=SIZE         don't transfer any file smaller than SIZE
     --partial               keep partially transferred files
     --partial-dir=DIR       put a partially transferred file into DIR
     --delay-updates         put all updated files into place at transfer's end
 -m, --prune-empty-dirs      prune empty directory chains from the file-list
     --numeric-ids           don't map uid/gid values by user/group name
     --usermap=STRING        custom username mapping
     --groupmap=STRING       custom groupname mapping
     --chown=USER:GROUP      simple username/groupname mapping
     --timeout=SECONDS       set I/O timeout in seconds
     --contimeout=SECONDS    set daemon connection timeout in seconds
 -I, --ignore-times          don't skip files that match in size and mod-time
 -M, --remote-option=OPTION  send OPTION to the remote side only
     --size-only             skip files that match in size
 -@, --modify-window=NUM     set the accuracy for mod-time comparisons
 -T, --temp-dir=DIR          create temporary files in directory DIR
 -y, --fuzzy                 find similar file for basis if no dest file
     --compare-dest=DIR      also compare destination files relative to DIR
     --copy-dest=DIR         ... and include copies of unchanged files
     --link-dest=DIR         hardlink to files in DIR when unchanged
 -z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with a suffix in LIST
 -C, --cvs-exclude           auto-ignore files the same way CVS does
 -f, --filter=RULE           add a file-filtering RULE
 -F                          same as --filter='dir-merge /.rsync-filter'
                             repeated: --filter='- .rsync-filter'
     --exclude=PATTERN       exclude files matching PATTERN
     --exclude-from=FILE     read exclude patterns from FILE
     --include=PATTERN       don't exclude files matching PATTERN
     --include-from=FILE     read include patterns from FILE
     --files-from=FILE       read list of source-file names from FILE
 -0, --from0                 all *-from/filter files are delimited by 0s
 -s, --protect-args          no space-splitting; only wildcard special-chars
     --address=ADDRESS       bind address for outgoing socket to daemon
     --port=PORT             specify double-colon alternate port number
     --sockopts=OPTIONS      specify custom TCP options
     --blocking-io           use blocking I/O for the remote shell
     --stats                 give some file-transfer stats
 -8, --8-bit-output          leave high-bit chars unescaped in output
 -h, --human-readable        output numbers in a human-readable format
     --progress              show progress during transfer
 -P                          same as --partial --progress
 -i, --itemize-changes       output a change-summary for all updates
     --out-format=FORMAT     output updates using the specified FORMAT
     --log-file=FILE         log what we're doing to the specified FILE
     --log-file-format=FMT   log updates using the specified FMT
     --password-file=FILE    read daemon-access password from FILE
     --list-only             list the files instead of copying them
     --bwlimit=RATE          limit socket I/O bandwidth
     --stop-at=y-m-dTh:m     Stop rsync at year-month-dayThour:minute
     --time-limit=MINS       Stop rsync after MINS minutes have elapsed
     --outbuf=N|L|B          set output buffering to None, Line, or Block
     --write-batch=FILE      write a batched update to FILE
     --only-write-batch=FILE like --write-batch but w/o updating destination
     --read-batch=FILE       read a batched update from FILE
     --protocol=NUM          force an older protocol version to be used
     --iconv=CONVERT_SPEC    request charset conversion of filenames
     --checksum-seed=NUM     set block/file checksum seed (advanced)
     --noatime               do not alter atime when opening source files
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (-h is --help only if used alone)