AICollection Help

rsync

rsync is a fast and versatile file copying tool commonly used for synchronizing files and directories either locally or remotely. It is renowned for its efficiency, as it transfers only the differences between the source and the destination, which minimizes data transfer and speeds up backups and mirroring. This comprehensive guide covers everything you need to know about rsync—from its basic syntax and options to advanced usage scenarios—with plenty of examples to help you integrate it into your workflow.

Table of Contents

  1. Introduction

  2. Basic Syntax and How rsync Works

  3. Common Command-Line Options and Parameters

  4. Practical Examples

  5. Advanced Usage and Tips

  6. Conclusion and Further Reading

Introduction

rsync is widely used for tasks such as:

  • Backing up data from one location to another.

  • Mirroring entire directories across servers.

  • Incremental file transfers where only changed portions are transmitted.

  • Synchronizing files over networks with minimal bandwidth usage.

The tool's power comes from its ability to compare the source and destination at the file or even block level, transferring only what’s needed. This not only saves time but also reduces network load, making rsync a favorite among system administrators and developers.

Basic Syntax and How rsync Works

The general syntax of rsync is as follows:

rsync [OPTIONS] SOURCE DESTINATION

  • SOURCE: The file or directory you wish to copy or synchronize.

  • DESTINATION: The target location, which can be local or remote. Remote locations are specified in the format user@host:/path.

Example: Basic Local Copy

rsync -av /path/to/source/ /path/to/destination/

This command synchronizes the contents of /path/to/source/ with /path/to/destination/ using archive mode and verbose output.

Common Command-Line Options and Parameters

rsync offers a plethora of options that allow you to fine-tune its behavior. Below are some of the most commonly used options:

Archive Mode (-a)

  • -a or --archive:
    Enables archive mode, which is a shorthand for a combination of options that preserve symbolic links, permissions, timestamps, and more. It also enables recursive copying.

    rsync -a source/ destination/

Verbose and Progress Options (-v, --progress, etc.)

  • -v or --verbose:
    Increases the verbosity of the output, giving you more insight into the transfer process.

    rsync -av source/ destination/

  • --progress:
    Displays progress information during the transfer, including percentage completed and transfer speed.

    rsync -av --progress source/ destination/

Compression (-z)

  • -z or --compress:
    Compresses file data during the transfer, which is especially useful for remote transfers over slower networks.

    rsync -avz source/ destination/

Deletion (--delete)

  • --delete:
    Deletes files in the destination that no longer exist in the source, making the destination an exact mirror of the source.

    rsync -av --delete source/ destination/

Partial Transfers and Resuming (--partial, --append)

  • --partial:
    Keeps partially transferred files so that if the transfer is interrupted, it can be resumed without starting over.

  • --append:
    Appends data to files that are already partially transferred.

    rsync -av --partial source/ destination/

Excluding Files and Directories (--exclude, --exclude-from)

  • --exclude 'pattern':
    Excludes files or directories matching the specified pattern from being synchronized.

  • --exclude-from=FILE:
    Reads a list of patterns from a file to exclude multiple files or directories.

    rsync -av --exclude 'tmp/' --exclude '*.log' source/ destination/

    Or, using an exclude file:

    rsync -av --exclude-from='exclude-list.txt' source/ destination/

Remote Transfers (-e ssh)

  • -e or --rsh=COMMAND:
    Specifies the remote shell to use. Typically, you use -e ssh for secure transfers.

    rsync -avz -e ssh source/ user@remotehost:/path/to/destination/

Practical Examples

Local Synchronization

Synchronize the contents of a local directory:

rsync -av /home/user/documents/ /mnt/backup/documents/

This command copies all files from /home/user/documents/ to /mnt/backup/documents/, preserving file permissions and timestamps.

Remote Synchronization over SSH

Transfer files securely to a remote server:

rsync -avz -e ssh /home/user/projects/ user@remote.server.com:/var/www/projects/

This command compresses data during transfer and uses SSH for secure communication.

Incremental Backups

Perform an incremental backup where only changed files are transferred:

rsync -av --delete /home/user/data/ /mnt/backup/data/

Files that have not changed since the last sync are skipped, and files removed from the source are also deleted from the backup.

Mirroring Directories

Create an exact mirror of a source directory, including deletion of files not present in the source:

rsync -av --delete /var/www/html/ /backup/www/html/

This is useful for web servers or other environments where the destination must match the source exactly.

Excluding Specific Files or Directories

Exclude temporary files and log files from the synchronization process:

rsync -av --exclude '*.tmp' --exclude 'logs/' /home/user/app/ /mnt/backup/app/

This command ensures that files ending in .tmp and the entire logs/ directory are not copied.

Advanced Usage and Tips

  • Dry Run (-n or --dry-run):
    Always test your rsync commands with the dry-run option to see what changes will be made without actually transferring files.

    rsync -av --delete --dry-run source/ destination/

  • Using Checksums (-c):
    If you need to verify file integrity beyond modification time and size, use the -c option to force rsync to compare files using a checksum.

    rsync -avc source/ destination/

  • Bandwidth Limiting (--bwlimit):
    Limit the bandwidth used by rsync during transfer to avoid saturating your network connection.

    rsync -av --bwlimit=1000 source/ destination/

  • Logging:
    Redirect output to a log file to keep a record of synchronization activity.

    rsync -av source/ destination/ | tee rsync.log

  • Combining with Cron for Scheduled Backups:
    Automate backups by scheduling rsync with cron. For example, create a cron job to run a backup every night:

    0 2 * * * rsync -av --delete /home/user/ /mnt/backup/

Conclusion and Further Reading

rsync is an incredibly powerful tool for file synchronization, backups, and mirroring. By mastering its numerous options—from simple file copies to complex remote synchronizations—you can automate and secure your data transfer processes with ease. Whether you're managing a single workstation or a fleet of servers, rsync provides the flexibility and efficiency required to keep your data in sync.

Further Reading and Resources

Experiment with different options and integrate rsync into your backup and deployment workflows to take full advantage of its capabilities. Happy synchronizing!

Last modified: 08 February 2025
pipe (|)sed