Linux Notes: experiments with rsync

The information presented here is intended for educational use by qualified computer technologists.
The information presented here is provided free of charge, as-is, with no warranty of any kind.

Introduction
My rsync scripts
New Block Diagram
Links

Edit: 2020-08-06 (fixed a potential SELinux problem)

rsync (a 3-km / 10,000-foot view)

"rsync -a" is a neat alternative to "cp -r"
use the "-X" or "--xattrs" switches to also copy meta-data which includes SELinux stuff
rsync can be used to do live incremental backups (only copy the files which were added or modified)
- a more dangerous mode allows rsync to delete destination files which were deleted from the source server
rsync is dependent upon SSH so you had better read this first
what follows are just extracts of my rsync experiments on a Linux system (running CentOS-7.5) connecting to a Linux system (also running CentOS-7.5) over a private maintenance network running 1 Gb/s

my rsync scripts

#!/bin/bash
# ===================================================================
# title  : neil_rsync_kawc4n_002.sh
# author : Neil Rieck
# created: 2019-09-23
# edit   : 2019-09-26
# edit   : 2020-08-06 (add support for SELinux)
# notes  : 
# 1) copy "some" folders via our private network
# 2) to develop skills for mirroring between Kitchener and Barrie
# 3) be very careful NOT to copy to self <<<---***
#    private network assignments:
#       IPv6   IPv4        host
#       fd::f0 192.168.200 kawc0f (PROD)
#       fd::f1 192.168.201 kawc3v (PROD - hot sync)
#       fd::e0 192.168.190 kawc4n (DVLP)
#       fd::e1 192.168.191 kawc4m (DVLP - hot sync)
# 4) place a public key at the destination to bypass the p/w prompt
# 5) rsync switches:
#    a) think of -a as -r with additional features
#    b) use -X or --xattrs to also copy meta-data including SELinux stuff
#    c) do not use -z on a fast link (compression slows the transfer)
#    d) use -P to see transmit speeds along with file percents
#    e) --bwlimit=125M should saturate a 1Gb/s link (careful!)
#    r) --bwlimit=0    means no limit (careful!)
# ===================================================================
#
#	SAFETY FIRST *** SAFETY FIRST *** SAFETY FIRST
#
#	define variables here to avoid typos below
#
safety="kawc4n"                  # change as required (check this)
#my_switch="-aXP --bwlimit=125M" # settings for a 1Gb/s private link (with percent)
my_switch="-aX --bwlimit=125M"   # settings for a 1Gb/s private link
#my_dest="[email protected]"  # IPv4 destination (check this)
my_dest="root@[fd::e1]"          # IPv6 destination (check this)
# ===================================================================
echo "-i-script: "$0
echo "-i-caveat: this script may only run on hostname '"${safety}"'"
set -e				# stop on error - VERY IMPORTANT
my_temp=${HOSTNAME}		# could return 'kawc??' or 'kawc??.on.bell.ca'
my_host=${my_temp%%.*}		# discard everything from the first dot onward
if [ ${my_host} == ${safety} ];
then
    echo "-i-okay to run on host: "${my_temp}
else
    echo "-e-not okay to run on host: "${my_temp}" so exiting"
    exit			# adios
fi
# ===================================================================
# copying begins
# ===================================================================
echo "-i-starting"
date
echo "task 1"
# -------------------------------------------------------------------
# copy web content to a remote machine (do not overwrite placeholder
# files already running there so choose a different folder)
#
# src:	  /var/www
# dst:    /backup/
# result: /backup/www/
# notes:
# 1) src - no trailing slash says to copy var and everything under it
# 2) dst - a trailing slash here means /backup must already exist on the
#          remote machine AND it is okay to create sub-directories
# 3) examples:
#    rsync -azPX	/var/www	[email protected]:/backup/
#    rsync -azPX	/var/www	root@[fd::e1]:/backup/
# -------------------------------------------------------------------
#
#	other important stuff
#
echo "task 2"
rsync ${my_switch}	/var/www	${my_dest}:/backup/
rsync ${my_switch}	/usr/local	${my_dest}:/backup/
rsync ${my_switch}	/etc		${my_dest}:/backup/
rsync ${my_switch} -P --delete /var/lib/maria-backups ${my_dest}:/backup/
#
#	copy "/home" part-3.1	(these three are copied in pieces)
#
echo "task 3.1"
rsync ${my_switch}	/home/neil	${my_dest}:/home/
rsync ${my_switch}	/home/dave	${my_dest}:/home/
rsync ${my_switch}	/home/vince	${my_dest}:/home/
#
#	copy "/home" part-3.2	(copy anything else here: mamgrp, etc.)
#
echo "task 3.2"
rsync ${my_switch} -P	/home		${my_dest}:/
#
#	scripts and utilities in /root
#	question: what would happen if we copy everything under .ssh (???)
#	Would it destroy files like identity and authorized_keys at the other side?
#
echo "task 4"
rsync ${my_switch}	/root/*.sh	${my_dest}:/root/
rsync ${my_switch}	/root/*.txt	${my_dest}:/root/
rsync ${my_switch}	/root/ssa*	${my_dest}:/root/
# ===================================================================
echo "-i-finished"
date
# ===================================================================

My New Block Diagram

   PROD (Linux)          DVLP (Linux)       other systems 
+-----------------+ +-----------------+ +-------------------+
| primary         | | primary         | | 4 OpenVMS systems |
+-----------------+ +-----------------+ +-------------------+

+-----------------+ +-----------------+ +-------------------+
| local stand by  | | local stand by  | | 2 Solaris systems |
+-----------------+ +-----------------+ +-------------------+

+-----------------+ +-----------------+
| remote stand by | | remote stand by |
+-----------------+ +-----------------+

primary employs rsync to copy to local stand by (same data facility) several times a day
primary employs rsync to copy to remote stand by (a different city more than 100-km away) several times a day
All Linux systems are currently running CentOS-7.7 with Apache and MariaDB
- having a local stand by can provide peace of mind when you wonder if the next YUM update might break something
- unlike Amazon or Alibaba, these systems do very little between 21:00 and 8:00
- this scheme is also useful when migrating to newer server hardware
The box labeled "4 other systems" are OpenVMS platforms
- these machines used to do daily backups to tape which were delivered off site (M-F, excluding holidays)
- Now, these machines copy their backups into a folder on "DVLP Linux primary" which are then rsync'd to local standby and remote standby every day

Linux Notes: experiments with rsync

rsync (a 3-km / 10,000-foot view)

my rsync scripts

My New Block Diagram

Links