public:snapshotting [Lorcan's Wiki]

Snapshotting your Home Directorys with Rsync

I use the following code to back up my home directories on my Mac using rsync. These scripts were created with advice from Mike Rubel's awesome page on snapshotting with Rsync. One script backs everything up every four hours, keeping the last four backups. The second backs it up every day, keeping the last four days. These scripts backup everything to the directory /snapshots which is owned and visible only by root.

The snapshots directory should be set up as follows:

mkdir /snapshots
sudo chown root:admin /snapshots
sudo chmod 740 /snapshots

Taking a Snapshot every Four Hours

I use this script to back up the home directorys every four hours:

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /Users whenever called
# ----------------------------------------------------------------------
# Amended by Lorcan Coyle, following on from
# http://www.mikerubel.org/computers/rsync_snapshots/#Rsync
# ----------------------------------------------------------------------

unset PATH      # suggestion from H. Milz: avoid accidental use of $PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/usr/bin/touch;
RSYNC=/usr/bin/rsync;

# ------------- file locations -----------------------------------------

# This directory will need to be created and should be owned by root and 
# invisible to everyone else
SNAPSHOTS_DIR=/snapshots;
# This is the location of the exclude file which controls what files are
# NOT backed up
EXCLUDES=/usr/local/etc/rsync_Users_exclude;
# Directory to be backed up
BACKUP_DIR=/Users/
# Snapshot file prefix
SNAPSHOT_PREFIX=Users

# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOTS_DIR/Users.hourly.3 ] ; then                         \
$RM -rf $SNAPSHOTS_DIR/Users.hourly.3 ;                                \
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.2 ] ; then                               \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.2 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.3 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.1 ] ; then                               \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.1 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.2 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.0 ] ; then                               \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.0 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.1 ; \
fi;

# step 4: rsync from the system into the latest snapshot (notice that

$RSYNC                                                                 \
        -a --delete --delete-excluded                                  \
        --link-dest=$SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.1           \
        --exclude-from="$EXCLUDES"                                     \
        $BACKUP_DIR $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.0 ;

# step 5: update the mtime of hourly.0 to reflect the snapshot time
$TOUCH $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.0 ;

# and thats it.

The script is stored in ~/bin/make_snapshot and called every four hours with the following entry in root's crontab (root's crontab must be used as the snapshot must be executed by root). root's crontab can be edited with the command:

sudo crontab -e

Add the following line to the crontab:

0 */4 * * * /home/lorcan/bin/make_snapshot.sh

The following is the contents of the exclude file for my /Users directory, which is saved as /usr/local/etc/rsync_Users_exclude:

.svn
.Trash
/lorcan/Pictures
/lorcan/Music
/lorcan/Movies
/lorcan/Library/Mail
/lorcan/Library/Thunderbird

This ensures that I do not back up my photos, music, movies, or email (these are backed up separately). I also don't backup any subversion directories (.svn) or the contents of Trash boxes (.Trash).

Taking a Daily Snapshot

The following script requires the above snapshot script to be in place and working. It copies the oldest of the above snapshots as a daily snapshot.

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility: daily snapshots
# ----------------------------------------------------------------------
# intended to be run daily as a cron job when hourly.3 contains the
# midnight (or whenever you want) snapshot; say, 13:00 for 4-hour snapshots.
# ----------------------------------------------------------------------
# Amended by Lorcan Coyle, following on from
# http://www.mikerubel.org/computers/rsync_snapshots/#Rsync
# ----------------------------------------------------------------------

unset PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CPIO=/usr/bin/cpio;
FIND=/usr/bin/find;
# ------------- file locations -----------------------------------------
# N.B. These file locations must match those in the earlier snapshot 
# script 

# This directory will need to be created and should be owned by root and 
# invisible to everyone else
SNAPSHOTS_DIR=/snapshots;
# Directory to be backed up
BACKUP_DIR=/Users/
# Snapshot file prefix
SNAPSHOT_PREFIX=Users
# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.6 ] ; then                   \
$RM -rf $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.6 ;                          \
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.5 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.5 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.6 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.4 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.4 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.5 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.3 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.3 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.4 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.2 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.2 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.3 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.1 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.1 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.2 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.0 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.0 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.1;  \
fi;

# step 3: make a hard-link-only (except for dirs) copy of
# hourly.3, assuming that exists, into daily.0
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.3 ] ; then                                                \
cd $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.hourly.3 && $FIND . -print | $CPIO -dpl ../$SNAPSHOT_PREFIX.daily.0 ; \
fi;

# note: do *not* update the mtime of daily.0; it will reflect
# when hourly.3 was made, which should be correct.

The daily snapshot script is stored in ~/bin/daily_snapshot_rotate.sh and called at 1am every day with the following entry in root's crontab.

0 1 * * *  /home/lorcan/bin/daily_snapshot_rotate.sh

Taking a Weekly Snapshot

The following script requires the daily script to be in place and working. It copies the oldest of the daily snapshots as a weekly snapshot.

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility: weekly snapshots
# ----------------------------------------------------------------------
# intended to be run weekly as a cron job when dailly.6 contains the
# oldest daily snapshot.
# ----------------------------------------------------------------------
# Amended by Lorcan Coyle, following on from
# http://www.mikerubel.org/computers/rsync_snapshots/#Rsync
# ----------------------------------------------------------------------

unset PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CPIO=/usr/bin/cpio;
FIND=/usr/bin/find;
# ------------- file locations -----------------------------------------
# N.B. These file locations must match those in the earlier snapshot
# script

# This directory will need to be created and should be owned by root and
# invisible to everyone else
SNAPSHOTS_DIR=/snapshots;
# Directory to be backed up
BACKUP_DIR=/Users/
# Snapshot file prefix
SNAPSHOT_PREFIX=Users
# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.4 ] ; then                   \
$RM -rf $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.4 ;                          \
fi ;

if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.3 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.3 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.4 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.2 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.2 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.3 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.1 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.1 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.2 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.0 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.0 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.1;  \
fi;

# step 3: make a hard-link-only (except for dirs) copy of
# daily.6, assuming that exists, into weekly.0
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.6 ] ; then                                                \
cd $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.daily.6 && $FIND . -print | $CPIO -dpl ../$SNAPSHOT_PREFIX.weekly.0 ; \
fi;

# note: do *not* update the mtime of weekly.0; it will reflect
# when daily.6 was made, which should be correct.

The weekly snapshot script is stored in ~/bin/weekly_snapshot_rotate.sh and called every Sunday at 2am with the following entry in root's crontab:

0 2 * * 0  /home/lorcan/bin/weekly_snapshot_rotate.sh

Taking a Monthly Snapshot

The following script requires the earlier script to be in place and working. It copies the oldest of the weekly snapshots as a monthly snapshot. It will keep six months worth of snapshots.

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility: monthly snapshots
# ----------------------------------------------------------------------
# intended to be run monthly as a cron job when weekly.4 contains the
# oldest weekly snapshot.
# ----------------------------------------------------------------------
# Amended by Lorcan Coyle, following on from
# http://www.mikerubel.org/computers/rsync_snapshots/#Rsync
# ----------------------------------------------------------------------

unset PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CPIO=/usr/bin/cpio;
FIND=/usr/bin/find;
# ------------- file locations -----------------------------------------
# N.B. These file locations must match those in the earlier snapshot
# script

# This directory will need to be created and should be owned by root and
# invisible to everyone else
SNAPSHOTS_DIR=/snapshots;
# Directory to be backed up
BACKUP_DIR=/Users/
# Snapshot file prefix
SNAPSHOT_PREFIX=Users
# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.6 ] ; then                   \
$RM -rf $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.6 ;                          \
fi ;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.5 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.5 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.6 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.4 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.4 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.5 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.3 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.3 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.4 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.2 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.2 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.3 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.1 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.1 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.2 ; \
fi;
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.0 ] ; then                              \
$MV $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.0 $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.monthly.1;  \
fi;

# step 3: make a hard-link-only (except for dirs) copy of
# weekly.4, assuming that exists, into monthly.0
if [ -d $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.4 ] ; then                                                \
cd $SNAPSHOTS_DIR/$SNAPSHOT_PREFIX.weekly.4 && $FIND . -print | $CPIO -dpl ../$SNAPSHOT_PREFIX.monthly.0 ; \
fi;

# note: do *not* update the mtime of monthly.0; it will reflect
# when weekly.4 was made, which should be correct.

The monthly snapshot script is stored in ~/bin/monthly_snapshot_rotate.sh and called on the first day of each month at 3am with the following entry in root's crontab:

0 3 * 1 *  /home/lorcan/bin/monthly_snapshot_rotate.sh

Complete crontab

For linux the complete crontab might look like this:

0 */4 * * * /home/lorcan/bin/make_snapshot.sh
0 1 * * *  /home/lorcan/bin/daily_snapshot_rotate.sh
0 2 * * 0  /home/lorcan/bin/weekly_snapshot_rotate.sh
0 3 * 1 *  /home/lorcan/bin/monthly_snapshot_rotate.sh

and for OSX it might look like this:

0 */4 * * * /Users/lorcan/bin/make_snapshot.sh
0 1 * * *  /Users/lorcan/bin/daily_snapshot_rotate.sh
0 2 * * 0  /Users/lorcan/bin/weekly_snapshot_rotate.sh
0 3 * 1 *  /Users/lorcan/bin/monthly_snapshot_rotate.sh

Openness: Advice and Improvements

Like Mike Rubel I am putting this code up here knowing that it is somewhat of a security risk. However, I'd appreciate any comments or suggestions for improvement of the scripts. I'll try to keep this page up to date with my current rsync settings. Thanks Mike :-)