Automated OSX backups with launchd and rsync

In this article, we’ll take a look at using Leopard’s launchd and rsync to develop a automated data backup methodology. In addition to rsync and launchd, we will also integrate Bernhard Baehr’s SleepWatcher utility.

In our environment, we use NetApp filers to host our network shares and are using CIFS network shares. However, with NetApp and Mac OS X, those file shares could be NFS, just as well. This will also work with your AFP shares.

Components

• launchd
  launchd is Apple’s (not-so-new) system startup program. launchd is more powerful than older programs such as init and cron, since it can not only start and stop programs but can also be triggered by other events, such as availability of network shares etc, which we will use in this setup.
• rsync
  Most people reading this article will already know what rsync is. rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated. In this setup, rsync does the heavy lifting of actually performing the backups.
• SleepWatcher
  is a niftly command line tool (daemon) developed by Bernhard Baehr for Mac OS X that monitors sleep, wakeup and idleness of a Mac. It can be used to execute a Unix command when the Mac or the display of the Mac goes to sleep mode or wakes up or after a given time without user interaction. We will be using SleepWatcher to do just that – to automatically mount and unmount network shares.

1. SleepWatcher
   As mentioned above, SleepWatcher monitors the sleep and wake states of a Mac (among other things). We’ll be using both of the SleepWatcher’s components, the SleepWatcher command line tool and the SleepWatcher StartupItem. As part of the tool’s functionality, SleepWatcher employs two hidden files, .wakeup and .sleep in the user’s home directory to execute user specific commands when a Mac wakes up from sleep or goes to sleep, as the case may be.

   First, we need a wakeup script to automatically mount our network shares when the laptop/Mac wakes up from sleep.

   /Users/username/.wakeup
   

   #!/bin/bash
   
   filer="bitbucket.example.com";
   test "$SHELLOPTS" && shopt -s xpg_echo
   
   function checkWorkNetwork() {
     # sleep for a bit, wait for network to show up
     sleep 30
     count=$(/sbin/ping -c 3 -o $filer | grep "received" | awk -F',' '{ print $2 }' | awk '{ print $1 }')
     if [ $count -eq 0 ]; then
       # 100% failed - try one more time after sleeping for 30 secs
       sleep 30
       count=$(/sbin/ping -c 3 -o $filer | grep "received" | awk -F',' '{ print $2 }' | awk '{ print $1 }')
       if [ $count -eq 0 ]; then
         /usr/bin/logger -t SleepWatcher:wakeup "filer not reachable...Exiting".
         exit;
       fi
     else
       /usr/bin/logger -t SleepWatcher:wakeup "filer is reachable...Continuing".
     fi
   }
   
   function doMounts {
     vol=$1;
     mnt_cnt=$(/sbin/mount | grep -ic "/Volumes/$vol")
     if [ $mnt_cnt -le 0 ]; then
       /usr/bin/osascript <<EOT
       tell application "Finder" to mount volume "cifs://$filer/$vol"
   EOT
     fi
   }
   
   checkWorkNetwork
   doMounts Backups
   
   exit 0
   

   In the above wakeup script, first we check to see if we are on our work network using the function
   checkWorkNetwork . and mount the network resources. This function can easily be modified to adapt to a home network. At home, if we are on a wireless network, a function like the following may be used.

   function checknMountHomeNetwork {
    # sleep for a bit, wait for network to show up
    sleep 60
    homelan=$(/usr/sbin/system_profiler SPAirPortDataType | awk -F': ' '/Current Wireless Network/{print $2}')
   
    if [ "$homelan" == "JewelThief-Fake" ]; then
           /usr/bin/osascript <<EOT
           tell application "Finder" to mount volume "cifs://192.168.1.3/Documents"
   EOT
           /usr/bin/logger -t SleepWatcher:wakeup "Mounted Home Volumes...Exiting"
           exit 0
    else
           /usr/bin/logger -t SleepWatcher:wakeup "Cannot ascertain home network...Exiting"
    fi
   }
   

   Next, we need a sleep script to trigger the stoppage of any running rsync backups and unmount the network shares.

   The sleep script needs to execute and finish within 15 seconds. After this timeout, the system forces the sleep mode and your script may not finish properly.

   /Users/username/.sleep
   

   #!/bin/bash
   
   test "$SHELLOPTS" && shopt -s xpg_echo
   
   function doUmounts {
           /usr/bin/logger -t SleepWatcher:sleep "Going to sleep...unmounting network volumes"
           /usr/bin/osascript -e 'tell application "Finder" to eject (every disk whose local volume is false)'
   }
   
   function kill_rsync {
           /usr/bin/logger -t SleepWatcher:sleep "Stopping rsync"
           pid=`ps auxw | grep rsync | grep -v grep | awk '{print $2}'`
           while [ ! -z "$pid" ]; do
                   echo -n "."
                   for pid in $pid
                   do
                           kill -9 $pid
                   done
                   sleep 3
                   pid=`ps auxw | grep rsync | grep -v grep | awk '{print $2}'`
           done
   }
   
   kill_rsync
   doUmounts
   
   exit 0
   

   In the above sleep script, we first kill off any running rsync jobs using the kill_rsync function and unmount the network shares using the doUmounts function.

Now that we have the wakeup and sleep scripts, our network shares will be automatically mounted and unmounted. We will now need a way to detect these network shares transitions and periodically launch our backup scripts. For this, we need a rsync backup script and a launchd hook to manage the automatic launch of this script.

Let’s first take a look at the rsync script itself. In our example, the script is installed in /usr/local/.

2. /usr/local/laptop_rsync.sh
   

   #!/bin/sh
   
   PROG=$0
   SRC="$HOME/Documents/"
   DST="/Volumes/Backups/$USER/"
   CMD="/usr/bin/rsync -q  --numeric-ids -E  -axzS --exclude-from=$HOME/.rsync_excludes.txt $SRC $DST"
   
   if [ ! -r $SRC ]; then
    /usr/bin/logger -t $PROG "Source $SRC not readable - Cannot start the sync process"
    exit;
   fi
   
   if [ ! -w $DST ]; then
    /usr/bin/logger -t $PROG "Destination $DST not writeable - Cannot start the sync process"
    exit;
   fi
   
   /usr/bin/logger -t $PROG "Start rsync"
   $CMD
   /usr/bin/logger -t $PROG "End rsync"
   
   exit 0
   

   In this script, we are using rsync to synchronize the Mac user’s Documents/ directory to a network share. We are also using a exclude list here called .ib_backup_excludes.txt to prevent any unwanted/personal files and directories from being synchronized to the network share.

   $HOME/.rsync_excludes.txt
/tmp/*
/Network/*
/cores/*
*/.Trash
/afs/*
/automount/*
/private/tmp/*
/private/var/run/*
/private/var/spool/postfix/*
/private/var/vm/*
/Previous Systems.localized
.Spotlight-*/
._*
Personal
personal


   Most importantly, if we create a directory called Personal or personal within our Documents/ directory and place all of our personal files, they will not be synchronized to our network files share.

3. Lastly, we need way to automatically run this rsync script (and not run it when the network shares have disappeared). Here’s where Apple’s launchd comes in. We will give control of the above laptop_rsync.sh script to launchd by creating a launchd agent file. From the launchd.plist man page, the agent files can be deployed in several locations, depending on whether this is done by a regular user or by an administrator. In our example, I am using the administrator location and installing the launchd agent file in /Library/LaunchAgents/.

   The rsync launchd agent file is /Library/LaunchAgents/name.rajeev.laptop_rsync.plist

   <?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
   <plist version="1.0">
   <dict>
           <key>KeepAlive</key>
           <dict>
                   <key>PathState</key>
                   <dict>
                           <key>/Volumes/Documents/</key>
                           <true/>
                   </dict>
           </dict>
           <key>Label</key>
           <string>name.rajeev.laptop_rsync</string>
           <key>Program</key>
           <string>/usr/local/laptop_rsync.sh</string>
           <key>StartInterval</key>
           <integer>3600</integer>
           <key>ThrottleInterval</key>
           <integer>3600</integer>
           <key>WorkingDirectory</key>
           <string>/usr/local/</string>
   </dict>
   </plist>
   

   In the above plist agent file, note that we are using the key PathState. launchd monitors for this path specified by this key to become available and launches the script once the path becomes available. In our example, once this path becomes available (via the wakeup script which is in turn launched by SleepWatcher), the rsync backup job is kicked off. Also note that we are using a ThrottleInterval to prevent launchd from starting the rsync script too fast.

   This agent file can itself be loaded upon restart by using the $HOME/.launchd.conf or /etc/launchd.conf. The command to load and unload this agent into launchd is below:

   • launchd load /Library/LaunchAgents/name.rajeev.laptop_rsync.plist. This will load this agent into launchd
   • launchd unload /Library/LaunchAgents/name.rajeev.laptop_rsync.plist. This will unload this agent into launchd