November 22, 2014

Automated OSX backups with launchd and rsync

In this article, we’ll take a look at using Leopard’s launchd and rsync to develop a automated data backup methodology. In addition to rsync and launchd, we will also integrate Bernhard Baehr’s SleepWatcher utility.

In our environment, we use NetApp filers to host our network shares and are using CIFS network shares. However, with NetApp and Mac OS X, those file shares could be NFS, just as well. This will also work with your AFP shares.


Components

  • launchd
    launchd is Apple’s (not-so-new) system startup program. launchd is more powerful than older programs such as init and cron, since it can not only start and stop programs but can also be triggered by other events, such as availability of network shares etc, which we will use in this setup.
  • rsync
    Most people reading this article will already know what rsync is. rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated. In this setup, rsync does the heavy lifting of actually performing the backups.
  • SleepWatcher
    is a niftly command line tool (daemon) developed by Bernhard Baehr for Mac OS X that monitors sleep, wakeup and idleness of a Mac. It can be used to execute a Unix command when the Mac or the display of the Mac goes to sleep mode or wakes up or after a given time without user interaction. We will be using SleepWatcher to do just that – to automatically mount and unmount network shares.
  1. SleepWatcher
    As mentioned above, SleepWatcher monitors the sleep and wake states of a Mac (among other things). We’ll be using both of the SleepWatcher’s components, the SleepWatcher command line tool and the SleepWatcher StartupItem. As part of the tool’s functionality, SleepWatcher employs two hidden files, .wakeup and .sleep in the user’s home directory to execute user specific commands when a Mac wakes up from sleep or goes to sleep, as the case may be.

    First, we need a wakeup script to automatically mount our network shares when the laptop/Mac wakes up from sleep.

    /Users/username/.wakeup

    #!/bin/bash
    
    filer="bitbucket.example.com";
    test "$SHELLOPTS" && shopt -s xpg_echo
    
    function checkWorkNetwork() {
      # sleep for a bit, wait for network to show up
      sleep 30
      count=$(/sbin/ping -c 3 -o $filer | grep "received" | awk -F',' '{ print $2 }' | awk '{ print $1 }')
      if [ $count -eq 0 ]; then
        # 100% failed - try one more time after sleeping for 30 secs
        sleep 30
        count=$(/sbin/ping -c 3 -o $filer | grep "received" | awk -F',' '{ print $2 }' | awk '{ print $1 }')
        if [ $count -eq 0 ]; then
          /usr/bin/logger -t SleepWatcher:wakeup "filer not reachable...Exiting".
          exit;
        fi
      else
        /usr/bin/logger -t SleepWatcher:wakeup "filer is reachable...Continuing".
      fi
    }
    
    function doMounts {
      vol=$1;
      mnt_cnt=$(/sbin/mount | grep -ic "/Volumes/$vol")
      if [ $mnt_cnt -le 0 ]; then 
        /usr/bin/osascript <<EOT
        tell application "Finder" to mount volume "cifs://$filer/$vol"
    EOT
      fi
    }
    
    checkWorkNetwork
    doMounts Backups
    
    exit 0
    
    

    In the above wakeup script, first we check to see if we are on our work network using the function
    checkWorkNetwork . and mount the network resources. This function can easily be modified to adapt to a home network. At home, if we are on a wireless network, a function like the following may be used.

    function checknMountHomeNetwork {
     # sleep for a bit, wait for network to show up
     sleep 60
     homelan=$(/usr/sbin/system_profiler SPAirPortDataType | awk -F': ' '/Current Wireless Network/{print $2}')
    
     if [ "$homelan" == "JewelThief-Fake" ]; then
            /usr/bin/osascript <<EOT
            tell application "Finder" to mount volume "cifs://192.168.1.3/Documents"
    EOT
            /usr/bin/logger -t SleepWatcher:wakeup "Mounted Home Volumes...Exiting"
            exit 0
     else
            /usr/bin/logger -t SleepWatcher:wakeup "Cannot ascertain home network...Exiting"
     fi
    }
    
    

    Next, we need a sleep script to trigger the stoppage of any running rsync backups and unmount the network shares.

    The sleep script needs to execute and finish within 15 seconds. After this timeout, the system forces the sleep mode and your script may not finish properly.

    /Users/username/.sleep

    #!/bin/bash
    
    test "$SHELLOPTS" && shopt -s xpg_echo
    
    function doUmounts {
            /usr/bin/logger -t SleepWatcher:sleep "Going to sleep...unmounting network volumes"
            /usr/bin/osascript -e 'tell application "Finder" to eject (every disk whose local volume is false)'
    }
    
    function kill_rsync {
            /usr/bin/logger -t SleepWatcher:sleep "Stopping rsync"
            pid=`ps auxw | grep rsync | grep -v grep | awk '{print $2}'` 
            while [ ! -z "$pid" ]; do
                    echo -n "."
                    for pid in $pid
                    do
                            kill -9 $pid
                    done
                    sleep 3
                    pid=`ps auxw | grep rsync | grep -v grep | awk '{print $2}'` 
            done
    }
    
    kill_rsync
    doUmounts
    
    exit 0
    
    

    In the above sleep script, we first kill off any running rsync jobs using the kill_rsync function and unmount the network shares using the doUmounts function.

  2. Now that we have the wakeup and sleep scripts, our network shares will be automatically mounted and unmounted. We will now need a way to detect these network shares transitions and periodically launch our backup scripts. For this, we need a rsync backup script and a launchd hook to manage the automatic launch of this script.

    Let’s first take a look at the rsync script itself. In our example, the script is installed in /usr/local/.

  3. /usr/local/laptop_rsync.sh
    #!/bin/sh
    
    PROG=$0
    SRC="$HOME/Documents/"
    DST="/Volumes/Backups/$USER/"
    CMD="/usr/bin/rsync -q  --numeric-ids -E  -axzS --exclude-from=$HOME/.rsync_excludes.txt $SRC $DST"
    
    if [ ! -r $SRC ]; then
     /usr/bin/logger -t $PROG "Source $SRC not readable - Cannot start the sync process"
     exit;
    fi
    
    if [ ! -w $DST ]; then
     /usr/bin/logger -t $PROG "Destination $DST not writeable - Cannot start the sync process"
     exit;
    fi
    
    /usr/bin/logger -t $PROG "Start rsync"
    $CMD
    /usr/bin/logger -t $PROG "End rsync"
    
    exit 0
    
    

    In this script, we are using rsync to synchronize the Mac user’s Documents/ directory to a network share. We are also using a exclude list here called .ib_backup_excludes.txt to prevent any unwanted/personal files and directories from being synchronized to the network share.

    $HOME/.rsync_excludes.txt
    /tmp/*
    /Network/*
    /cores/*
    */.Trash
    /afs/*
    /automount/*
    /private/tmp/*
    /private/var/run/*
    /private/var/spool/postfix/*
    /private/var/vm/*
    /Previous Systems.localized
    .Spotlight-*/
    ._*
    Personal
    personal

    Most importantly, if we create a directory called Personal or personal within our Documents/ directory and place all of our personal files, they will not be synchronized to our network files share.

  4. Lastly, we need way to automatically run this rsync script (and not run it when the network shares have disappeared). Here’s where Apple’s launchd comes in. We will give control of the above laptop_rsync.sh script to launchd by creating a launchd agent file. From the launchd.plist man page, the agent files can be deployed in several locations, depending on whether this is done by a regular user or by an administrator. In our example, I am using the administrator location and installing the launchd agent file in /Library/LaunchAgents/.

    The rsync launchd agent file is /Library/LaunchAgents/name.rajeev.laptop_rsync.plist

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
            <key>KeepAlive</key>
            <dict>
                    <key>PathState</key>
                    <dict>
                            <key>/Volumes/Documents/</key>
                            <true/>
                    </dict>
            </dict>
            <key>Label</key>
            <string>name.rajeev.laptop_rsync</string>
            <key>Program</key>
            <string>/usr/local/laptop_rsync.sh</string>
            <key>StartInterval</key>
            <integer>3600</integer>
            <key>ThrottleInterval</key>
            <integer>3600</integer>
            <key>WorkingDirectory</key>
            <string>/usr/local/</string>
    </dict>
    </plist>
    

    In the above plist agent file, note that we are using the key PathState. launchd monitors for this path specified by this key to become available and launches the script once the path becomes available. In our example, once this path becomes available (via the wakeup script which is in turn launched by SleepWatcher), the rsync backup job is kicked off. Also note that we are using a ThrottleInterval to prevent launchd from starting the rsync script too fast.

    This agent file can itself be loaded upon restart by using the $HOME/.launchd.conf or /etc/launchd.conf. The command to load and unload this agent into launchd is below:

    • launchd load /Library/LaunchAgents/name.rajeev.laptop_rsync.plist. This will load this agent into launchd
    • launchd unload /Library/LaunchAgents/name.rajeev.laptop_rsync.plist. This will unload this agent into launchd

Comments

  1. have you looked at backuppc for this? i’ve used it in limited trials, and it works seamlessly. check the project at: http://backuppc.sourceforge.net/

  2. I like your script and it works fine on a personal computer when backs up to internal or external HD!
    But how would you incorporate this rsyn&Launchd command into a netware directory using afp or smb where the path to the server changes regularly (for security reason) so if I use:
    mount_afp afp://usernam:password@myserver-nw/KRIA.myserver myserver, ths KRIA part changes to WOW after two days:
    mount_afp afp://usernam:password@myserver-nw/WOW.myserver myserver
    How can I avoid disconnections and to ensure that is running smoothly?
    Thanks, Xosraw

  3. Thanks for posting this, it’s very useful.

Trackbacks

  1. [...] Automated OSX backups with launchd and rsync · Rajeev Karamchedu (tags: backup rsync macosx launchd ) [...]

  2. [...] Automated OSX backups with launchd and rsync · Rajeev Karamchedu (tags:backup rsync macosx launchd ) [...]

  3. [...] Automated OSX backups with launchd and rsync · Rajeev Karamchedu By mohangk Automated OSX backups with launchd and rsync · Rajeev Karamchedu [...]

  4. [...] View the Gist. Adapted from the rsync script at Automated OSX backups with launchd and rsync [...]

Speak Your Mind

*