2 Commits

Author SHA1 Message Date
Laurent Cozic cca0c1ac56 Removed test code 2013-11-15 15:34:17 +08:00
Laurent Cozic 78003f6b94 Support for Mac OS X 2013-11-15 14:59:59 +08:00
3 changed files with 170 additions and 459 deletions
-3
View File
@@ -1,3 +0,0 @@
.idea
test.sh
*~
+17 -99
View File
@@ -1,48 +1,14 @@
# Rsync time backup # Rsync time backup
This script offers Time Machine-style backup using rsync. It creates incremental backups of files and directories to the destination of your choice. The backups are structured in a way that makes it easy to recover any file at any point in time. Time Machine style backup with rsync. Should work on Linux, Mac OS X and Windows with Cygwin.
It works on Linux, macOS and Windows (via WSL or Cygwin). The main advantage over Time Machine is the flexibility as it can backup from/to any filesystem and works on any platform. You can also backup, for example, to a Truecrypt drive without any problem. # Installation
On macOS, it has a few disadvantages compared to Time Machine - in particular it does not auto-start when the backup drive is plugged (though it can be achieved using a launch agent), it requires some knowledge of the command line, and no specific GUI is provided to restore files. Instead files can be restored by using any file explorer, including Finder, or the command line.
## Installation
git clone https://github.com/laurent22/rsync-time-backup git clone https://github.com/laurent22/rsync-time-backup
## Usage # Usage
Usage: rsync_tmbackup.sh [OPTION]... <[USER@HOST:]SOURCE> <[USER@HOST:]DESTINATION> [exclude-pattern-file] rsync_tmbackup.sh <source> <destination> [excluded-pattern-path]
Options
-p, --port SSH port.
-h, --help Display this help message.
--rsync-get-flags Display the default rsync flags that are used for backup.
--rsync-set-flags Set the rsync flags that are going to be used for backup.
--log-dir Set the log file directory. If this flag is set, generated files will
not be managed by the script - in particular they will not be
automatically deleted.
--strategy Set the expiration strategy. Default: "365:30 30:7 1:1" means after one
day, keep one backup per day. After 30 days, keep one backup every 7 days.
After 365 days keep one backup every 30 days.
## Features
* Each backup is on its own folder named after the current timestamp. Files can be copied and restored directly, without any intermediate tool.
* Backup to/from remote destinations over SSH.
* Files that haven't changed from one backup to the next are hard-linked to the previous backup so take very little extra space.
* Safety check - the backup will only happen if the destination has explicitly been marked as a backup destination.
* Resume feature - if a backup has failed or was interrupted, the tool will resume from there on the next backup.
* Exclude file - support for pattern-based exclusion via the `--exclude-from` rsync parameter.
* Automatically purge old backups - within 24 hours, all backups are kept. Within one month, the most recent backup for each day is kept. For all previous backups, the most recent of each month is kept.
* "latest" symlink that points to the latest successful backup.
## Examples ## Examples
@@ -53,79 +19,31 @@ On macOS, it has a few disadvantages compared to Time Machine - in particular it
* Backup with exclusion list: * Backup with exclusion list:
rsync_tmbackup.sh /home /mnt/backup_drive excluded_patterns.txt rsync_tmbackup.sh /home /mnt/backup_drive excluded_patterns.txt
* Backup to remote drive over SSH, on port 2222:
rsync_tmbackup.sh -p 2222 /home user@example.com:/mnt/backup_drive
* Backup from remote drive over SSH:
rsync_tmbackup.sh user@example.com:/home /mnt/backup_drive
* To mimic Time Machine's behaviour, a cron script can be setup to backup at regular interval. For example, the following cron job checks if the drive "/mnt/backup" is currently connected and, if it is, starts the backup. It does this check every 1 hour.
0 */1 * * * if [[ -d /mnt/backup ]]; then rsync_tmbackup.sh /home /mnt/backup; fi
## Backup expiration logic
The script automatically deletes old backups using the following logic:
- Within the last 24 hours, all the backups are kept.
- Within the last 31 days, the most recent backup of each day is kept.
- After 31 days, only the most recent backup of each month is kept.
- Additionally, if the backup destination directory is full, the oldest backups are deleted until enough space is available.
## Exclude file ## Exclude file
An optional exclude file can be provided as a third parameter. It should be compatible with the `--exclude-from` parameter of rsync. See [this tutorial](https://sites.google.com/site/rsync2u/home/rsync-tutorial/the-exclude-from-option) for more information. An optional exclude file can be provided as a third parameter. It should be compabible with the `--exclude-from` parameter of rsync. See [this tutorial] (https://sites.google.com/site/rsync2u/home/rsync-tutorial/the-exclude-from-option) for more information.
## Built-in lock # Features
The script is designed so that only one backup operation can be active for a given directory. If a new backup operation is started while another is still active (i.e. it has not finished yet), the new one will be automaticalled interrupted. Thanks to this the use of `flock` to run the script is not necessary. * Each backup is on its own folder named after the current timestamp. Files can be copied and restored directly, without any intermediate tool.
## Rsync options * Files that haven't changed from one backup to the next are hard-linked to the previous backup so take very little extra space.
To display the rsync options that are used for backup, run `./rsync_tmbackup.sh --rsync-get-flags`. It is also possible to add or remove options using the `--rsync-set-flags` option. For example, to exclude backing up permissions and groups: * Safety check - the backup will only happen if the destination has explicitely been marked as a backup destination.
rsync_tmbackup --rsync-set-flags "--numeric-ids --links --hard-links \ * Resume feature - if a backup has failed or was interrupted, the tool will resume from there on the next backup.
--one-file-system --archive --no-perms --no-groups --itemize-changes" /src /dest
## How to restore
The script creates a backup in a regular directory so you can simply copy the files back to the original directory. You could do that with something like `rsync -aP /path/to/last/backup/ /path/to/restore/to/`. Consider using the `--dry-run` option to check what exactly is going to be copied. Use `--delete` if you also want to delete files that exist in the destination but not in the backup (obviously extra care must be taken when using this option). * Exclude file - support for pattern-based exclusion via the `--exclude-from` rsync parameter.
## Extensions * The application is one bash script that can be easily edited.
* [rtb-wrapper](https://github.com/thomas-mc-work/rtb-wrapper): Allows creating backup profiles in config files. Handles both backup and restore operations. # TODO
* [time-travel](https://github.com/joekerna/time-travel): Smooth integration into OSX Notification Center
## TODO
* Check source and destination file-system (`df -T /dest`). If one of them is FAT, use the --modify-window rsync parameter (see `man rsync`) with a value of 1 or 2
* Add `--whole-file` arguments on Windows? See http://superuser.com/a/905415/73619
* Minor changes (see TODO comments in the source). * Minor changes (see TODO comments in the source).
## LICENSE # LICENSE
The MIT License (MIT) [MIT](http://opensource.org/licenses/MIT)
Copyright (c) 2013-2017 Laurent Cozic [![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/laurent22/rsync-time-backup/trend.png)](https://bitdeli.com/free "Bitdeli Badge")
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
Executable → Regular
+153 -357
View File
@@ -1,20 +1,19 @@
#!/usr/bin/env bash #!/usr/bin/env bash
APPNAME=$(basename $0 | sed "s/\.sh$//")
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Log functions # Log functions
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
fn_log_info() { echo "$APPNAME: $1"; } fn_log_info() {
fn_log_warn() { echo "$APPNAME: [WARNING] $1" 1>&2; } echo "rsync_tmbackup: $1"
fn_log_error() { echo "$APPNAME: [ERROR] $1" 1>&2; } }
fn_log_info_cmd() {
if [ -n "$SSH_DEST_FOLDER_PREFIX" ]; then fn_log_warn() {
echo "$APPNAME: $SSH_CMD '$1'"; echo "rsync_tmbackup: [WARNING] $1"
else }
echo "$APPNAME: $1";
fi fn_log_error() {
echo "rsync_tmbackup: [ERROR] $1"
} }
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
@@ -22,7 +21,7 @@ fn_log_info_cmd() {
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
fn_terminate_script() { fn_terminate_script() {
fn_log_info "SIGINT caught." echo "rsync_tmbackup: SIGINT caught."
exit 1 exit 1
} }
@@ -31,276 +30,42 @@ trap 'fn_terminate_script' SIGINT
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Small utility functions for reducing code duplication # Small utility functions for reducing code duplication
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
fn_display_usage() {
echo "Usage: $(basename $0) [OPTION]... <[USER@HOST:]SOURCE> <[USER@HOST:]DESTINATION> [exclude-pattern-file]"
echo ""
echo "Options"
echo " -p, --port SSH port."
echo " -h, --help Display this help message."
echo " --rsync-get-flags Display the default rsync flags that are used for backup."
echo " --rsync-set-flags Set the rsync flags that are going to be used for backup."
echo " --log-dir Set the log file directory. If this flag is set, generated files will"
echo " not be managed by the script - in particular they will not be"
echo " automatically deleted."
echo " Default: $LOG_DIR"
echo " --strategy Set the expiration strategy. Default: \"365:30 30:7 1:1\" means after one"
echo " day, keep one backup per day. After 30 days, keep one backup every 7 days."
echo " After 365 days keep one backup every 30 days."
echo ""
echo "For more detailed help, please see the README file:"
echo ""
echo "https://github.com/laurent22/rsync-time-backup/blob/master/README.md"
}
fn_parse_date() { fn_parse_date() {
local date_string="$1"
local date_format="$2"
# Converts YYYY-MM-DD-HHMMSS to YYYY-MM-DD HH:MM:SS and then to Unix Epoch. # Converts YYYY-MM-DD-HHMMSS to YYYY-MM-DD HH:MM:SS and then to Unix Epoch.
case "$OSTYPE" in
if [[ -z "$date_format" || "$date_format" == "Y-m-d H:i:s" ]]; then linux*) date -d "${1:0:10} ${1:11:2}:${1:13:2}:${1:15:2}" +%s ;;
case "$OSTYPE" in darwin*) date -j -f "%Y-%m-%d-%H%M%S" "$1" "+%s" ;;
linux*) date -d "${date_string:0:10} ${date_string:1date_string:2}:${date_string:13:2}:${date_string:15:2}" +%s ;; esac
cygwin*) date -d "${date_string:0:10} ${date_string:1date_string:2}:${date_string:13:2}:${date_string:15:2}" +%s ;;
darwin*) date -j -f "%Y-%m-%d-%H%M%S" "$date_string" "+%s" ;;
FreeBSD*) date -j -f "%Y-%m-%d-%H%M%S" "$date_string" "+%s" ;;
esac
else
case "$OSTYPE" in
linux*) date -d "${date_string:0:10} 00:00:00" +%s ;;
cygwin*) date -d "${date_string:0:10} 00:00:00" +%s ;;
darwin*) date -j -f "%Y-%m-%d" "$date_string" "+%s" ;;
FreeBSD*) date -j -f "%Y-%m-%d" "$date_string" "+%s" ;;
esac
fi
}
fn_run_cmd() {
if [ -n "$SSH_DEST_FOLDER_PREFIX" ]
then
eval "$SSH_CMD '$1'"
else
eval $1
fi
} }
fn_find_backups() { fn_find_backups() {
fn_run_cmd "find "$DEST_FOLDER" -maxdepth 1 -type d -name \"????-??-??-??????\" -prune | sort -r" find "$DEST_FOLDER" -type d -name "????-??-??-??????" -prune
} }
fn_expire_backup() { fn_expire_backup() {
# Double-check that we're on a backup destination to be completely # Double-check that we're on a backup destination to be completely
# sure we're deleting the right folder # sure we're deleting the right folder
if [ -z "$(fn_find_backup_marker "$(dirname -- "$1")")" ]; then if [ -z "$(fn_is_backup_destination "$(dirname -- "$1")")" ]; then
fn_log_error "$1 is not on a backup destination - aborting." fn_log_error "$1 is not on a backup destination - aborting."
exit 1 exit 1
fi fi
fn_log_info "Expiring $1" fn_log_info "Expiring $1"
fn_rm_dir "$1" rm -rf -- "$1"
}
fn_expire_backups() {
local current_timestamp=$EPOCH
local last_kept_timestamp=9999999999
# Process each backup dir from most recent to oldest
for backup_dir in $(fn_find_backups | sort -r); do
local backup_date=$(basename "$backup_dir")
local backup_day=${backup_date:0:10}
local backup_timestamp=$(fn_parse_date $backup_day "Y-m-d")
# Skip if failed to parse date...
if [ -z "$backup_timestamp" ]; then
fn_log_warn "Could not parse date: $backup_dir"
continue
fi
# Find which strategy token applies to this particular backup
for strategy_token in $(echo $EXPIRATION_STRATEGY | tr " " "\n" | sort -r -n); do
IFS=':' read -r -a t <<< "$strategy_token"
# After which date (relative to today) this token applies (X)
local cut_off_timestamp=$((current_timestamp - ${t[0]} * 86400))
# Every how many days should a backup be kept past the cut off date (Y)
local cut_off_interval=$((${t[1]} * 86400))
# If we've found the strategy token that applies to this backup
if [ "$backup_timestamp" -le "$cut_off_timestamp" ]; then
# Special case: if Y is "0" we delete every time
if [ $cut_off_interval -eq "0" ]; then
fn_expire_backup "$backup_dir"
break
fi
# Check if the current backup is in the interval between
# the last backup that was kept and Y
local interval_since_last_kept=$((last_kept_timestamp - backup_timestamp))
if [ "$interval_since_last_kept" -lt "$cut_off_interval" ]; then
# Yes: Delete that one
fn_expire_backup "$backup_dir"
else
# No: Keep it
last_kept_timestamp=$backup_timestamp
fi
break
fi
done
done
}
fn_parse_ssh() {
if [[ "$DEST_FOLDER" =~ ^[A-Za-z0-9\._%\+\-]+@[A-Za-z0-9.\-]+\:.+$ ]]
then
SSH_USER=$(echo "$DEST_FOLDER" | sed -E 's/^([A-Za-z0-9\._%\+\-]+)@([A-Za-z0-9.\-]+)\:(.+)$/\1/')
SSH_HOST=$(echo "$DEST_FOLDER" | sed -E 's/^([A-Za-z0-9\._%\+\-]+)@([A-Za-z0-9.\-]+)\:(.+)$/\2/')
SSH_DEST_FOLDER=$(echo "$DEST_FOLDER" | sed -E 's/^([A-Za-z0-9\._%\+\-]+)@([A-Za-z0-9.\-]+)\:(.+)$/\3/')
SSH_CMD="ssh -p $SSH_PORT ${SSH_USER}@${SSH_HOST}"
SSH_DEST_FOLDER_PREFIX="${SSH_USER}@${SSH_HOST}:"
elif [[ "$SRC_FOLDER" =~ ^[A-Za-z0-9\._%\+\-]+@[A-Za-z0-9.\-]+\:.+$ ]]
then
SSH_USER=$(echo "$SRC_FOLDER" | sed -E 's/^([A-Za-z0-9\._%\+\-]+)@([A-Za-z0-9.\-]+)\:(.+)$/\1/')
SSH_HOST=$(echo "$SRC_FOLDER" | sed -E 's/^([A-Za-z0-9\._%\+\-]+)@([A-Za-z0-9.\-]+)\:(.+)$/\2/')
SSH_SRC_FOLDER=$(echo "$SRC_FOLDER" | sed -E 's/^([A-Za-z0-9\._%\+\-]+)@([A-Za-z0-9.\-]+)\:(.+)$/\3/')
SSH_CMD="ssh -p $SSH_PORT ${SSH_USER}@${SSH_HOST}"
SSH_SRC_FOLDER_PREFIX="${SSH_USER}@${SSH_HOST}:"
fi
}
fn_find() {
fn_run_cmd "find '$1'" 2>/dev/null
}
fn_get_absolute_path() {
fn_run_cmd "cd '$1';pwd"
}
fn_mkdir() {
fn_run_cmd "mkdir -p -- '$1'"
}
# Removes a file or symlink - not for directories
fn_rm_file() {
fn_run_cmd "rm -f -- '$1'"
}
fn_rm_dir() {
fn_run_cmd "rm -rf -- '$1'"
}
fn_touch() {
fn_run_cmd "touch -- '$1'"
}
fn_ln() {
fn_run_cmd "ln -s -- '$1' '$2'"
} }
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Source and destination information # Source and destination information
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
SSH_USER=""
SSH_HOST=""
SSH_DEST_FOLDER=""
SSH_SRC_FOLDER=""
SSH_CMD=""
SSH_DEST_FOLDER_PREFIX=""
SSH_SRC_FOLDER_PREFIX=""
SSH_PORT="22"
SRC_FOLDER="" SRC_FOLDER=${1%/}
DEST_FOLDER="" DEST_FOLDER=${2%/}
EXCLUSION_FILE="" EXCLUSION_FILE=$3
LOG_DIR="$HOME/.$APPNAME"
AUTO_DELETE_LOG="1"
EXPIRATION_STRATEGY="365:30 1:1 30:7"
RSYNC_FLAGS="-D --compress --numeric-ids --links --hard-links --one-file-system --itemize-changes --times --recursive --perms --owner --group --stats --human-readable" for arg in "$SRC_FOLDER" "$DEST_FOLDER" "$EXCLUSION_FILE"; do
if [[ "$arg" == *"'"* ]]; then
while :; do fn_log_error 'Arguments may not have any single quote characters.'
case $1 in
-h|-\?|--help)
fn_display_usage
exit
;;
-p|--port)
shift
SSH_PORT=$1
;;
--rsync-get-flags)
shift
echo $RSYNC_FLAGS
exit
;;
--rsync-set-flags)
shift
RSYNC_FLAGS="$1"
;;
--strategy)
shift
STRATEGY="$1"
;;
--log-dir)
shift
LOG_DIR="$1"
AUTO_DELETE_LOG="0"
;;
--)
shift
SRC_FOLDER="$1"
DEST_FOLDER="$2"
EXCLUSION_FILE="$3"
break
;;
-*)
fn_log_error "Unknown option: \"$1\""
fn_log_info ""
fn_display_usage
exit 1
;;
*)
SRC_FOLDER="$1"
DEST_FOLDER="$2"
EXCLUSION_FILE="$3"
break
esac
shift
done
# Display usage information if required arguments are not passed
if [[ -z "$SRC_FOLDER" || -z "$DEST_FOLDER" ]]; then
fn_display_usage
exit 1
fi
# Strips off last slash from dest. Note that it means the root folder "/"
# will be represented as an empty string "", which is fine
# with the current script (since a "/" is added when needed)
# but still something to keep in mind.
# However, due to this behavior we delay stripping the last slash for
# the source folder until after parsing for ssh usage.
DEST_FOLDER="${DEST_FOLDER%/}"
fn_parse_ssh
if [ -n "$SSH_DEST_FOLDER" ]; then
DEST_FOLDER="$SSH_DEST_FOLDER"
fi
if [ -n "$SSH_SRC_FOLDER" ]; then
SRC_FOLDER="$SSH_SRC_FOLDER"
fi
# Now strip off last slash from source folder.
SRC_FOLDER="${SRC_FOLDER%/}"
for ARG in "$SRC_FOLDER" "$DEST_FOLDER" "$EXCLUSION_FILE"; do
if [[ "$ARG" == *"'"* ]]; then
fn_log_error 'Source and destination directories may not contain single quote characters.'
exit 1 exit 1
fi fi
done done
@@ -311,14 +76,19 @@ done
# TODO: check that the destination supports hard links # TODO: check that the destination supports hard links
fn_backup_marker_path() { echo "$1/backup.marker"; } fn_backup_marker_path() {
fn_find_backup_marker() { fn_find "$(fn_backup_marker_path "$1")" 2>/dev/null; } echo "$1/backup.marker"
}
if [ -z "$(fn_find_backup_marker "$DEST_FOLDER")" ]; then fn_is_backup_destination() {
find "$(fn_backup_marker_path "$1")" 2>/dev/null
}
if [ -z "$(fn_is_backup_destination $DEST_FOLDER)" ]; then
fn_log_info "Safety check failed - the destination does not appear to be a backup folder or drive (marker file not found)." fn_log_info "Safety check failed - the destination does not appear to be a backup folder or drive (marker file not found)."
fn_log_info "If it is indeed a backup folder, you may add the marker file by running the following command:" fn_log_info "If it is indeed a backup folder, you may add the marker file by running the following command:"
fn_log_info "" fn_log_info ""
fn_log_info_cmd "mkdir -p -- \"$DEST_FOLDER\" ; touch \"$(fn_backup_marker_path "$DEST_FOLDER")\"" fn_log_info "touch \"$(fn_backup_marker_path $DEST_FOLDER)\""
fn_log_info "" fn_log_info ""
exit 1 exit 1
fi fi
@@ -327,177 +97,203 @@ fi
# Setup additional variables # Setup additional variables
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Date logic
NOW=$(date +"%Y-%m-%d-%H%M%S")
EPOCH=$(date "+%s")
KEEP_ALL_DATE=$((EPOCH - 86400)) # 1 day ago
KEEP_DAILIES_DATE=$((EPOCH - 2678400)) # 31 days ago
export IFS=$'\n' # Better for handling spaces in filenames. export IFS=$'\n' # Better for handling spaces in filenames.
DEST="$DEST_FOLDER/$NOW" NOW=$(date +"%Y-%m-%d-%H%M%S")
PREVIOUS_DEST="$(fn_find_backups | head -n 1)" PROFILE_FOLDER="$HOME/.rsync_tmbackup"
INPROGRESS_FILE="$DEST_FOLDER/backup.inprogress" LOG_FILE="$PROFILE_FOLDER/$NOW.log"
MYPID="$$" DEST=$DEST_FOLDER/$NOW
PREVIOUS_DEST=$(fn_find_backups | sort | tail -n 1)
INPROGRESS_FILE=$DEST_FOLDER/backup.inprogress
case "$OSTYPE" in
linux*)
KEEP_ALL_DATE=$(date -d '-1 day' +%s)
KEEP_DAILIES_DATE=$(date -d '-1 month' +%s)
;;
darwin*)
KEEP_ALL_DATE=$(date -j -f "%a %b %d %T %Z %Y" "`date -v -1d`" "+%s")
KEEP_DAILIES_DATE=$(date -j -f "%a %b %d %T %Z %Y" "`date -v -1m`" "+%s")
;;
esac
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Create log folder if it doesn't exist # Create profile folder if it doesn't exist
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
if [ ! -d "$LOG_DIR" ]; then if [ ! -d "$PROFILE_FOLDER" ]; then
fn_log_info "Creating log folder in '$LOG_DIR'..." fn_log_info "Creating profile folder in '$PROFILE_FOLDER'..."
mkdir -- "$LOG_DIR" mkdir -- "$PROFILE_FOLDER"
fi fi
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Handle case where a previous backup failed or was interrupted. # Handle case where a previous backup failed or was interrupted.
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
if [ -n "$(fn_find "$INPROGRESS_FILE")" ]; then if [ -f "$INPROGRESS_FILE" ]; then
if [ "$OSTYPE" == "cygwin" ]; then if [ "$PREVIOUS_DEST" != "" ]; then
# 1. Grab the PID of previous run from the PID file
RUNNINGPID="$(fn_run_cmd "cat $INPROGRESS_FILE")"
# 2. Get the command for the process currently running under that PID and look for our script name
RUNNINGCMD="$(procps -wwfo cmd -p $RUNNINGPID --no-headers | grep "$APPNAME")"
# 3. Grab the exit code from grep (0=found, 1=not found)
GREPCODE=$?
# 4. if found, assume backup is still running
if [ "$GREPCODE" = 0 ]; then
fn_log_error "Previous backup task is still active - aborting (command: $RUNNINGCMD)."
exit 1
fi
else
RUNNINGPID="$(fn_run_cmd "cat $INPROGRESS_FILE")"
if [ "$RUNNINGPID" = "$(pgrep "$APPNAME")" ]; then
fn_log_error "Previous backup task is still active - aborting."
exit 1
fi
fi
if [ -n "$PREVIOUS_DEST" ]; then
# - Last backup is moved to current backup folder so that it can be resumed. # - Last backup is moved to current backup folder so that it can be resumed.
# - 2nd to last backup becomes last backup. # - 2nd to last backup becomes last backup.
fn_log_info "$SSH_DEST_FOLDER_PREFIX$INPROGRESS_FILE already exists - the previous backup failed or was interrupted. Backup will resume from there." fn_log_info "$INPROGRESS_FILE already exists - the previous backup failed or was interrupted. Backup will resume from there."
fn_run_cmd "mv -- $PREVIOUS_DEST $DEST" LINE_COUNT=$(fn_find_backups | sort | tail -n 2 | wc -l)
if [ "$(fn_find_backups | wc -l)" -gt 1 ]; then mv -- "$PREVIOUS_DEST" "$DEST"
PREVIOUS_DEST="$(fn_find_backups | sed -n '2p')" if [ "$LINE_COUNT" -gt 1 ]; then
PREVIOUS_PREVIOUS_DEST=$(fn_find_backups | sort | tail -n 2 | head -n 1)
PREVIOUS_DEST=$PREVIOUS_PREVIOUS_DEST
else else
PREVIOUS_DEST="" PREVIOUS_DEST=""
fi fi
# update PID to current process to avoid multiple concurrent resumes
fn_run_cmd "echo $MYPID > $INPROGRESS_FILE"
fi fi
fi fi
# Run in a loop to handle the "No space left on device" logic. # Run in a loop to handle the "No space left on device" logic.
while : ; do while [ "1" ]; do
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Check if we are doing an incremental backup (if previous backup exists). # Check if we are doing an incremental backup (if previous backup exists) or not
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
LINK_DEST_OPTION="" LINK_DEST_OPTION=""
if [ -z "$PREVIOUS_DEST" ]; then if [ "$PREVIOUS_DEST" == "" ]; then
fn_log_info "No previous backup - creating new one." fn_log_info "No previous backup - creating new one."
else else
# If the path is relative, it needs to be relative to the destination. To keep # If the path is relative, it needs to be relative to the destination. To keep
# it simple, just use an absolute path. See http://serverfault.com/a/210058/118679 # it simple, just use an absolute path. See http://serverfault.com/a/210058/118679
PREVIOUS_DEST="$(fn_get_absolute_path "$PREVIOUS_DEST")" PREVIOUS_DEST=`cd \`dirname -- "$PREVIOUS_DEST"\`; pwd`"/"`basename -- "$PREVIOUS_DEST"`
fn_log_info "Previous backup found - doing incremental backup from $SSH_DEST_FOLDER_PREFIX$PREVIOUS_DEST" fn_log_info "Previous backup found - doing incremental backup from $PREVIOUS_DEST"
LINK_DEST_OPTION="--link-dest='$PREVIOUS_DEST'" LINK_DEST_OPTION="--link-dest=$PREVIOUS_DEST"
fi fi
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Create destination folder if it doesn't already exists # Create destination folder if it doesn't already exists
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
if [ -z "$(fn_find "$DEST -type d" 2>/dev/null)" ]; then if [ ! -d "$DEST" ]; then
fn_log_info "Creating destination $SSH_DEST_FOLDER_PREFIX$DEST" fn_log_info "Creating destination $DEST"
fn_mkdir "$DEST" mkdir -p -- "$DEST"
fi fi
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Purge certain old backups before beginning new backup. # Purge certain old backups before beginning new backup.
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
fn_expire_backups # Default value for $prev ensures that the most recent backup is never deleted.
prev="0000-00-00-000000"
for fname in $(fn_find_backups | sort -r); do
date=$(basename "$fname")
stamp=$(fn_parse_date $date)
# Skip if failed to parse date...
# TODO: display warning
[ -n "$stamp" ] || continue
if [ $stamp -ge $KEEP_ALL_DATE ]; then
true
elif [ $stamp -ge $KEEP_DAILIES_DATE ]; then
# Delete all but the most recent of each day.
if [ ${date:0:7} == ${prev:0:7} ]; then
[ ${date:8:2} -eq ${prev:8:2} ] && fn_expire_backup "$fname"
fi
else
# Delete all but the most recent of each month.
if [ ${date:0:4} == ${prev:0:4} ]; then
[ ${date:5:2} -eq ${prev:5:2} ] && fn_expire_backup "$fname"
fi
fi
prev=$date
done
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Start backup # Start backup
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
LOG_FILE="$LOG_DIR/$(date +"%Y-%m-%d-%H%M%S").log" LOG_FILE="$PROFILE_FOLDER/$(date +"%Y-%m-%d-%H%M%S").log"
fn_log_info "Starting backup..." fn_log_info "Starting backup..."
fn_log_info "From: $SSH_SRC_FOLDER_PREFIX$SRC_FOLDER/" fn_log_info "From: $SRC_FOLDER"
fn_log_info "To: $SSH_DEST_FOLDER_PREFIX$DEST/" fn_log_info "To: $DEST"
CMD="rsync" CMD="rsync"
if [ -n "$SSH_CMD" ]; then CMD="$CMD --compress"
CMD="$CMD -e 'ssh -p $SSH_PORT -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'" CMD="$CMD --numeric-ids"
fi CMD="$CMD --links"
CMD="$CMD $RSYNC_FLAGS" CMD="$CMD --hard-links"
CMD="$CMD --delete"
CMD="$CMD --delete-excluded"
CMD="$CMD --one-file-system"
CMD="$CMD --archive"
CMD="$CMD --itemize-changes"
CMD="$CMD --verbose"
CMD="$CMD --log-file '$LOG_FILE'" CMD="$CMD --log-file '$LOG_FILE'"
if [ -n "$EXCLUSION_FILE" ]; then if [ "$EXCLUSION_FILE" != "" ]; then
# We've already checked that $EXCLUSION_FILE doesn't contain a single quote # We've already checked that $EXCLUSION_FILE doesn't contain a single quote
CMD="$CMD --exclude-from '$EXCLUSION_FILE'" CMD="$CMD --exclude-from '$EXCLUSION_FILE'"
fi fi
CMD="$CMD $LINK_DEST_OPTION" CMD="$CMD $LINK_DEST_OPTION"
CMD="$CMD -- '$SSH_SRC_FOLDER_PREFIX$SRC_FOLDER/' '$SSH_DEST_FOLDER_PREFIX$DEST/'" CMD="$CMD -- '$SRC_FOLDER/' '$DEST/'"
CMD="$CMD | grep -E '^deleting|[^/]$'"
fn_log_info "Running command:" fn_log_info "Running command:"
fn_log_info "$CMD" fn_log_info "$CMD"
fn_run_cmd "echo $MYPID > $INPROGRESS_FILE" touch -- "$INPROGRESS_FILE"
eval $CMD eval $CMD
RSYNC_EXIT_CODE=$?
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Check if we ran out of space # Check if we ran out of space
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
NO_SPACE_LEFT="$(grep "No space left on device (28)\|Result too large (34)" "$LOG_FILE")" # TODO: find better way to check for out of space condition without parsing log.
grep --quiet "No space left on device (28)" "$LOG_FILE"
NO_SPACE_LEFT="$?"
if [ "$NO_SPACE_LEFT" != "0" ]; then
# This error might also happen if there is no space left
grep --quiet "Result too large (34)" "$LOG_FILE"
NO_SPACE_LEFT="$?"
fi
rm -- "$LOG_FILE"
if [ "$NO_SPACE_LEFT" == "0" ]; then
# TODO: -y flag
read -p "It looks like there is no space left on the destination. Delete old backup? (Y/n) " yn
case $yn in
[Nn]* ) exit 0;;
esac
if [ -n "$NO_SPACE_LEFT" ]; then
fn_log_warn "No space left on device - removing oldest backup and resuming." fn_log_warn "No space left on device - removing oldest backup and resuming."
if [[ "$(fn_find_backups | wc -l)" -lt "2" ]]; then BACKUP_FOLDER_COUNT=$(fn_find_backups | wc -l)
if [ "$BACKUP_FOLDER_COUNT" -lt "2" ]; then
fn_log_error "No space left on device, and no old backup to delete." fn_log_error "No space left on device, and no old backup to delete."
exit 1 exit 1
fi fi
fn_expire_backup "$(fn_find_backups | tail -n 1)" OLD_BACKUP_PATH=$(fn_find_backups | head -n 1)
if [ "$OLD_BACKUP_PATH" == "" ]; then
fn_log_error "No space left on device, and cannot get path to oldest backup to delete."
exit 1
fi
fn_expire_backup "$OLD_BACKUP_PATH"
# Resume backup # Resume backup
continue continue
fi fi
# ----------------------------------------------------------------------------- if [ "$RSYNC_EXIT_CODE" != "0" ]; then
# Check whether rsync reported any errors fn_log_error "Exited with error code $RSYNC_EXIT_CODE"
# ----------------------------------------------------------------------------- exit $RSYNC_EXIT_CODE
EXIT_CODE="1"
if [ -n "$(grep "rsync error:" "$LOG_FILE")" ]; then
fn_log_error "Rsync reported an error. Run this command for more details: grep -E 'rsync:|rsync error:' '$LOG_FILE'"
elif [ -n "$(grep "rsync:" "$LOG_FILE")" ]; then
fn_log_warn "Rsync reported a warning. Run this command for more details: grep -E 'rsync:|rsync error:' '$LOG_FILE'"
else
fn_log_info "Backup completed without errors."
if [[ $AUTO_DELETE_LOG == "1" ]]; then
rm -f -- "$LOG_FILE"
fi
EXIT_CODE="0"
fi fi
# ----------------------------------------------------------------------------- rm -- "$INPROGRESS_FILE"
# Add symlink to last backup # TODO: grep for "^rsync error:.*$" in log
# ----------------------------------------------------------------------------- fn_log_info "Backup completed without errors."
exit 0
fn_rm_file "$DEST_FOLDER/latest"
fn_ln "$(basename -- "$DEST")" "$DEST_FOLDER/latest"
fn_rm_file "$INPROGRESS_FILE"
exit $EXIT_CODE
done done