I have to transfer large amounts of data (>80 GB) over ssh using rsync. Everything is working fine, but the DSL connection where the backup data is sent from will drop once every 24h for up to 3 minutes (switching providers is not an option).
How do I:
Automatically restart the transfer when the connection is back up?
Make sure there are not by accident two rsync commands running at the same time?
5 Answers
The following should be helpful:
#!/bin/bash
while [ 1 ]
do rsync -avz --timeout=60 --partial source dest if [ "$?" = "0" ] ; then echo "rsync completed normally" exit else echo "Rsync failure. Backing off and retrying..." sleep 180 fi
doneWhen the connection dies, rsync will quit with a non-zero exit code. This script simply keeps re-running rsync, letting it continue until the synchronisation completes normally.
7This does much the same as Peter's answer, but gives the user the option of which remote file he wants, and where he wants to save it (as well as conducting the rsync over ssh). Replace the USER and HOST with your username and host respectively.
#!/bin/bash echo -e "Please enter the full (escaped) file path:" read -r path echo "Path: $path" echo -e "Enter the destination:" read -r dst echo "Destination: $dst" while [ 1 ] do rsync --progress --partial --append -vz -e ssh "USER@HOST:$path" $dst if [ "$?" = "0" ] ; then echo "rsync completed normally" exit else echo "rsync failure. Retrying in a minute..." sleep 60 fi done
The rsync options used here enable the progress stats during transfer, the saving of partial files upon unexpected failure, and the ability to append to partially completed files upon resume. The -v option increases verbosity, the -z option enables compression (good for a slow connection, but requires more cpu power on both ends), and the -e option enables us to conduct this transfer over ssh (encryption is always good).
Note: Use this only if you have public key login enabled with your ssh, otherwise it will ask you for a password when it restarts (killing all functionality of the script).
1supervisor daemon (a process control manager) could work very well after creating rsa certificates of both sides, with a similar configuration as follows: (/etc/supervisor/ supervisord.conf is the configuration file path on debian based systems )
[program:rsync-remoteserver]
command=rsync -avz --progress :/destination /backup-path
stdout_logfile=/out-log-path
stderr_logfile=/errlogpathA quick and dirty way to do this when you don't feel like opening up an editor:
while ( ! rsync -avzP <source> <dest> ); do sleep 1; done-a archive mode
-v increase verbosity
-z compress file data during the transfer
-P same as --partial --progressThe @Peter's answer seems be very useful, but for me it was important to use --update option. After connection has been resumed, without --update rsync was trying to sync all from the very begining. With --update, files that already exists are skipped.
rsync --partial --update --progress -r [SOURCE] [DESTINATION]
