rsync - reduce server load on receiving end

[WhiteDog]

I am currently using rsync to backup my cPanel Full Backups (compressed, not incremental) to another sever.

I'm using this command-line on the source server (push method):

nice -n19 ionice -c2 -n7 rsync -tr -e "ssh -p 1234" /backup/cpbackup/daily [email protected]:/rsbackup/SERVERNAME

This in itself works fine and the server load on the source server stays below 4.00.

However on the receiving end, the server load goes up to 15.00 from time to time. Since i'm making backups of backups, time is of no concern. Is there any way to instruct rsync to also take the server load on the receiving end into consideration and slow things down if needed?

Things I came up with myself:
- Use --bwlimit XXXXX (can anyone suggest a good value for a 100 Mbit connection?).
- Create a script / cron job that runs every 30 minutes and only backs up a maximum of X files each run.
- Start rsync, SSH over to the other end and send instructions to lower the priority of rsync.

Thanks!

 

[cPanelTristan]

How about trying lsyncd instead? It only pushes files when they change:

lsyncd - Lsyncd (Live Syncing Daemon) synchronizes local directories with a remote targets - Google Project Hosting

It is supposed to be far less taxing on the system over rsync.

 

[WhiteDog]

How about trying lsyncd instead? It only pushes files when they change:

lsyncd - Lsyncd (Live Syncing Daemon) synchronizes local directories with a remote targets - Google Project Hosting

It is supposed to be far less taxing on the system over rsync.

Tristan, as always, thanks for pointing out "something new" to me. The quality of your answers around these forums is always top notch

I just don't completely understand how lsync could contain the server load on the "receiving end", as it seems to use rsync beneath thus I expect the behavior to be the same.

For the record: the rsync command I posted above is executed after cpbackup completes using /scripts/postcpbackup

 

[cPanelTristan]

It does use rsync, but it doesn't spawn any processes to the receiving server unless there are file changes and, because it changes the file to the receiving server when the file is changed, it has less files at once being sent to the receiving server. Compare this. If you do an rsync every 3 days, that rsync has more files to push. The receiving server receives a list of all files and then will get those files pushed one by one. The magnitude of the file numbers is more likely to cause a load on the receiving server for that reason. If you run an rsync every day, it would have less files it is pushing and cause less time to rsync over due to the reduced number of files.

With lsync, you are minimizing the amount of data at one time being pushed at a time. There will be more pushes but those will have less files each time, reducing the likelihood of having a high load when it occurs.