The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

rsync - reduce server load on receiving end

Discussion in 'Workarounds and Optimization' started by WhiteDog, Aug 5, 2012.

  1. WhiteDog

    WhiteDog Well-Known Member

    Joined:
    Feb 19, 2008
    Messages:
    118
    Likes Received:
    0
    Trophy Points:
    16
    I am currently using rsync to backup my cPanel Full Backups (compressed, not incremental) to another sever.

    I'm using this command-line on the source server (push method):
    Code:
    nice -n19 ionice -c2 -n7 rsync -tr -e "ssh -p 1234" /backup/cpbackup/daily rsbackup@server.domain.com:/rsbackup/SERVERNAME
    This in itself works fine and the server load on the source server stays below 4.00.

    However on the receiving end, the server load goes up to 15.00 from time to time. Since i'm making backups of backups, time is of no concern. Is there any way to instruct rsync to also take the server load on the receiving end into consideration and slow things down if needed?

    Things I came up with myself:
    - Use --bwlimit XXXXX (can anyone suggest a good value for a 100 Mbit connection?).
    - Create a script / cron job that runs every 30 minutes and only backs up a maximum of X files each run.
    - Start rsync, SSH over to the other end and send instructions to lower the priority of rsync.

    Thanks!
     
  2. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
  3. WhiteDog

    WhiteDog Well-Known Member

    Joined:
    Feb 19, 2008
    Messages:
    118
    Likes Received:
    0
    Trophy Points:
    16
    Tristan, as always, thanks for pointing out "something new" to me. The quality of your answers around these forums is always top notch :)

    I just don't completely understand how lsync could contain the server load on the "receiving end", as it seems to use rsync beneath thus I expect the behavior to be the same.

    For the record: the rsync command I posted above is executed after cpbackup completes using /scripts/postcpbackup
     
  4. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    It does use rsync, but it doesn't spawn any processes to the receiving server unless there are file changes and, because it changes the file to the receiving server when the file is changed, it has less files at once being sent to the receiving server. Compare this. If you do an rsync every 3 days, that rsync has more files to push. The receiving server receives a list of all files and then will get those files pushed one by one. The magnitude of the file numbers is more likely to cause a load on the receiving server for that reason. If you run an rsync every day, it would have less files it is pushing and cause less time to rsync over due to the reduced number of files.

    With lsync, you are minimizing the amount of data at one time being pushed at a time. There will be more pushes but those will have less files each time, reducing the likelihood of having a high load when it occurs.
     
Loading...

Share This Page