The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

rsync takes ages ...

Discussion in 'General Discussion' started by WebHostDog, Oct 14, 2006.

  1. WebHostDog

    WebHostDog Well-Known Member

    Joined:
    Sep 3, 2006
    Messages:
    144
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Website Owner
    Hello for some days rsync takes ages (even do not do nothing) to move incremental backups to off-site server.


    root@server [~]# rsync -avze ssh /backups user@xxx.xx.xxx.xxx:folder
    user@xxx.xx.xxx.xxx's password:
    building file list ...



    and ... before all was very fast and worked.


    Any ideas ?
    Seems it is from remote (off-site) server cause to a different off-site server worked as before (fast)


    Any ideas what may be the reason ?


    Thanks.
     
  2. jsnape

    jsnape Well-Known Member

    Joined:
    Mar 11, 2002
    Messages:
    174
    Likes Received:
    0
    Trophy Points:
    16
    If the server is using maildir format, and there are a ton of mail files it will take ages. And it will take ages to transfer the files. It'd be nice if there was a way to automatically archive those into ONE file. I have one server that has hosted about 500 sites for 4 years. It had over 2 million spam emails sitting on it from users who don't take care of their mailboxes.
     
  3. WebHostDog

    WebHostDog Well-Known Member

    Joined:
    Sep 3, 2006
    Messages:
    144
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Website Owner
    No it works great for 4 servers, now for 3 does not work and for 4th works. Got to be some explanation :)
     
  4. rikgarner

    rikgarner Well-Known Member

    Joined:
    Mar 31, 2006
    Messages:
    75
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    /dev/null
    Have you got any way to see whether it is fully utilising its bandwidth - ie, is the server sat there using 5% of its bandwith to the other server, whereas the server which works is maxing its connection?

    Im just thinking about what slows a tape backup down - things like locked files, databases etc...

    Maybe.. Its a stab in the dark!

    Rich
     
  5. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Personally, I'd try it without compression which has never been very reliable with rsync over the network, i.e. don't use z in -avze
     
  6. WebHostDog

    WebHostDog Well-Known Member

    Joined:
    Sep 3, 2006
    Messages:
    144
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Website Owner
    I have found the issue was the slow network (was a lot of collisions).



    Thanks.
     
  7. jamesbond

    jamesbond Well-Known Member

    Joined:
    Oct 9, 2002
    Messages:
    738
    Likes Received:
    1
    Trophy Points:
    18
    Unreliable as in resulting in non-functional backups?

    I haven't noticed any issues yet with -z, but if rsyncing without compression increases the reliability I guess I'll try that.
     
  8. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    In the past I've found that the rsync with compression can often fail to completely transfer the files. It also can consume considerable CPU. If you're copying full backups it is probably a complete waste of resources anyway since the tarballs are already compressed (with gzip).
     
  9. twhiting9275

    twhiting9275 Well-Known Member

    Joined:
    Sep 26, 2002
    Messages:
    538
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    For the best usage of system resources, I have found (and recommend) that this works quite well:

    Back the server up onto an additional hard drive with NO compression (zip, etc). This will do two things to save your server CPU time

    Firstly, it won't compress everything (yay)
    Secondly, it won't copy everything, just the MODIFIED files. Again, this saves your server CPU time/load

    Once that's done, rsync it across to the backup server. Again, this saves the compression as stated above, and is just doing a compare, tossing only what's CHANGED over there.

    This provides the ultimate in backups, as you have OFF site backups, and ON site backups as well.

    As far as the speed, that's going to depend on how well network A talks to network B. It's always best when using rsync, if you CAN , with this much data to use something like vpn or private networking which will usually help out a great bit.
     
  10. Valetia

    Valetia Well-Known Member

    Joined:
    Jun 20, 2002
    Messages:
    207
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    I've been wanting to do this, but how do you set the the off-site backup to run after the on site backup finishes (so that they don't run at the same time)?
     
  11. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    You can now create a script called /scripts/postcpbackup which will run after the cpbackup has completed and add your shell commands into it, e.g.:

    #cat /scripts/postcpbackup

    #!/bin/sh
    rsync --delete --stats -vae ssh /backup/cpbackup/daily remote@remote.server.com/backup/server
     
Loading...

Share This Page