rsync takes ages ...

[WebHostDog]

Hello for some days rsync takes ages (even do not do nothing) to move incremental backups to off-site server.


root@server [~]# rsync -avze ssh /backups user@xxx.xx.xxx.xxx:folder
user@xxx.xx.xxx.xxx's password:
building file list ...



and ... before all was very fast and worked.


Any ideas ?
Seems it is from remote (off-site) server cause to a different off-site server worked as before (fast)


Any ideas what may be the reason ?


Thanks.

[jsnape]

If the server is using maildir format, and there are a ton of mail files it will take ages. And it will take ages to transfer the files. It'd be nice if there was a way to automatically archive those into ONE file. I have one server that has hosted about 500 sites for 4 years. It had over 2 million spam emails sitting on it from users who don't take care of their mailboxes.

[WebHostDog]

No it works great for 4 servers, now for 3 does not work and for 4th works. Got to be some explanation

[rikgarner]

Have you got any way to see whether it is fully utilising its bandwidth - ie, is the server sat there using 5% of its bandwith to the other server, whereas the server which works is maxing its connection?

Im just thinking about what slows a tape backup down - things like locked files, databases etc...

Maybe.. Its a stab in the dark!

Rich

[chirpy]

Personally, I'd try it without compression which has never been very reliable with rsync over the network, i.e. don't use z in -avze

[WebHostDog]

I have found the issue was the slow network (was a lot of collisions).



Thanks.

[jamesbond]

Personally, I'd try it without compression which has never been very reliable with rsync over the network, i.e. don't use z in -avze

Unreliable as in resulting in non-functional backups?

I haven't noticed any issues yet with -z, but if rsyncing without compression increases the reliability I guess I'll try that.

[chirpy]

Unreliable as in resulting in non-functional backups?

I haven't noticed any issues yet with -z, but if rsyncing without compression increases the reliability I guess I'll try that.

In the past I've found that the rsync with compression can often fail to completely transfer the files. It also can consume considerable CPU. If you're copying full backups it is probably a complete waste of resources anyway since the tarballs are already compressed (with gzip).

[twhiting9275]

For the best usage of system resources, I have found (and recommend) that this works quite well:

Back the server up onto an additional hard drive with NO compression (zip, etc). This will do two things to save your server CPU time

Firstly, it won't compress everything (yay)
Secondly, it won't copy everything, just the MODIFIED files. Again, this saves your server CPU time/load

Once that's done, rsync it across to the backup server. Again, this saves the compression as stated above, and is just doing a compare, tossing only what's CHANGED over there.

This provides the ultimate in backups, as you have OFF site backups, and ON site backups as well.

As far as the speed, that's going to depend on how well network A talks to network B. It's always best when using rsync, if you CAN , with this much data to use something like vpn or private networking which will usually help out a great bit.

[Valetia]

For the best usage of system resources, I have found (and recommend) that this works quite well:

Back the server up onto an additional hard drive with NO compression (zip, etc). This will do two things to save your server CPU time

Firstly, it won't compress everything (yay)
Secondly, it won't copy everything, just the MODIFIED files. Again, this saves your server CPU time/load

Once that's done, rsync it across to the backup server. Again, this saves the compression as stated above, and is just doing a compare, tossing only what's CHANGED over there.

This provides the ultimate in backups, as you have OFF site backups, and ON site backups as well.

As far as the speed, that's going to depend on how well network A talks to network B. It's always best when using rsync, if you CAN , with this much data to use something like vpn or private networking which will usually help out a great bit.

I've been wanting to do this, but how do you set the the off-site backup to run after the on site backup finishes (so that they don't run at the same time)?

[chirpy]

You can now create a script called /scripts/postcpbackup which will run after the cpbackup has completed and add your shell commands into it, e.g.:

#cat /scripts/postcpbackup

#!/bin/sh
rsync --delete --stats -vae ssh /backup/cpbackup/daily remote@remote.server.com/backup/server