WebHostDog

Well-Known Member
Sep 3, 2006
144
1
166
cPanel Access Level
Website Owner
Hello for some days rsync takes ages (even do not do nothing) to move incremental backups to off-site server.


[email protected] [~]# rsync -avze ssh /backups [email protected]:folder
[email protected]'s password:
building file list ...



and ... before all was very fast and worked.


Any ideas ?
Seems it is from remote (off-site) server cause to a different off-site server worked as before (fast)


Any ideas what may be the reason ?


Thanks.
 

jsnape

Well-Known Member
Mar 11, 2002
174
0
316
If the server is using maildir format, and there are a ton of mail files it will take ages. And it will take ages to transfer the files. It'd be nice if there was a way to automatically archive those into ONE file. I have one server that has hosted about 500 sites for 4 years. It had over 2 million spam emails sitting on it from users who don't take care of their mailboxes.
 

WebHostDog

Well-Known Member
Sep 3, 2006
144
1
166
cPanel Access Level
Website Owner
No it works great for 4 servers, now for 3 does not work and for 4th works. Got to be some explanation :)
 

rikgarner

Well-Known Member
Mar 31, 2006
75
1
158
/dev/null
Have you got any way to see whether it is fully utilising its bandwidth - ie, is the server sat there using 5% of its bandwith to the other server, whereas the server which works is maxing its connection?

Im just thinking about what slows a tape backup down - things like locked files, databases etc...

Maybe.. Its a stab in the dark!

Rich
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,465
30
473
Go on, have a guess
Personally, I'd try it without compression which has never been very reliable with rsync over the network, i.e. don't use z in -avze
 

WebHostDog

Well-Known Member
Sep 3, 2006
144
1
166
cPanel Access Level
Website Owner
I have found the issue was the slow network (was a lot of collisions).



Thanks.
 

jamesbond

Well-Known Member
Oct 9, 2002
738
1
168
chirpy said:
Personally, I'd try it without compression which has never been very reliable with rsync over the network, i.e. don't use z in -avze
Unreliable as in resulting in non-functional backups?

I haven't noticed any issues yet with -z, but if rsyncing without compression increases the reliability I guess I'll try that.
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,465
30
473
Go on, have a guess
jamesbond said:
Unreliable as in resulting in non-functional backups?

I haven't noticed any issues yet with -z, but if rsyncing without compression increases the reliability I guess I'll try that.
In the past I've found that the rsync with compression can often fail to completely transfer the files. It also can consume considerable CPU. If you're copying full backups it is probably a complete waste of resources anyway since the tarballs are already compressed (with gzip).
 

twhiting9275

Well-Known Member
Sep 26, 2002
560
27
178
cPanel Access Level
Root Administrator
Twitter
For the best usage of system resources, I have found (and recommend) that this works quite well:

Back the server up onto an additional hard drive with NO compression (zip, etc). This will do two things to save your server CPU time

Firstly, it won't compress everything (yay)
Secondly, it won't copy everything, just the MODIFIED files. Again, this saves your server CPU time/load

Once that's done, rsync it across to the backup server. Again, this saves the compression as stated above, and is just doing a compare, tossing only what's CHANGED over there.

This provides the ultimate in backups, as you have OFF site backups, and ON site backups as well.

As far as the speed, that's going to depend on how well network A talks to network B. It's always best when using rsync, if you CAN , with this much data to use something like vpn or private networking which will usually help out a great bit.
 

Valetia

Well-Known Member
Jun 20, 2002
216
10
168
cPanel Access Level
Root Administrator
twhiting9275 said:
For the best usage of system resources, I have found (and recommend) that this works quite well:

Back the server up onto an additional hard drive with NO compression (zip, etc). This will do two things to save your server CPU time

Firstly, it won't compress everything (yay)
Secondly, it won't copy everything, just the MODIFIED files. Again, this saves your server CPU time/load

Once that's done, rsync it across to the backup server. Again, this saves the compression as stated above, and is just doing a compare, tossing only what's CHANGED over there.

This provides the ultimate in backups, as you have OFF site backups, and ON site backups as well.

As far as the speed, that's going to depend on how well network A talks to network B. It's always best when using rsync, if you CAN , with this much data to use something like vpn or private networking which will usually help out a great bit.
I've been wanting to do this, but how do you set the the off-site backup to run after the on site backup finishes (so that they don't run at the same time)?
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,465
30
473
Go on, have a guess
You can now create a script called /scripts/postcpbackup which will run after the cpbackup has completed and add your shell commands into it, e.g.:

#cat /scripts/postcpbackup

#!/bin/sh
rsync --delete --stats -vae ssh /backup/cpbackup/daily [email protected]/backup/server