Transfer Incremental Backups to Remote FTP

jasons

Registered
Mar 24, 2004
3
0
151
I perform incremental backups of my server on a secondary hard drive, as WHM will not allow incremental backups to a remote FTP. For added security, I recently purchased a remote FTP to store copies of the backups. My problem is, how am I to go about transfering the files stored on my server to the remote FTP (I am running a Linux based system)? I know that programs exist to transfer files, but I cannot find one to transfer directory structures as well. The size of the backups is about 50-60GB. Additionally, is there a way to make sure that as the transfer is taking place, it is not overloading the server?
 

brianoz

Well-Known Member
Mar 13, 2004
1,146
7
168
Melbourne, Australia
cPanel Access Level
Root Administrator
Moving a 50-60Gb backup over the network isn't a fantastic idea, it will take ages, unless you're local to the FTP box you've bought.

The best way to do it is probably to tar the directory hierarchy up into a compressed tar image. (Scp will take forever on a backup of that size.) Once you have the backup, then you scan scp or ftp it to your remote box in one hit. This isn't exactly a trivial exercise as far as writing shell or PHP code is concerned, so good luck! :)

(The good news is that compression will probably make the backup a lot smaller, maybe even 80% smaller if it is mainly web pages and databases and logs).
 

dgbaker

Well-Known Member
PartnerNOC
Sep 20, 2002
2,531
9
343
Toronto, Ontario Canada
cPanel Access Level
DataCenter Provider
Try this for using rsync (much better than SFTP or SCP)

Code:
#/bin/bash

# Backup to backup server 

DATE=`/bin/date +%Y.%m.%d::%H:%M`
HOST=`hostname`
BCKUSER="REMOTE USER ID"
BCKSERV="REMOTE SERVER NAME"
SSH="/usr/bin/ssh -l $BCKUSER $BCKSERV "
list="/backup/cpbackup/daily"

rsync -av --delete -e ssh $list [email protected]$BCKSERV:/backup/$HOST
Best is to also setup a public/private key between the two so that you do not have to deal with supplying or hardcoding a password.

But I must agree, unless the ftp server is local to your server, it will take a painstakingly long time and will be using your bandwidth. Keep also in mind that by using rsync, future copy overs will only copy what has changed, so it will/can be less time.
 
Last edited:

brianoz

Well-Known Member
Mar 13, 2004
1,146
7
168
Melbourne, Australia
cPanel Access Level
Root Administrator
Actually, I think the best rsync tool is rsnapshot - http://www.rsnapshot.org - check it out, it keeps several days worth of backups online using minimal disk space. Very nice stuff :)

But definitely, I should have mentioned rsync too, it's advantage is that it will only send the files that have changed each time you do a backup.