HOW TO: make full backups use --rsyncable w/ gzip

[ispro]

Would like to share with the community the "issue" I have faced with trying to understand why the rsyncable not work... It was a lame, but anyway. If you use rsync with the option -W or --whole-file rsync will not use rsync algorythm on any files incl. gzip and will transfer whole file.

Just a note

 

[Blue Steel]

pros/cons inc full, inc, or full .. which will be best for me

Edits: The simplified version after much testing and trial is provided below. Thanks to Chirpy for finding the exact method that works so simply!

Modify the root crontab changing the line shown as follows:

Quote:
0 1 * * * /scripts/cpbackup

to:

0 1 * * * export GZIP="--rsyncable" ; /scripts/cpbackup

That's it!

ok so my system is currently set up to do full baackups .. and its taking about 3-4 hours
(some of the sites are over 15gb in size)

What i want to know is the answer to these questions

1. is that ALL that i have to change to get it to do FULL baxkups increments ..
2. will it still be able tro to a full system / site / account restore
3. will clients still be able to do backups through cpanel
4. how much cpu power/time will it save if i change to this method
5. will it change what needs to be kept .. or how will it effect the weekly/monthly/yearle backups
6. If i just tell whm to do incrimental backups .. what differences to the above questions

I really need to work out a bett backup system because the full backups put way to much load on the server for too long.. I'm backing up the accountsa to a second hd.. mounted as /backup

I've searched and read through over 18 pages of finds for backing up.. and can't find a clear answer to my delemour.. I know it would be a lot better to have a second system doing all the backups and just reading the data off the main system.. but thats pout of the question at the moment..

I need something where users can still do full system backups for transfering to new servers etc.. and so that if heaven forbid we loose our backups to so that they can easily recover .. and so that we can easily recover / import including from, other servers accounts and setups etc..

Sorry.. too many questions that i can't find answers to.. and I don't wqant to change from full backups IF its going to change the manual backups on cpanel for our endusers. or if its not going to change greatly the CPU usagew and the time it takes

 

[ispro]

The single answer is that:
This change useful when you store backups on remote (NAS, etc.) servers/filesystems.
You will save traffic and the time for the transfer.

Nothing more will be changed! Nothing.
E.g. neither the time nor CPU load nor ability to backup/restore.

This option self-explaining - it is for Rsync.

P.S. And yes, cPanel backups are sucks, there should be other possibilities to grab the content which doesn't push too much load.

 

[Blue Steel]

and how about the standard whm standard incremental backups.. is there any downside to them.. eg: will users still be able to do backups through cpanel.. ?? and would i still be able to recover the backups using whm.. also will it allow for rtransfering of accounts etc to and from our server to /from another server running cpanel..

you only answered half of the questions i asked.. and fair enough as thats what this topic was about.. just looking for a way to do all of this without compromising any accessability.. i'm not really worried about HD space used at the moment.. (but that could become an issue down the track i suppose)

Just that our webhosting grinds to a crawl during backups.. but the customers expect backups to be kept for them and if we don't then they'll go elsewhere

This is why i'm looking for a better way.. at the moment we only have the one server

irrespective that the other servers they go to may be worse than ours

 

[Lyttek]

What i want to know is the answer to these questions

1. is that ALL that i have to change to get it to do FULL baxkups increments ..
2. will it still be able tro to a full system / site / account restore
3. will clients still be able to do backups through cpanel
4. how much cpu power/time will it save if i change to this method
5. will it change what needs to be kept .. or how will it effect the weekly/monthly/yearle backups
6. If i just tell whm to do incrimental backups .. what differences to the above questions

1. it doesn't change the manner of the backup, only the compression algorithm used in the backup method you specify.
2. no change
3. no change
4. no change
5. no change
6. no change

This option, of using the -rsyncable switch on gzip, simply makes the backup file work well with rsync for transfering that backup file offsite. It doesn't affect anything regarding full or incremental, daily or weekly or monthly... nothing.

As for myself, I never, ever use incremental backups. Every increment introduces another point of failure, as if there is a problem in any backup, the entire system fails. I keep multiple full backups.

If the backup process is dragging your server down to the point of unusability, then you need to either beef up the server, or move some client off it to another.

 

[Scotty_B]

Glad I found this, am currently using the built in incremental option and rsyncing the contents of that to a bqbackup account. The problem with that though is ownership permissions are lost so when restoring a bit of work would have to be put in to get everything back up.

Going to give this a try tonight, although with a 5GB account it might cause some problems with gzipping it?

 

[Drew Nichols]

I've been using it and it works perfectly. I've even done some restores to test. What I want to do is write a script to automate rotation on the target server.

So that the system rsyncs to the target and then the target makes a snapshot and copies it elsewhere on its disk so that the next rsync gives you the new updates, but you still have the old updates.

We have a PC at the office with two 300 GB drives that were $70 each (IDE). In an emergency, we'd simply take this PC to the datacenter and do the restore. We have 6 Mbps DSL so we can take ~200 GB of data every night, since we only move the deltas.

 

[lloyd_tennison]

Look at rvm and rdiff-backup, for better usage of backup.

 

[Lyttek]

Here's the script I use to rotate through 7 days:

#!/bin/sh

# This script rotates the backup files daily, keeping 7 days worth.

DATEFORMAT=$(date +%a)
BACKUPSOURCE=/backup
BACKUPDEST=/Data/backup/rotation/

rm -Rf $BACKUPDEST/$DATEFORMAT
mkdir $BACKUPDEST/$DATEFORMAT
cp -R $BACKUPSOURCE/* $BACKUPDEST/$DATEFORMAT

 

[RandyO]

I install backup drives on my boxes. A couple of them are getting to the point were backups are taking 12 hours. Is there a way maintain the day/week/monthly backups using the rsync files and only replacing those that have the archive bit set?. I would bet that 75% of the accounts are pretty much static day to day and seen lots of room from improvement.

 

[Drew Nichols]

I install backup drives on my boxes. A couple of them are getting to the point were backups are taking 12 hours. Is there a way maintain the day/week/monthly backups using the rsync files and only replacing those that have the archive bit set?. I would bet that 75% of the accounts are pretty much static day to day and seen lots of room from improvement.

You should look at rdiff-backup. It will allow you to keep incrementals.

 

[RandyO]

You should look at rdiff-backup. It will allow you to keep incrementals.

Defeats the entire purpose. The incremental backups are uncompressed and are a single set of files. (no weekly, monthly sets). As they are uncompressed, they are a terrible waste of space as well. They are efficient after the first run but I do need to keep more than 1 set of files

 

[ispro]

Defeats the entire purpose. The incremental backups are uncompressed and are a single set of files. (no weekly, monthly sets). As they are uncompressed, they are a terrible waste of space as well. They are efficient after the first run but I do need to keep more than 1 set of files

If you could afford to have ONE uncompressed SET for each account you may get almost 'free' changes sets with hard-links. The simpliest is copy whole structure via 'cp -l', rotate directories to have 'original' to be, say, 'backup01' and copy as, say, 'backup' and then rsync to 'backup' to get new snapshot.

However it is usefull for user's dir copy, not for SQL databses, etc.

 

[jeremy_reliable]

Here's an option I just used to copy an account from a server running out of space to a new cpanel server. Figured I'd post it here since an extensive search turned up nothing on the issue packaging from a full server.

1. Share drive on dest server using nfs
2. Mount drive nfs on source server
3. run /scripts/pkgacct username /mount_path/

This runs pkgacct without using any space on the source server..

you need the knowledge to setup the NFS mount, but that info is readily available.. good luck

 

[ispro]

Doing backups on NFS will cause bad performance and unneeded traffic waste.

It has MORE sense either to do FEW backups in this way or just move files to NFS share and then perform backups on server itself, then copy them to NFS.

We have done this for some times and it works better than direct NFS backups.

 

[jeremy_reliable]

Doing backups on NFS will cause bad performance and unneeded traffic waste.

It has MORE sense either to do FEW backups in this way or just move files to NFS share and then perform backups on server itself, then copy them to NFS.

We have done this for some times and it works better than direct NFS backups.

Yeah, this may be the wrong thread.. this was not intended for scheduled backups.. the problem was that the drive on server was totally full, so without some help /scripts/pkgacct failes due to lack of space.

This is a one off solution in my case..

 

[camay123]

ok, I have read whole thread, at the risk of repeating, I just want to make sure of something :

0 1 * * * /scripts/cpbackup

to:

0 1 * * * export GZIP="--rsyncable" ; /scripts/cpbackup

This is the only thing I need to do in order the get it working ?

So this will create tar.gz files that are rsync compatible ?

 

[Lyttek]

That's correct.

Simple, ain't it?

Now, what you do with the backups once they're 'rsync-friendly' is up to you, but that simple change makes them play nice with rsync.

 

[AngelofDeth]

Does anyone know where I can get a gzip with this so called option? The standard gzip doesn't come with it... I found a .diff, but it doesn't seem to apply to the latest version properly.

 

[Rooter]

Does anyone know where I can get a gzip with this so called option? The standard gzip doesn't come with it... I found a .diff, but it doesn't seem to apply to the latest version properly.

How old is the gzip release you're using; what specific version do you have?

I confirm that on at least gzip v1.3.3, dated 2002-03-08, the option "--rsyncable" to "make rsync-friendly archives" is included.