Make Backups go faster?

[jamina1]

We have four domains on our server, two with little to no actual data and only email, and the other two are very large, heavy with data and email.
Needless to say, the backups take about 4-6 hours to complete.

The server is running cPanel 11.24.4-R32603 - WHM 11.24.2 - X 3.9
REDHAT Enterprise 4 i686 on standard.

Normally, 4-6 hours wouldn't be a problem. We were running our backups at 1am and they were not causing us problems, but we've just opened an office 14 hours off of our main office (in hong kong), so in order to run the backups without killing the new offices' connection we moved our backups to 3am.
Only problem now is that when WE get into the office at 8, the backup is still running -especially if there's a weekly or monthly being compiled. (server load averages about 3 during this time, and slows everything right down).
The backup itself is usually done by this point, its gzip eating up all the processing power.

Is there any way to make the backup run faster or, if anything run a less demanding process?

The exact complaint I'm hearing from my boss every morning: "Can't we get in the logs and see whats going on? They say its *every day* at 3am the server just locks up".

[sparek-3]

Have you considered using incremental backups?

Chances are its the gzip process that is killing the server. You might want to look at the enhancement request at:

http://bugzilla.cpanel.net/show_bug.cgi?id=4090

which is a request for cPanel to support non-gzipped backups, just using tar for the backups. You can do this with a customized backup solution, but cPanel's backup system does not natively support it as yet.

[jamina1]

Originally Posted by sparek-3

Have you considered using incremental backups?

Chances are its the gzip process that is killing the server. You might want to look at the enhancement request at:

http://bugzilla.cpanel.net/show_bug.cgi?id=4090

which is a request for cPanel to support non-gzipped backups, just using tar for the backups. You can do this with a customized backup solution, but cPanel's backup system does not natively support it as yet.

Yes, it is gzip that is bogging the server down (at least in the ending hours when we need to be using it).

My only hesitation is that it says its incompatible with FTP backups, and also how would one go about restoring such a backup if its only certain files?

The way our backup is currently setup, the backup runs, offloads it onto our backup server and then a cronjob over there restores it so we have a semi-current backup web server ready to go in case of catastrophic circumstances.

[sparek-3]

You could use rsync to copy the incremental backup between servers. This would also save you some bandwidth because only the changed parts would be transferred.

However for restoring an incremental backup, I'm not sure if there is a way to automate this. See the thread at:

How to restore incremental backup

The only way that I know of to restore an incremental backup is through the WHM interface, or like suggested in this thread, tar up the incremental backup and use the restorepkg script to recreate the account.

You may want to look into a customized backup solution. Instead of using's cPanel backup script, use cPanel pkgacct script to create a backup package of the accounts, but don't compress it. Use something like:

/scripts/pkgacct <user> <destination> backup nocompress

The third parameter (backup) tells the script that you want to create a backup of the account. The fourth parameter (nocompress) tells pkgacct to not compress or gzip the backup. This results in a user.tar file instead of a user.tar.gz file. The user.tar file can be restored just like a user.tar.gz file.

[jamina1]

I think for the moment, we're gonna try the cPanel incremental backups. Though I'm still gonna need to figure out a way to rsync it over to our backup server as it says DOES NOT SUPPORT FTP with incremental. >_<

We checked our logs, and we discovered that it was indeed the weekly backup eating up the resources for us this morning. The main daily backup finished just after 8am, but the weekly continued to go for an additional 4 hours (mostly because we were trying to use the server at the same time).

Is there any way to tell it to just use the backup it just made as the weekly or monthly instead of compiling a whole new one?

[jamina1]

Originally Posted by sparek-3

You may want to look into a customized backup solution. Instead of using's cPanel backup script, use cPanel pkgacct script to create a backup package of the accounts, but don't compress it. Use something like:

/scripts/pkgacct <user> <destination> backup nocompress

Is there a way to automate that for each account so that going forward I won't have to write an account specific script?

Like var = find all accounts
for each var ( /scripts/pkgacct var user) blah

You get it I hope.

[sparek-3]

Bash is not a scripting language that I am very good at.

#!/bin/bash
for i in $( ls /var/cpanel/users ); do
	/scripts/pkgacct $i /backup/cpbackup/daily backup nocompress
done

I think that will work.

Change /backup/cpbackup/daily to whatever directory you want the backups stored in.

One thing to note, this won't backup system files and directories that the cPanel backup script normally backs up (i.e. the files in /backup/cpbackup/daily/files and /backup/cpbackup/daily/dirs). This may or may not be a concern for you.

[jamina1]

No, that is fine I think.. given that I can at least tar up that data and use it on a remote server to restore.

Ugh. Now my boss says "No more cpanel backups period! Its taking too long and we can't work while its going. I don't want it running on our server again!"

I think he'll fire me the next time it happens.

So I have to figure out a way to 1) backup our databases and restore them on the backup server each day (which I'm already doing with much success) and 2) backup our emails and restore them on the backup server each day

and apparently he doesn't care about the website itself, which has me jumping out of my pants in like wtf-ness.

[Infopro]

You'll forgive me for mentioning the obvious, but why not try to slim down the account a bit? And/Or, exclude files that don't change often using the cpbackup-exclude.conf file in the root of your account?

[jamina1]

That exclude file will help quite a bit thanks!

We run a retail store, and unfortunately due to the bulk number of products we carry (almost 30,000 at this point) that means quite a bit of images and a large database. Most products also come with pdf files. Being able to exclude those files will go a long way in making our backups go faster - thank you very much!

[jamina1]

Originally Posted by Infopro

You'll forgive me for mentioning the obvious, but why not try to slim down the account a bit? And/Or, exclude files that don't change often using the cpbackup-exclude.conf file in the root of your account?

Also, any particular format for entries in this file?
I've added one as

/home/<account>/public_html/<folder to exclude>

Will that work?

[Infopro]

Try this:

public_html/<folder to exclude>

I just tried it on a very small account after doing a normal Full Backup before and after to compare.

I added this:
public_html/fun
And it is not in the homedir backup. (which is inside the Full Backup)

Seems to work. This is new to cPanel, so YMMV.

[jamina1]

Thank you so very much for your assistance! Hopefully this will go a long way toward solving our problems!