cpbackup problem

[lynnette]

We have a problem after moving to a new server where we had roughly the same setup as now

WHM 11.15.0 cPanel 11.17.0-R19434
CENTOS Enterprise 4.4 i686 on standard - WHM X v3.1.0

On the previous server the backup took about an hour to do (we don't have a terrific amount of data - 10% of the home directory is being used) backing up to a separate drive (also only using around 7%) now it is taking around 7-8 hours and it is impossible while its packaging an account, to get to the site that its backing up.

I really hope someone can help with this.

 

[lynnette]

Should also add, we upgraded to Apache 2 on this new machine.

 

[Imago]

You are lucky that automated backup is working at all. Now three months that I cannot force cpbackup to do automated backups under CentOS 5 x86_64 and Apache 2.2

Anyway, add RAM, if possible.

 

[lynnette]

You are lucky that automated backup is working at all. Now three months that I cannot force cpbackup to do automated backups under CentOS 5 x86_64 and Apache 2.2

Anyway, add RAM, if possible.

Thanks for the input, but I have twice the RAM on this server compared to the last one so its not a memory problem,

 

[Keegan]

Lynette, is it with each account that's packaged that you see it basically overwhelm the server? (I know, dumb question).

Do you see a ton of memory being used which its hanging?

watch -n 1 -d free -m

I would disable backup in the configuration, kill the main process and then children

Find pid for backup: ps -aux | grep cpbackup

Have two shells open one running memory and iostat (ram, hdd) before I ran this command and watch.

/scripts/cpbackup

I'm no expert here, I'd just want to rule out the hardware being a factor first.

 

[lynnette]

Lynette, is it with each account that's packaged that you see it basically overwhelm the server? (I know, dumb question).

Do you see a ton of memory being used which its hanging?

watch -n 1 -d free -m

I would disable backup in the configuration, kill the main process and then children

Find pid for backup: ps -aux | grep cpbackup

Have two shells open one running memory and iostat (ram, hdd) before I ran this command and watch.

/scripts/cpbackup

I'm no expert here, I'd just want to rule out the hardware being a factor first.

Thanks Keegan, I will try that and post what the results are.

With regard to your first question, yes it is, each account thats packaged.

Many thanks for the suggestions.

 

[hikaro]

Hi,

Im having same problem, can you type
ps aux|grep pkgacct
from my size i saw two process running at same time, i suspect that cause the server backup very long time and take high cpu resource, it load till 6 - 10

root 13384 0.0 0.0 1448 360 ? SN 01:43 0:00 /usr/local/cpanel/bin/cpuwatch 11.0 /scripts/pkgacct afymasor /backup/cpbackup/daily backup
root 13385 1.4 0.3 9056 7576 ? SNs 01:43 0:00 pkgacct - afymasor
root 13726 14.4 0.5 13688 11084 ? DN 01:43 0:01 pkgacct - afymasor
root 13983 0.0 0.0 4924 648 pts/1 S+ 01:43 0:00 grep pkgacct

 

[hikaro]

Lynette, have any news?

 

[lynnette]

Sorry to take so long answering but I have been trying to work through this with the technical support people and so far no joy. They have done a full check on the second hard drive to rule out hardware problems. It is still next to useless though, some of the accounts are not getting finished daily at all, some are. The server load is miles too high for what is being done.

 

[cPanelKenneth]

Lynette,

If your support ticket is with cPanel, please PM me your ticket number.

Thank you.

 

[dwh2]

Just noticed that someone had a thread on this already and I just started a new one:
http://forums.cpanel.net/showthread.php?p=354055#post354055

I have identical problem...sounds like a cpanel config issue...
Also, php memory limits had to be upgraded to work for file uploads.

 

[dwh2]

After much research it seems that there are several issues here.

If you wind up with tons of files, it can slow down the cpbackup process. When mail was converted to maildir, I noticed that the /home/username/mail/cur directory in some cases had thousands or hundreds of thousands of files. Make sure there aren't spam-accepting emails on your server. And try to clean this situation. Note that just deleting the files won't do the trick...

Another thing is that gzipping huge files kills servers. I haven't tried this yet, but apparently if you enable incremental backups, gzip is turned off so that should help a lot...but make sure your backup directory doesn't get filled up!

Another option to at least get the situation under better control is instead of daily backups every day, only backup 2-3 times a week on your least busy days (usually Sunday is low web day).

Another option is a fellow here created a backup script to deal with this problem:
http://forums.cpanel.net/showthread.php?t=58096

Some suggestions for cpanel which I may suggest in a feature request ticket if the feedback here is positive:

• It would be wonderful if cpbackup did one account at a time with a little break in between to let the server catch up a bit.
• Before beginning the backup, it could check this setting before beginning: "The load average above the number of cpus at which logs file processing should be suspended (default 0)"
• Even better, creating a load average just for backups

Those ideas came from that script

In fact, maybe they can cut a deal with the guy and use his script instead of cpbackup

 

[cPanelKenneth]

After much research it seems that there are several issues here.

If you wind up with tons of files, it can slow down the cpbackup process. When mail was converted to maildir, I noticed that the /home/username/mail/cur directory in some cases had thousands or hundreds of thousands of files. Make sure there aren't spam-accepting emails on your server. And try to clean this situation. Note that just deleting the files won't do the trick...

Another thing is that gzipping huge files kills servers. I haven't tried this yet, but apparently if you enable incremental backups, gzip is turned off so that should help a lot...but make sure your backup directory doesn't get filled up!

That is correct. Incremental backups are not compressed.

Another option to at least get the situation under better control is instead of daily backups every day, only backup 2-3 times a week on your least busy days (usually Sunday is low web day).

Another option is a fellow here created a backup script to deal with this problem:
http://forums.cpanel.net/showthread.php?t=58096

Some suggestions for cpanel which I may suggest in a feature request ticket if the feedback here is positive:

• It would be wonderful if cpbackup did one account at a time with a little break in between to let the server catch up a bit.
• Before beginning the backup, it could check this setting before beginning: "The load average above the number of cpus at which logs file processing should be suspended (default 0)"
• Even better, creating a load average just for backups

Those ideas came from that script

In fact, maybe they can cut a deal with the guy and use his script instead of cpbackup

cpbackup uses /scripts/pkgacct to create the actual archive of a user account (for non-incremental backups). pkgacct is monitored by the cpuwatch binary which is supposed to prevent a backup from exceeding a certain load average (I cannot recall the exact limit right now). If the limit is exceeded, the pkgacct processes is put to sleep until the load drops below the limit. This meets the criteria set in your list above.

The cpuwatch behavior can cause the backup to take a much longer time period on systems with a lot of accounts and/or accounts with a large amount of file space consumed. Especially when the server is already under load from external or other reasons.

Granted there are other things still to do to improve the processing time for backups, but load average is generally handled very well by cpuwatch.

 

[dwh2]

That's a relief. I'm afraid it doesn't work right though because two days ago I saw my server load hit 250 and yesterday my host's support saw 50... I got 3am and 5am calls because my server went down.

Any suggestions for verifying if CPUWatch is even working or why it doesn't seem to be working?

That is correct. Incremental backups are not compressed.

Can I just switch from one to the other cleanly even though I had a full backup running for a couple years?
What would happen? Would the old backups be deleted and recreated an incremental?

 

[dwh2]

I tried to turn on incremental backups and got this message:

Quotas cannot be built! Your cpbackup destination is on a filesystem which has quotas enabled. Please move it to a filesystem which does not have quotas turned on or a separate partition/disk slice mounted at /backup.Backup has been disabled to prevent quota problems...
Backup Configuration Saved!

huh?

 

[dwh2]

Now I try to turn it back on and it won't... Same error as above.


Help!

Why is it every single time I try to dig deep and do research and fix all the little server problems and upgrade to the "secure" versions of things, my "admin" work seems to break things more than fix them?

 

[cPanelKenneth]

That's a relief. I'm afraid it doesn't work right though because two days ago I saw my server load hit 250 and yesterday my host's support saw 50... I got 3am and 5am calls because my server went down.

Any suggestions for verifying if CPUWatch is even working or why it doesn't seem to be working?



Can I just switch from one to the other cleanly even though I had a full backup running for a couple years?
What would happen? Would the old backups be deleted and recreated an incremental?

If you system is under heavy load it can take awhile before cpuwatch is given CPU time to put the pkgacct process to sleep.

Executing ps aux | grep -i cpuwatch will show whether cpuwatch is running.

If cpuwatch is not found by cpbackup (it's normally /usr/local/cpanel/bin), then it defaults to logwatch, which does similar handing of processes, jsut not as well. If logwatch is not found, then the pkgacct command is run directly.

As for the quota issue, that's by design, to prevent the backup files from consuming the user quota. That is why we recommend placing /backup (or where ever the backups reside) on a separate partition or drive.

 

[dwh2]

If you system is under heavy load it can take awhile before cpuwatch is given CPU time to put the pkgacct process to sleep.

Executing ps aux | grep -i cpuwatch will show whether cpuwatch is running.

If cpuwatch is not found by cpbackup (it's normally /usr/local/cpanel/bin), then it defaults to logwatch, which does similar handing of processes, jsut not as well. If logwatch is not found, then the pkgacct command is run directly.

As for the quota issue, that's by design, to prevent the backup files from consuming the user quota. That is why we recommend placing /backup (or where ever the backups reside) on a separate partition or drive.

I thought that's how my host set it up. I wonder why it worked for two years though and only now did cpbackup notice?

The planet has some kind of network drive that is mapped to /mnt/backup. Can I just backup directly to there or will that be an issue since it isn't a local hard drive and there are lots of files/data to backup?

 

[cPanelKenneth]

I thought that's how my host set it up. I wonder why it worked for two years though and only now did cpbackup notice?

The planet has some kind of network drive that is mapped to /mnt/backup. Can I just backup directly to there or will that be an issue since it isn't a local hard drive and there are lots of files/data to backup?

We don't check unless you update the configuration.

Backup performance to /mnt/backup will depend upon the network configuration. Try selecting a couple large accounts for backup and test with those. That should give you an idea as to the overall impact.

 

[dwh2]

Thanks. The support guy says I should just turn off quotas by messing with the fstab. Other than hosting a few friends for whom I've given limits, I don't use the quota system. Any danger in turning it off?