The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

cpbackup problem

Discussion in 'Data Protection' started by lynnette, Jan 20, 2008.

  1. lynnette

    lynnette Member

    Joined:
    Mar 27, 2004
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    We have a problem after moving to a new server where we had roughly the same setup as now

    WHM 11.15.0 cPanel 11.17.0-R19434
    CENTOS Enterprise 4.4 i686 on standard - WHM X v3.1.0

    On the previous server the backup took about an hour to do (we don't have a terrific amount of data - 10% of the home directory is being used) backing up to a separate drive (also only using around 7%) now it is taking around 7-8 hours and it is impossible while its packaging an account, to get to the site that its backing up.

    I really hope someone can help with this.
     
  2. lynnette

    lynnette Member

    Joined:
    Mar 27, 2004
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Should also add, we upgraded to Apache 2 on this new machine.
     
  3. Imago

    Imago Active Member

    Joined:
    Feb 21, 2004
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    6
    You are lucky that automated backup is working at all. Now three months that I cannot force cpbackup to do automated backups under CentOS 5 x86_64 and Apache 2.2

    Anyway, add RAM, if possible.
     
  4. lynnette

    lynnette Member

    Joined:
    Mar 27, 2004
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Thanks for the input, but I have twice the RAM on this server compared to the last one so its not a memory problem,
     
  5. Keegan

    Keegan Well-Known Member

    Joined:
    Oct 22, 2001
    Messages:
    110
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    DataCenter Provider
    Lynette, is it with each account that's packaged that you see it basically overwhelm the server? (I know, dumb question).

    Do you see a ton of memory being used which its hanging?

    watch -n 1 -d free -m

    I would disable backup in the configuration, kill the main process and then children

    Find pid for backup: ps -aux | grep cpbackup

    Have two shells open one running memory and iostat (ram, hdd) before I ran this command and watch.

    /scripts/cpbackup

    I'm no expert here, I'd just want to rule out the hardware being a factor first.
     
  6. lynnette

    lynnette Member

    Joined:
    Mar 27, 2004
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Thanks Keegan, I will try that and post what the results are.

    With regard to your first question, yes it is, each account thats packaged.

    Many thanks for the suggestions.
     
    #6 lynnette, Jan 20, 2008
    Last edited: Jan 20, 2008
  7. hikaro

    hikaro Well-Known Member

    Joined:
    Nov 22, 2005
    Messages:
    99
    Likes Received:
    0
    Trophy Points:
    16
    Hi,

    Im having same problem, can you type
    ps aux|grep pkgacct
    from my size i saw two process running at same time, i suspect that cause the server backup very long time and take high cpu resource, it load till 6 - 10

    root 13384 0.0 0.0 1448 360 ? SN 01:43 0:00 /usr/local/cpanel/bin/cpuwatch 11.0 /scripts/pkgacct afymasor /backup/cpbackup/daily backup
    root 13385 1.4 0.3 9056 7576 ? SNs 01:43 0:00 pkgacct - afymasor
    root 13726 14.4 0.5 13688 11084 ? DN 01:43 0:01 pkgacct - afymasor
    root 13983 0.0 0.0 4924 648 pts/1 S+ 01:43 0:00 grep pkgacct
     
  8. hikaro

    hikaro Well-Known Member

    Joined:
    Nov 22, 2005
    Messages:
    99
    Likes Received:
    0
    Trophy Points:
    16
    Lynette, have any news?
     
  9. lynnette

    lynnette Member

    Joined:
    Mar 27, 2004
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Sorry to take so long answering but I have been trying to work through this with the technical support people and so far no joy. They have done a full check on the second hard drive to rule out hardware problems. It is still next to useless though, some of the accounts are not getting finished daily at all, some are. The server load is miles too high for what is being done. :mad:
     
  10. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    Lynette,

    If your support ticket is with cPanel, please PM me your ticket number.

    Thank you.
     
  11. dwh2

    dwh2 Well-Known Member

    Joined:
    Jan 14, 2004
    Messages:
    106
    Likes Received:
    0
    Trophy Points:
    16
  12. dwh2

    dwh2 Well-Known Member

    Joined:
    Jan 14, 2004
    Messages:
    106
    Likes Received:
    0
    Trophy Points:
    16
    After much research it seems that there are several issues here.

    If you wind up with tons of files, it can slow down the cpbackup process. When mail was converted to maildir, I noticed that the /home/username/mail/cur directory in some cases had thousands or hundreds of thousands of files. Make sure there aren't spam-accepting emails on your server. And try to clean this situation. Note that just deleting the files won't do the trick...

    Another thing is that gzipping huge files kills servers. I haven't tried this yet, but apparently if you enable incremental backups, gzip is turned off so that should help a lot...but make sure your backup directory doesn't get filled up!

    Another option to at least get the situation under better control is instead of daily backups every day, only backup 2-3 times a week on your least busy days (usually Sunday is low web day).

    Another option is a fellow here created a backup script to deal with this problem:
    http://forums.cpanel.net/showthread.php?t=58096

    Some suggestions for cpanel which I may suggest in a feature request ticket if the feedback here is positive:

    • It would be wonderful if cpbackup did one account at a time with a little break in between to let the server catch up a bit.
    • Before beginning the backup, it could check this setting before beginning: "The load average above the number of cpus at which logs file processing should be suspended (default 0)"
    • Even better, creating a load average just for backups
    Those ideas came from that script :)

    In fact, maybe they can cut a deal with the guy and use his script instead of cpbackup :)
     
  13. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    That is correct. Incremental backups are not compressed.

    cpbackup uses /scripts/pkgacct to create the actual archive of a user account (for non-incremental backups). pkgacct is monitored by the cpuwatch binary which is supposed to prevent a backup from exceeding a certain load average (I cannot recall the exact limit right now). If the limit is exceeded, the pkgacct processes is put to sleep until the load drops below the limit. This meets the criteria set in your list above.

    The cpuwatch behavior can cause the backup to take a much longer time period on systems with a lot of accounts and/or accounts with a large amount of file space consumed. Especially when the server is already under load from external or other reasons.

    Granted there are other things still to do to improve the processing time for backups, but load average is generally handled very well by cpuwatch.
     
  14. dwh2

    dwh2 Well-Known Member

    Joined:
    Jan 14, 2004
    Messages:
    106
    Likes Received:
    0
    Trophy Points:
    16
    That's a relief. I'm afraid it doesn't work right though because two days ago I saw my server load hit 250 and yesterday my host's support saw 50... I got 3am and 5am calls because my server went down.

    Any suggestions for verifying if CPUWatch is even working or why it doesn't seem to be working?

    Can I just switch from one to the other cleanly even though I had a full backup running for a couple years?
    What would happen? Would the old backups be deleted and recreated an incremental?
     
    #14 dwh2, Feb 27, 2008
    Last edited: Feb 27, 2008
  15. dwh2

    dwh2 Well-Known Member

    Joined:
    Jan 14, 2004
    Messages:
    106
    Likes Received:
    0
    Trophy Points:
    16
    I tried to turn on incremental backups and got this message:

    huh?
     
  16. dwh2

    dwh2 Well-Known Member

    Joined:
    Jan 14, 2004
    Messages:
    106
    Likes Received:
    0
    Trophy Points:
    16
    Now I try to turn it back on and it won't... Same error as above.


    Help!

    Why is it every single time I try to dig deep and do research and fix all the little server problems and upgrade to the "secure" versions of things, my "admin" work seems to break things more than fix them? :(
     
  17. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    If you system is under heavy load it can take awhile before cpuwatch is given CPU time to put the pkgacct process to sleep.

    Executing ps aux | grep -i cpuwatch will show whether cpuwatch is running.

    If cpuwatch is not found by cpbackup (it's normally /usr/local/cpanel/bin), then it defaults to logwatch, which does similar handing of processes, jsut not as well. If logwatch is not found, then the pkgacct command is run directly.

    As for the quota issue, that's by design, to prevent the backup files from consuming the user quota. That is why we recommend placing /backup (or where ever the backups reside) on a separate partition or drive.
     
  18. dwh2

    dwh2 Well-Known Member

    Joined:
    Jan 14, 2004
    Messages:
    106
    Likes Received:
    0
    Trophy Points:
    16
    I thought that's how my host set it up. I wonder why it worked for two years though and only now did cpbackup notice?

    The planet has some kind of network drive that is mapped to /mnt/backup. Can I just backup directly to there or will that be an issue since it isn't a local hard drive and there are lots of files/data to backup?
     
  19. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    We don't check unless you update the configuration.

    Backup performance to /mnt/backup will depend upon the network configuration. Try selecting a couple large accounts for backup and test with those. That should give you an idea as to the overall impact.
     
  20. dwh2

    dwh2 Well-Known Member

    Joined:
    Jan 14, 2004
    Messages:
    106
    Likes Received:
    0
    Trophy Points:
    16
    Thanks. The support guy says I should just turn off quotas by messing with the fstab. Other than hosting a few friends for whom I've given limits, I don't use the quota system. Any danger in turning it off?
     
Loading...

Share This Page