The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Issue with cpbackup causing continual high loads

Discussion in 'Data Protection' started by simonautomatic, Sep 9, 2016.

Tags:
  1. simonautomatic

    Joined:
    Jul 17, 2008
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    I'm seeing a new issue in WHM58.0 build 28 where load averages rise to between 7-10 or higher (I have 4 Cores) and stay consistently over 5 forever, once cpbackup has started.

    I tested this by rebooting the server early one morning (07:00). Load averages returned to around 0.6 - 1, which is normal. Then according analysis from sar, at 01:10 the load averages had immediately risen to 4-5, and then carried on rising to 7-8 where they mostly plateaued.

    Checking crontab revealed that the only task which commences at 01:00 is cpbackup. Sure enough I found a few cpbackup tasks still running at around 11:00 / 12:00. Normally backups should have completed by around 08:30 at the latest. I killed the cpbackup processes, but this still doesn't return load average to anything normal.

    I'm at a loss. This looks like it might be IO related, as processor usage doesn't look dramatic at all. But what on earth is hanging on to so many IO processes, even after cpbackup has been killed off? And why is it suddenly having such a struggle to complete? I'd actually reduced the number of accounts and filesize of stuff that needs to be backed up.

    -------------
    CENTOS 6.8 x86_64 standard
    8GB Ram / 4 cores
     
  2. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,450
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Is the system sending you an email about backups completing with log attached? You might want to review that log for clues as to whats holding things up.
     
  3. simonautomatic

    Joined:
    Jul 17, 2008
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    Normally it would, but now because the load is so high, the backups never complete and thus no report is emailed. I guess I can check the log file, it must be on disk somewhere, right?
     
  4. simonautomatic

    Joined:
    Jul 17, 2008
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    So looking at /usr/local/cpanel/logs/cpbackup/.... doesn't really give any clues. I see the backups running, and gradually pausing more and more, waiting for the server load to reduce below 3.5

    Also from time to time I see:

    Load watching suspended due to SIGUSR1
     
  5. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,450
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Going by your first post, is it normal for your backups to take 7 hours?

    How big are your largest accounts?
     
  6. simonautomatic

    Joined:
    Jul 17, 2008
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    Hi,

    Yes it's normal as there are 50-60 accounts, and the largest is 30GB.

    However what has been happening for the past few days (since this problem with high loads began) is that the backup never completes. BY the morning, loads are way up (7 or 8, ie: double the number of cores) so the backup sits there doing nothing as the load never goes down.

    Please note that it is *not* other activity on the server that is causing this high load. I can see that quite clearly by a) the fact that processor usage is not abnormally high and b) after a reboot, load returned to its usual averages (<1) until cpbackup started up again.
     
  7. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,450
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    If it was me, I might exclude (from within Backup Settings) that largest account from being backed up temporarily, to see if the backup completes successfully without it. You can force a manual backup with this command:
    Code:
    /usr/local/cpanel/bin/backup --force
     
  8. simonautomatic

    Joined:
    Jul 17, 2008
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    I could try that, but I think you're misunderstanding the issue here.

    The issue is whay on earth am I seeing such huge load averages, continually, after cpbackup has started running. They never go down, the only solution I found is to reboot the server.
     
  9. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,450
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Understood. Would you still see high load without that 30GB account in the mix though? Might be worth trying to find out.
     
Loading...

Share This Page