The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Sites hanging during backup

Discussion in 'General Discussion' started by airaid, Jan 19, 2016.

  1. airaid

    airaid Member

    Joined:
    Feb 18, 2014
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    CENTOS 6.7 x86_64 xenpv – WHM 54.0 (build 1) 4GB RAM 4 CPU

    Recently I've started to encounter a problem where sites are going offline temporarily during the nightly backup to S3. This is now being accompanied by the chkservd process becoming non-responsive. The backups used to take approx 40 mins but now take 6.5 hours to complete and sites are offline for up to an hour.

    I run 12 not very busy sites on this server with backup sizes ranging from 10MB to 4GB. Going back several months, there was one site with a large db (2.5GB) that would go offline for a minute or two during the backup. I put this down to the db size and because no other sites were affected I thought it was safe to ignore. Now however the problem has grown to impact the entire backup process.

    My limited troubleshooting skills show that during the backup process memory might be being exhausted:

    total: 3904496
    used: 3886328
    free: 18168

    - Removed please attach images to your posts -

    I could add more memory but don't want to waste the money if it won't solve the issue. How can I properly troubleshoot this?

    I'm happy to provide more info but not sure what might be useful.

    Many thanks
     
  2. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,146
    Likes Received:
    34
    Trophy Points:
    48
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hello,

    I will suggest you please monitor your server process with top and ps aufx command during backup process so that you can find out the exact root cause your site down issues.
     
  3. airaid

    airaid Member

    Joined:
    Feb 18, 2014
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Thank you. Forgive me for asking but is there a way I can set these up to run automatically and record to a log file for later review? The backup starts at 3am so running these manually isn't practical.
     
  4. airaid

    airaid Member

    Joined:
    Feb 18, 2014
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Also, shouldn't there be existing log files that would help track the issue?
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    Please review the backup log files stored in:

    /usr/local/cpanel/logs/cpbackup/

    Also, since you are transporting the backup archives to a remote destination, the transport log may offer some information:

    /usr/local/cpanel/logs/cpbackup_transporter.log

    Do you notice any particular output to these logs files during the backup process that suggest a reason for the CPU/Memory increase?

    Thank you.
     
  6. airaid

    airaid Member

    Joined:
    Feb 18, 2014
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Thank you, very helpful. There's nothing to note in the transfer logs but the backup logs are littered with lines like the one below (including right at the top of the file which suggests the CPUs are already under heavy load):

    cpuwatch (Tue Jan 19 02:04:34 2016): System load is currently 3.54; waiting for it to go down to 3.50 to continue …

    These lines appear mostly but not exclusively during the archive creation.

    Which begs the question; what's the best way to find out what's hogging the CPU? Is there a log I can check or something I can implement to track which processes could be causing this? I found /var/log/dcpumon which looks like it could help but this only seems to store the last 30 mins of data.

    Thanks again.
     
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
Loading...

Share This Page