Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Server crash during backup, high CPU usage

Discussion in 'Data Protection' started by Denis Gomes Franco, Mar 3, 2019.

  1. Denis Gomes Franco

    Denis Gomes Franco Active Member

    Joined:
    Sep 3, 2018
    Messages:
    43
    Likes Received:
    6
    Trophy Points:
    8
    Location:
    Tupã, São Paulo, Brazil
    cPanel Access Level:
    Root Administrator
    Hello.

    Something weird happened today at 3 am when I run my backups. Ever since installing cPanel I've enabled full daily backups transported to Amazon S3 (no local backups, and S3 is configured via official options, not mounted to the server) and cPanel excelled at it. But today it crashed mid backup and took the whole server down. Woke up with lots of notifications from Freshping and one customer asking why their site was down

    Restarted the server and all's fine. So I checked the logs and the last line was:

    [2019-03-03 03:52:23 -0300] Creating Archive .................................................................................... ..............................cpuwatch (Sun Mar 3 03:58:15 2019): System load is currently 3.75; waiting for it to go down below 3.50 to continue …

    After that everything was down, all websites and the panel, until I restarted the server.

    This was related to an account that has around 9gb of files but I have other accounts as large as this one that were backed up successfully on that run. Based on what was saved to S3, the server backed up about 1/5th of the accounts.

    Server is a 4-core and has plenty of free space. Vultr logs shows that in that moment the CPU usage didn't go over 200%, however that graph (last 24h) doesn't have much resolution so I can't confirm it didn't go over 375%.

    So my questions are:
    1. Anyone has any idea about what happened? Plenty of full backups completed just fine the previous days.
    2. Any way to set up an automatic server restart if it gets unresponsive like this or the backup never finishes?
    3. I know cPanel can't restore backups stored on S3, but why does the restore option in WHM only shows some accounts, even though I can see them all on the S3 interface? Just curious about this one, though.
    4. There is an option "Extra CPUs for server load" that is currently set at zero. What does this do? Also I learned that server load can't go over 4 as this is a 4-core server. Seeing that backup stopped at 3.75 load, it seems that it correctly identified the number of CPUs/cores.

    Additionally, if there's a better way to configure these backups, I'd like to hear.
     
  2. Denis Gomes Franco

    Denis Gomes Franco Active Member

    Joined:
    Sep 3, 2018
    Messages:
    43
    Likes Received:
    6
    Trophy Points:
    8
    Location:
    Tupã, São Paulo, Brazil
    cPanel Access Level:
    Root Administrator
    I just realized that it could be just Apache that had crashed but I am not sure. I also said that the panel was down but now I am not sure too as I panicked and restarted the server almost immediately, I don't remember if I tried opening WHM or not.
     
  3. GOT

    GOT Get Proactive! PartnerNOC

    Joined:
    Apr 8, 2003
    Messages:
    1,486
    Likes Received:
    188
    Trophy Points:
    193
    Location:
    Chesapeake, VA
    cPanel Access Level:
    DataCenter Provider
    The most likely scenario is that something else caused load to go high and crash the server. The cPanel logs actually show that it detected the high load and paused the backup process.

    It could have been anything causing server load to go high though usually its some form of a web attack like a WP xmlrpc attach or brute force logins.

    Might want to look at your domlogs during that period and see if you can see any attacking IPs.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. Denis Gomes Franco

    Denis Gomes Franco Active Member

    Joined:
    Sep 3, 2018
    Messages:
    43
    Likes Received:
    6
    Trophy Points:
    8
    Location:
    Tupã, São Paulo, Brazil
    cPanel Access Level:
    Root Administrator
    Thanks, I'll look into it. By the way I'm using Cloudflare on all my domains, with default security options.
     
  5. Denis Gomes Franco

    Denis Gomes Franco Active Member

    Joined:
    Sep 3, 2018
    Messages:
    43
    Likes Received:
    6
    Trophy Points:
    8
    Location:
    Tupã, São Paulo, Brazil
    cPanel Access Level:
    Root Administrator
    Quick update: today the backup ran just fine. No crashes, all accounts were copied. @GOT seems to be right, thanks for the pointer (even though I still couldn't figure out what happened yesterday).

    [Moderator Note: Moved second issue to a separate thread]
     
    #5 Denis Gomes Franco, Mar 4, 2019
    Last edited by a moderator: Mar 5, 2019
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice