The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

weekly Backup, server becomes unresponsive

Discussion in 'Data Protection' started by mohit, Aug 29, 2010.

  1. mohit

    mohit Well-Known Member

    Joined:
    Jul 12, 2005
    Messages:
    553
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sticky On Internet
    since last 2 sundays I am having a issue with a server.

    normally on weekdays the server remains under load of 1.0
    its a intel 3450 with 4GB ram, 2x 500GB HDD but no RAID, running cPanel 11.25.0-R46156

    everyweek when backup runs and completes 400 accounts out of 600 accounts the server goes wild, with extreeme load and then becomes un-responsive.

    Got the HDD checked by data center, they say there's no issue with HDD.

    Is there a way to restart the backup from the same accoutn from where it stopped. I don't want to re-run the full 400 accounts all over again when its already done.

    Any help appreciated

    thanks
     
    #1 mohit, Aug 29, 2010
    Last edited: Aug 29, 2010
  2. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,468
    Likes Received:
    196
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    There is no way that I'm aware of to pick up where it left off. You might want to do Incremental backups instead which may help a bit.

    Personally I'd be more interested in whats causing that load and would be checking those backups already taken and the accounts for signs of problems.

    Is that server running SuPHP? If no, might there be a boat load of files in one or more accounts owned by nobody uploaded thru a script of some sort that backup is choking on? Are one or more accounts running out of space? Just guessing of course but I think more investigation of the accounts being backed up might be a sound idea.

    If all is well, you might want to disable the largest accounts from the Backup in your config, and/or have those largest accounts make better use of the file called cpbackup-exclude.conf which should be located in the root of all accounts.

    With that you could for example, set certain directories of some accounts to NOT be backed up, while still backing up the most important data for those accounts. For instance, if a user has a directory full of videos, or MP3s and it doesn't change much.

    There are several threads around here on that file called cpbackup-exclude.conf you might want to look into if this is an option for you.

    HTH.
     
  3. mohit

    mohit Well-Known Member

    Joined:
    Jul 12, 2005
    Messages:
    553
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sticky On Internet
    Any other thoughts, I think time line for high load is same every sunday.
    data center upgraded kernel, swap was not used after update this was also fixed still the same issue.

    user cron are checked but anyone with expert tips on how to check thoroughly if they are a causing a problem.

    thanks for your time, any other thoughts.
     
  4. syslint

    syslint Well-Known Member

    Joined:
    Oct 9, 2006
    Messages:
    249
    Likes Received:
    6
    Trophy Points:
    18
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Twitter:
    Just a simple question , what type of backup you are using ? incremental or tar backup
     
  5. mohit

    mohit Well-Known Member

    Joined:
    Jul 12, 2005
    Messages:
    553
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sticky On Internet
    TAR Backup.

    I'll be running backup this saturday Night instead of sunday morning, will post if backup fails again.
     
  6. syslint

    syslint Well-Known Member

    Joined:
    Oct 9, 2006
    Messages:
    249
    Likes Received:
    6
    Trophy Points:
    18
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Twitter:
    That that will be the reason. Your I/O is killing the server . Try to change it to incremental backup. Hope this will fix your issue,
     
  7. mohit

    mohit Well-Known Member

    Joined:
    Jul 12, 2005
    Messages:
    553
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sticky On Internet
    Its not due to HIGH I/O, I've got this checked already.
     
  8. mohit

    mohit Well-Known Member

    Joined:
    Jul 12, 2005
    Messages:
    553
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sticky On Internet
    I am running backup now, the load seems ok but One thing I am not able to understand why SWAP is not being used.

    is it normal ?
    EDIT: got info that no swap usage is a good thing from this post.

    Result of top
     
    #8 mohit, Sep 4, 2010
    Last edited: Sep 4, 2010
  9. mohit

    mohit Well-Known Member

    Joined:
    Jul 12, 2005
    Messages:
    553
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sticky On Internet
    Ok, server went un-responsive in just a couple of hours of running backup.

    I was able to ssh but issue only one command and it got frozen.

    i see kswapd0 using 100% cpu
    381 root 10 -5 0 0 0 R 100.0 0.0 8:27.36 kswapd0

    can some one shed some light on this

    Data center checked the HDD twice and it has no errors.

     
  10. Alejandro P

    Alejandro P Well-Known Member

    Joined:
    Apr 6, 2007
    Messages:
    53
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    hello mohit, I have the same problem you are reporting, I though it was a hardware problem and the guys at the datacenter checked disks too(they are fine). Also kernel was updated with no different results.

    I had to disable backups to stop this from happening.

    I have the same behavior, this is what i got once the server becomes unresponsive, first service goes down is mysql, a top after mysql failure shows an increased load with kswapd using 100% cpu

    top - 10:39:36 up 1 day, 4:27, 8 users, load average: 270.10, 215.18, 122.13
    Tasks: 1032 total, 8 running, 1013 sleeping, 0 stopped, 11 zombie
    Cpu(s): 0.2%us, 6.4%sy, 0.0%ni, 93.1%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 12290532k total, 12249364k used, 41168k free, 314988k buffers
    Swap: 2096440k total, 560k used, 2095880k free, 8502188k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    580 root 10 -5 0 0 0 R 100.2 0.0 16:50.76 kswapd0
    11731 root 15 0 13364 1824 800 R 1.3 0.0 0:03.33 top
    12093 root 16 0 13404 1824 804 S 1.3 0.0 0:02.26 top
    12717 root 15 0 34852 2856 1424 S 1.0 0.0 0:00.03 couriertls
    12720 codigo 18 0 110m 10m 5640 R 1.0 0.1 0:00.03 php
    6519 named 18 0 436m 20m 2092 S 0.3 0.2 2:19.96 named
    12352 mailnull 15 0 67864 3960 2152 S 0.3 0.0 0:00.02 exim
    12709 mailnull 15 0 65812 3428 1632 S 0.3 0.0 0:00.01 exim
    1 root 15 0 10348 700 588 S 0.0 0.0 0:06.37 init
    2 root RT -5 0 0 0 S 0.0 0.0 0:00.92 migration/0
    3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
    4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
    5 root RT -5 0 0 0 S 0.0 0.0 0:00.08 migration/1
    6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
    7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
    8 root RT -5 0 0 0 S 0.0 0.0 0:00.02 migration/2
    9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2

    Seems this is not an isolate case, count mine as the same.

    Hopefully we could get someone to take a look on this.
     
    #10 Alejandro P, Sep 4, 2010
    Last edited: Sep 4, 2010
  11. mohit

    mohit Well-Known Member

    Joined:
    Jul 12, 2005
    Messages:
    553
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sticky On Internet
    Yes I am quite sure this is some bug, I am having sleepless nights.

    my server hardly crosses 1.0 load on most days, Only 1 busy site with 3GB content but its already excluded from backup list.
    gets traffic only for a day or two in a month.

    Even on peak traffic my server never even touches load of 2, but once backup has run for couple of hours its kswapd0 eats my cpu/ram and reboot brings it back to life.

    Both primary and backup drives have more than 200GB available.
    both drives checked by data center, cpu burn-in, ram tested, kernel updated

    do share if you find a solution and I'll do the same if i find some.
     
  12. Alejandro P

    Alejandro P Well-Known Member

    Joined:
    Apr 6, 2007
    Messages:
    53
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Mohit, I had to disable cpanel backups to avoid this from happening, it is really a nightmare to stay almost awake while backup runs.

    I would like to see some help from cpanel techs on this forum.
     
  13. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,468
    Likes Received:
    196
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    If you suspect a problem with your cPanel you should put in a ticket to support. These forums are not the official support channel.

    You'll find the link to Support on the top right corner of every page of these forums.
     
Loading...

Share This Page