weekly Backup, server becomes unresponsive

[mohit]

since last 2 sundays I am having a issue with a server.

normally on weekdays the server remains under load of 1.0
its a intel 3450 with 4GB ram, 2x 500GB HDD but no RAID, running cPanel 11.25.0-R46156

everyweek when backup runs and completes 400 accounts out of 600 accounts the server goes wild, with extreeme load and then becomes un-responsive.

Got the HDD checked by data center, they say there's no issue with HDD.

Is there a way to restart the backup from the same accoutn from where it stopped. I don't want to re-run the full 400 accounts all over again when its already done.

Any help appreciated

thanks

[Infopro]

There is no way that I'm aware of to pick up where it left off. You might want to do Incremental backups instead which may help a bit.

Personally I'd be more interested in whats causing that load and would be checking those backups already taken and the accounts for signs of problems.

Is that server running SuPHP? If no, might there be a boat load of files in one or more accounts owned by nobody uploaded thru a script of some sort that backup is choking on? Are one or more accounts running out of space? Just guessing of course but I think more investigation of the accounts being backed up might be a sound idea.

If all is well, you might want to disable the largest accounts from the Backup in your config, and/or have those largest accounts make better use of the file called cpbackup-exclude.conf which should be located in the root of all accounts.

With that you could for example, set certain directories of some accounts to NOT be backed up, while still backing up the most important data for those accounts. For instance, if a user has a directory full of videos, or MP3s and it doesn't change much.

There are several threads around here on that file called cpbackup-exclude.conf you might want to look into if this is an option for you.

HTH.

[mohit]

There is no way that I'm aware of to pick up where it left off. You might want to do Incremental backups instead which may help a bit.

Personally I'd be more interested in whats causing that load and would be checking those backups already taken and the accounts for signs of problems.

Is that server running SuPHP? If no, might there be a boat load of files in one or more accounts owned by nobody uploaded thru a script of some sort that backup is choking on? Are one or more accounts running out of space? Just guessing of course but I think more investigation of the accounts being backed up might be a sound idea.

Server is running SuPHP already and there's no sign of nobody owned files or such problem in earlier backups.

If all is well, you might want to disable the largest accounts from the Backup in your config, and/or have those largest accounts make better use of the file called cpbackup-exclude.conf which should be located in the root of all accounts.
Largest Account is already excluded from backup, last week it hung on account which is just 100MB or so.

With that you could for example, set certain directories of some accounts to NOT be backed up, while still backing up the most important data for those accounts. For instance, if a user has a directory full of videos, or MP3s and it doesn't change much.
as largest account is already having backup excluded this is not an issue with size as it seems.
There are several threads around here on that file called cpbackup-exclude.conf you might want to look into if this is an option for you.

HTH.

Any other thoughts, I think time line for high load is same every sunday.
data center upgraded kernel, swap was not used after update this was also fixed still the same issue.

user cron are checked but anyone with expert tips on how to check thoroughly if they are a causing a problem.

thanks for your time, any other thoughts.

[syslint]

Just a simple question , what type of backup you are using ? incremental or tar backup

[mohit]

Just a simple question , what type of backup you are using ? incremental or tar backup

TAR Backup.

I'll be running backup this saturday Night instead of sunday morning, will post if backup fails again.

[syslint]

That that will be the reason. Your I/O is killing the server . Try to change it to incremental backup. Hope this will fix your issue,

[mohit]

That that will be the reason. Your I/O is killing the server . Try to change it to incremental backup. Hope this will fix your issue,

Its not due to HIGH I/O, I've got this checked already.

[mohit]

I am running backup now, the load seems ok but One thing I am not able to understand why SWAP is not being used.

is it normal ?
EDIT: got info that no swap usage is a good thing from this post.

Result of top

root@abcd [~]# top
top - 09:17:57 up 6 days, 4:09, 1 user, load average: 2.02, 2.20, 2.17
Tasks: 294 total, 1 running, 289 sleeping, 0 stopped, 4 zombie
Cpu(s): 0.4%us, 0.3%sy, 0.0%ni, 86.6%id, 12.6%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 4035752k total, 3958968k used, 76784k free, 93168k buffers
Swap: 2096440k total, 180k used, 2096260k free, 2409600k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25807 blncomn 34 19 21284 1292 840 D 1.7 0.0 0:01.62 /bin/gtar -c -f - -X /home/blncomn/cpbackup-exclude.conf -X /etc/cpbacku
25879 root 15 0 12872 1252 804 R 0.7 0.0 0:00.18 top
25890 nobody 15 0 133m 45m 1904 S 0.7 1.2 0:00.08 /usr/local/apache/bin/httpd -k start -DSSL
25062 nobody 16 0 134m 45m 1936 S 0.3 1.2 0:00.29 /usr/local/apache/bin/httpd -k start -DSSL
25077 nobody 15 0 133m 45m 1936 S 0.3 1.2 0:00.18 /usr/local/apache/bin/httpd -k start -DSSL
25891 nobody 15 0 133m 45m 1908 S 0.3 1.2 0:00.08 /usr/local/apache/bin/httpd -k start -DSSL
25917 nobody 15 0 133m 45m 1940 S 0.3 1.2 0:00.03 /usr/local/apache/bin/httpd -k start -DSSL
1 root 15 0 10348 696 588 S 0.0 0.0 0:02.97 init [3]
2 root RT -5 0 0 0 S 0.0 0.0 0:00.17 [migration/0]
3 root 34 19 0 0 0 S 0.0 0.0 0:00.68 [ksoftirqd/0]
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/0]
5 root RT -5 0 0 0 S 0.0 0.0 0:00.46 [migration/1]
6 root 34 19 0 0 0 S 0.0 0.0 0:00.10 [ksoftirqd/1]
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/1]
8 root RT -5 0 0 0 S 0.0 0.0 0:00.20 [migration/2]
9 root 34 19 0 0 0 S 0.0 0.0 0:00.08 [ksoftirqd/2]
10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/2]
11 root RT -5 0 0 0 S 0.0 0.0 0:03.02 [migration/3]
12 root 34 19 0 0 0 S 0.0 0.0 0:00.27 [ksoftirqd/3]
13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/3]
14 root RT -5 0 0 0 S 0.0 0.0 0:00.13 [migration/4]
15 root 34 19 0 0 0 S 0.0 0.0 0:00.08 [ksoftirqd/4]
16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/4]

[mohit]

Ok, server went un-responsive in just a couple of hours of running backup.

I was able to ssh but issue only one command and it got frozen.

i see kswapd0 using 100% cpu
381 root 10 -5 0 0 0 R 100.0 0.0 8:27.36 kswapd0

can some one shed some light on this

Data center checked the HDD twice and it has no errors.

root@abcd [~]# top
top - 11:43:13 up 6 days, 6:34, 1 user, load average: 143.07, 95.98, 44.95
Tasks: 529 total, 7 running, 517 sleeping, 2 stopped, 3 zombie
Cpu(s): 0.1%us, 12.6%sy, 0.0%ni, 87.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4035752k total, 4012488k used, 23264k free, 57940k buffers
Swap: 2096440k total, 180k used, 2096260k free, 1103864k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
381 root 10 -5 0 0 0 R 100.0 0.0 8:27.36 kswapd0
9020 root 16 0 13004 1436 800 R 0.7 0.0 0:00.14 top
1 root 15 0 10348 696 588 S 0.0 0.0 0:02.99 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.18 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.69 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.47 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.11 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.21 migration/2
9 root 34 19 0 0 0 S 0.0 0.0 0:00.08 ksoftirqd/2
10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
11 root RT -5 0 0 0 S 0.0 0.0 0:03.05 migration/3
12 root 34 19 0 0 0 S 0.0 0.0 0:00.28 ksoftirqd/3
13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3

[Alejandro P]

hello mohit, I have the same problem you are reporting, I though it was a hardware problem and the guys at the datacenter checked disks too(they are fine). Also kernel was updated with no different results.

I had to disable backups to stop this from happening.

I have the same behavior, this is what i got once the server becomes unresponsive, first service goes down is mysql, a top after mysql failure shows an increased load with kswapd using 100% cpu

top - 10:39:36 up 1 day, 4:27, 8 users, load average: 270.10, 215.18, 122.13
Tasks: 1032 total, 8 running, 1013 sleeping, 0 stopped, 11 zombie
Cpu(s): 0.2%us, 6.4%sy, 0.0%ni, 93.1%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 12290532k total, 12249364k used, 41168k free, 314988k buffers
Swap: 2096440k total, 560k used, 2095880k free, 8502188k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
580 root 10 -5 0 0 0 R 100.2 0.0 16:50.76 kswapd0
11731 root 15 0 13364 1824 800 R 1.3 0.0 0:03.33 top
12093 root 16 0 13404 1824 804 S 1.3 0.0 0:02.26 top
12717 root 15 0 34852 2856 1424 S 1.0 0.0 0:00.03 couriertls
12720 codigo 18 0 110m 10m 5640 R 1.0 0.1 0:00.03 php
6519 named 18 0 436m 20m 2092 S 0.3 0.2 2:19.96 named
12352 mailnull 15 0 67864 3960 2152 S 0.3 0.0 0:00.02 exim
12709 mailnull 15 0 65812 3428 1632 S 0.3 0.0 0:00.01 exim
1 root 15 0 10348 700 588 S 0.0 0.0 0:06.37 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.92 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.08 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.02 migration/2
9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2

Seems this is not an isolate case, count mine as the same.

Hopefully we could get someone to take a look on this.

[mohit]

Yes I am quite sure this is some bug, I am having sleepless nights.

my server hardly crosses 1.0 load on most days, Only 1 busy site with 3GB content but its already excluded from backup list.
gets traffic only for a day or two in a month.

Even on peak traffic my server never even touches load of 2, but once backup has run for couple of hours its kswapd0 eats my cpu/ram and reboot brings it back to life.

Both primary and backup drives have more than 200GB available.
both drives checked by data center, cpu burn-in, ram tested, kernel updated

do share if you find a solution and I'll do the same if i find some.

[Alejandro P]

Mohit, I had to disable cpanel backups to avoid this from happening, it is really a nightmare to stay almost awake while backup runs.

I would like to see some help from cpanel techs on this forum.

[Infopro]

Mohit, I had to disable cpanel backups to avoid this from happening, it is really a nightmare to stay almost awake while backup runs.

I would like to see some help from cpanel techs on this forum.

If you suspect a problem with your cPanel you should put in a ticket to support. These forums are not the official support channel.

You'll find the link to Support on the top right corner of every page of these forums.