Server Load Problem

**celliott** · 01-30-2006, 01:14 PM

I am currently experiencing some problems with my server. Load used to be very good, at peak times it being just 0.6 or so.

Now the load is currently 3-10 and is a bit all over the place.

When looking at current cpu usage, I cant se any particular process which is using a lot of cpu or memory:

Things in Top also look normal, cpu usage at 7%. A quick restart of Apache brings the load down, however I cant see that its using any excessive amount of CPU Or Memory. Could this be a possible attack?

Code:

Pid Owner Priority Cpu % Mem % Command 
21471 nobody 0  1.6  1.6 /usr/local/apache/bin/httpd -DSSL  
21474 nobody 0  1.6  1.6 /usr/local/apache/bin/httpd -DSSL  
21472 nobody 0  1.3  1.6 /usr/local/apache/bin/httpd -DSSL  
21483 nobody 0  1.3  1.6 /usr/local/apache/bin/httpd -DSSL  
21488 nobody 0  1.3  1.7 /usr/local/apache/bin/httpd -DSSL  
22029 root 0  0.7  0.1 top -n 2 -b -c  
17531 root 0  0.3  0.2 ./sc_serv config11.conf  
1 root 0  0.0  0.1 init [3]  
2 root 19  0.0  0.0 [ksoftirqd/0]  
28 root 0  0.0  0.0 [pdflush]  
29 root 0  0.0  0.0 [pdflush]  
19 root 0  0.0  0.0 [khubd]  
30 root 0  0.0  0.0 [kswapd0]  
105 root 0  0.0  0.0 [kseriod]  
173 root 0  0.0  0.0 [kjournald]  
1008 root 0  0.0  0.0 [shpchpd_event]  
1316 root 0  0.0  0.0 [kjournald]  
1858 root 0  0.0  0.1 syslogd -m 0  
1862 root 0  0.0  0.0 klogd -x  
1896 root 0  0.0  0.1 rpc.idmapd  
1963 root 0  0.0  0.1 /usr/sbin/smartd  
1972 root 0  0.0  0.0 /usr/sbin/acpid  
3807 named 0  0.0  0.3 /usr/sbin/named -u named -t /var/named/chroot  
3850 root 0  0.0  0.1 /usr/sbin/sshd  
3863 root 0  0.0  0.1 xinetd -stayalive -pidfile /var/run/xinetd.pid  
3964 root 0  0.0  1.0 /usr/sbin/clamd  
3970 mailnull 0  0.0  0.2 /usr/sbin/exim -bd -q60m  
3975 mailnull 0  0.0  0.1 /usr/sbin/exim -tls-on-connect -bd -oX 465  
3980 root 0  0.0  0.1 antirelayd 
4017 root 0  0.0  0.1 crond  
4144 root 0  0.0  0.3 cpsrvd - waiting for connections 
4149 root 19  0.0  0.7 cpanellogd - setting up logs for bujingai 
4211 cpanel 0  0.0  0.1 /usr/bin/stunnel-4.04local /usr/local/cpanel/etc/stunnel/default/stunnel.conf.run  
4229 root 0  0.0  0.4 cppop - accepting on port 110 
4277 root 0  0.0  0.1 pure-ftpd (SERVER)  
4280 root 0  0.0  0.1 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth  
4334 mailman 0  0.0  0.1 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/mailmanctl -s start  
4342 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=ArchRunner:0:1 -s  
4343 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=BounceRunner:0:1 -s  
4344 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=CommandRunner:0:1 -s  
4345 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s  
4346 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=NewsRunner:0:1 -s  
4347 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s  
4348 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=VirginRunner:0:1 -s  
4349 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=RetryRunner:0:1 -s  
4353 dbus 0  0.0  0.1 dbus-daemon-1 --system  
4368 root 0  0.0  0.2 hald  
4394 root 0  0.0  0.0 /usr/sbin/portsentry -tcp  
4433 root 0  0.0  0.0 /sbin/mingetty tty1  
4434 root 0  0.0  0.0 /sbin/mingetty tty2  
4435 root 0  0.0  0.0 /sbin/mingetty tty3  
4436 root 0  0.0  0.0 /sbin/mingetty tty4  
4437 root 0  0.0  0.0 /sbin/mingetty tty5  
4438 root 0  0.0  0.0 /sbin/mingetty tty6  
4826 root 0  0.0  0.1 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid  
4850 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
4863 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
4864 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
4902 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
5029 root 0  0.0  0.8 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5 
5044 root 0  0.0  2.0 spamd child 
5045 root 0  0.0  1.7 spamd child 
5483 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
5702 jcoenen 0  0.0  0.1 imapd  
5746 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
5750 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
6336 mailnull 0  0.0  0.1 /usr/sbin/exim -oX 26 -bd  
6388 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
6394 mailnull 0  0.0  0.2 /usr/bin/perl /usr/local/cpanel/bin/eximstats  
6439 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
6489 root 0  0.0  0.2 chkservd 
8039 root 0  0.0  0.1 ./ventrilo_srv  
15292 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
17218 root 19  0.0  0.2 ./server_linux  
17219 root 0  0.0  0.2 ./server_linux  
17220 root 0  0.0  0.2 ./server_linux  
17221 root 0  0.0  0.2 ./server_linux  
17222 root 0  0.0  0.2 ./server_linux  
17223 root 0  0.0  0.2 ./server_linux  
17224 root 0  0.0  0.2 ./server_linux  
17225 root 0  0.0  0.2 ./server_linux  
17226 root 0  0.0  0.2 ./server_linux  
17227 root 0  0.0  0.2 ./server_linux  
17228 root 0  0.0  0.2 ./server_linux  
17229 root 0  0.0  0.2 ./server_linux  
17339 root 0  0.0  0.2 ./sc_serv config1.conf  
17478 root 0  0.0  0.2 ./sc_serv config10.conf  
18389 root 0  0.0  0.2 ./sc_serv config15.conf  
18435 root 0  0.0  0.2 ./sc_serv config2.conf  
18479 root 0  0.0  0.2 ./sc_serv config21.conf  
18523 root 0  0.0  0.2 ./sc_serv config20.conf  
19203 root 0  0.0  0.2 ./sc_serv config3.conf  
19250 root 0  0.0  0.2 ./sc_serv config5.conf  
19302 root 0  0.0  0.2 ./sc_serv config6.conf  
19346 root 0  0.0  0.2 ./sc_serv config7.conf  
19438 root 0  0.0  0.2 ./sc_serv config8.conf  
19977 root 0  0.0  0.2 ./sc_serv econfig1.conf  
20020 root 0  0.0  0.2 ./sc_serv econfig2.conf  
20028 qhostnet 0  0.0  0.1 pure-ftpd (IDLE)  
20124 root 0  0.0  0.2 ./sc_serv econfig3.conf  
20277 root 0  0.0  0.2 ./sc_serv econfig4.conf  
20364 root 0  0.0  0.2 ./sc_serv econfig5.conf  
20394 khalij 0  0.0  0.2 pure-ftpd (IDLE)  
20412 mitch 0  0.0  0.1 pure-ftpd (IDLE)  
20460 root 0  0.0  0.2 ./sc_serv econfig6.conf  
21382 bujingai 19  0.0  0.7 cpanellogd - http logs for bujingai 
21383 bujingai 19  0.0  0.0 /usr/local/cpanel/bin/logrunner 1.0 /usr/local/cpanel/3rdparty/bin/english/webalizer -N 10 -D /home/bujingai/tmp/webalizer/dns_cache.db -R 250 -p -n bujingai.monkeybum.net -o /home/bujingai/tmp/webalizer /usr/local/apache/domlogs/bujingai.monkeybum.net  
21384 bujingai 19  0.0  0.0 /usr/local/cpanel/3rdparty/bin/english/webalizer -N 10 -D /home/bujingai/tmp/webalizer/dns_cache.db -R 250 -p -n bujingai.monkeybum.net -o /home/bujingai/tmp/webalizer /usr/local/apache/domlogs/bujingai.monkeybum.net  
21464 root 0  0.0  1.4 /usr/local/apache/bin/httpd -DSSL  
21470 root 0  0.0  0.3 /usr/bin/perl /usr/local/cpanel/bin/leechprotect  
21473 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
21475 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
21477 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
21490 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
21510 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
21602 mitch 0  0.0  0.1 pure-ftpd (IDLE)  
21613 root 0  0.0  0.1 crond  
21617 qhostnet 0  0.0  0.1 wget -q -O /dev/null http://pheonixhosting.co.uk/Cowtoon/socket.php  
22007 nobody 0  0.0  1.5 /usr/local/apache/bin/httpd -DSSL  
22018 root 0  0.0  0.5 whostmgrd - serving 86.137.110.158 
22019 nobody 0  0.0  1.5 /usr/local/apache/bin/httpd -DSSL  
22020 root 0  0.0  2.1 /usr/local/cpanel/whostmgr/bin/whostmgr2 ./top  
22022 root 0  0.0  0.5 whostmgrd - serving 86.137.110.158 
22023 root 0  0.0  0.5 whostmgrd - serving 86.137.110.158

**celliott** · 01-30-2006, 01:45 PM

I have tracked this down to it being multiple connections from one host to apache.

I have banned the IP with Iptables. It is only lowering load for a few seconds and then its back.

Any ideas?

**Dub** · 01-31-2006, 05:45 PM

happend to me twice ( *2 diffrent servers) to me in the last 2 days server load suddenly shoots up to extreme levels then everything crashes and i lose all access to the box till its manually restarted.

*2 diffrent servers in 2 datacenters almost 5000 miles apart two nights in a row at exactly 11:30 GMT

**AndyReed** · 01-31-2006, 06:54 PM

Originally Posted by Dub

happend to me twice ( *2 diffrent servers) to me in the last 2 days server load suddenly shoots up to extreme levels then everything crashes and i lose all access to the box till its manually restarted.

*2 diffrent servers in 2 datacenters almost 5000 miles apart two nights in a row at exactly 11:30 GMT

You need to find out who/what is causing this high load problem. Did you install and properly configured the security patches/programs mentioned in these forums? It is very crucial to secure/protect your server. Good luck!

**celliott** · 02-01-2006, 09:58 AM

After adding a few more ip's and hosts i spotted in netstat the load has sorted itself all out now.

I run APF, BFD & Mod_Security with a decent ruleset. I have gone over the basic guide in the general section and have not missed anything.

All good now though

**taotoon** · 02-02-2006, 12:42 PM

I face the same problem since 8-JAN-2006

**fmalekpour** · 02-02-2006, 01:23 PM

same here, On all servers on and off. It mostly will happen after 4-6 days of a reboot. It seems something will get lots of memory and wont release it (leaking ???), It's not Apache, Exim, CPanel (pop,etc), MySQL. So what else do we have? Even if I stop all known services including network still memory usage is sky high, of cource after 4-6 days and on some of the boxes.

F.Malekpour

**AndyReed** · 02-02-2006, 03:02 PM

Originally Posted by fmalekpour

same here, On all servers on and off. It mostly will happen after 4-6 days of a reboot. It seems something will get lots of memory and wont release it (leaking ???), It's not Apache, Exim, CPanel (pop,etc), MySQL. So what else do we have? Even if I stop all known services including network still memory usage is sky high, of cource after 4-6 days and on some of the boxes.

If you are positively sure that the high load issue you have is not software related, nor network, then it must be a hardware problem.

**celliott** · 02-02-2006, 03:33 PM

My problem was pretty difficult to find. When running top the cpu and memory usage for all processes seemed about normal yet my server load still went high.

When running in netstat in SSH I could clearly see about 4 IP's straight off, each of them having at least 10 connections to my server. Shortly after banning these IP's my server load returned to normal.

I would advise running top just to have a quick check and then netstat. If you dont spot anything obvious then as AnyReed said, it sounds hardware related.

**fmalekpour** · 02-02-2006, 03:44 PM

Originally Posted by AndyReed

If you are positively sure that the high load issue you have is not software related, nor network, then it must be a hardware problem.

27 different servers, some RH9 and others CentOS 4.2, latest rpms, different hardware, from 512MB ram to 2GB, from 10 to 700 clients on one server. Some single processor some SMB, mostly IDE, Some SATA and a few SCSI, all on CPanel Stable tree, Perl 5.8.7, PHP 4.4.1 and PHP 5.0.5.

but the problem is the same. It should be a software issue, When server works for about 4-6 days there are lots of disk writes which will get the speed of the HDD and that will cause a high load, Here is a sample "vmstat 1" result on a server booted 8 days ago:

Code:

   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0  1  0  78980  51212  92044 489200    0    0   240   868  399   405 36 25 38
 1  1  0  78988  52668  91980 489392    0    8   284     8  353   190 27 25 47
 1  0  0  78988  52620  92168 489336    0    0   316     0  368   222 18 24 59
 1  0  0  78996  52808  92068 489568    0    8   276     8  342   191 24 26 50
 0  1  0  78996  52988  92252 489512    0    0   312     0  378   219 20 26 54
 7  0  0  79004  52460  92172 489716    0    8   252   736  388   478 36 33 30
 0  2  1  79004  52584  92216 489836    0    0   152   216  317   977 65 28  7
 2  0  0  79012  51640  92108 490072    0    8   268    12  360   188 25 29 46
 1  1  0  79012  51396  92308 490000    0    0   328     0  313   189 19 30 51
 2  0  0  79020  45684  92248 490244    0    8   236  1020  389   267 47 25 27
 4  0  0  79020  49008  92312 490484    0    0   352    84  398   185 62 15 24
 5  1  0  79028  44104  92172 490824    0    8   272   160  368   217 66 15 20
 3  0  0  79028  47700  92312 490844    0    0   224   812  450   348 64 20 16
 0  2  1  79036  50804  92180 491188    0    8   276  1108  384   247 52 22 26
 0  4  2  79036  50920  92288 491256    0    0   180   780  427   192 30 11 59
 0  1  0  79044  53308  92252 491452    0    8   316   308  423   498 29 25 45
 0  1  0  79044  53196  92452 491380    0    0   328     0  343   225 26 26 49
 0  0  0  79044  54248  92300 491828    0    8   356     8  324   381 34 24 43
 2  0  0  79052  51144  92104 492180    0    8   132   248  325   183 45 19 37

** lots of "bi" and wont get down **

But when we restart the same server and let it works for a few hours result will be something like this:

Code:

   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0  0  0  44984  44168 232872 686380    0    0    56     0  156    99 18  5 77
 0  0  0  44984  43744 232872 686380    0    0     0     0  137    50 13  2 85
 0  0  0  44984  42956 232872 686380    0    0     0     0  151    72 20  0 80
 1  0  0  44984  43200 232876 686400    0    0     0   444  193   117 14  4 82
 3  0  0  44984  40736 232884 686460    0    0    44   524  243   241 81  8 11
 0  0  0  44984  44092 232884 686464    0    0    16   128  152   121 13  2 85
 0  0  0  44984  44188 232884 686464    0    0     0     0  134    50  8  1 91
 0  0  0  44984  49572 232904 686468    0    0    20     0  148    66  0  0 100
 0  0  0  44984  49708 232908 686468    0    0     0   572  156    77  0  1 99
 0  0  0  44984  49996 232908 686484    0    0    16     0  180    90 17  5 78
10  0  0  44984  50048 232908 686384    0    0     0     0  174    90  2  6 92
 0  0  0  44984  50200 232908 686404    0    0     0   356  146   150 24 15 61
 0  0  0  44984  50260 232924 686416    0    0    20   488  205   126  9  3 88
 0  0  0  44984  50252 232924 686420    0    0     0     0  126    48  8  2 90
 0  0  0  44984  50200 232928 686588    0    0   172     0  158   137  4  9 87

** bi going up and down which is normal **

When hard drive getting slow even a simple "find" command will raise the load.

F.Malekpour

**fmalekpour** · 02-02-2006, 03:52 PM

Originally Posted by celliott

My problem was pretty difficult to find. When running top the cpu and memory usage for all processes seemed about normal yet my server load still went high.

When running in netstat in SSH I could clearly see about 4 IP's straight off, each of them having at least 10 connections to my server. Shortly after banning these IP's my server load returned to normal.

I would advise running top just to have a quick check and then netstat. If you dont spot anything obvious then as AnyReed said, it sounds hardware related.

So you mean that could be some kind of attack, That's interesting. I'll definitely check it next time one of the servers alarmed. netstat -nope should do that.

F.Malekpour

**celliott** · 02-02-2006, 03:59 PM

Im not 100% sure whether this was the same problem as yours but yes I had some form of attack on my server. Since banning these IP's I have had a nice consistant load of about 0.6 and it has not gone above since.

**Dub** · 02-02-2006, 11:44 PM

this is happening to me every 1 day 9 hours on 2 seperate servers get a

"cpsrvd failed @ Fri Feb 3 05:06:27 2006. A restart was attempted automagicly."

email then it goes down hill from there. Running stable release atm.

Server Load Averages 8.08, 7.54, 6.24 (and climbing)
Server Uptime 1 day, 9:32

The load normally sits at .2-.4

it must be somthing running at that time but i cant work out what

Forums

Thread: Server Load Problem

LinkBack

Thread Tools

Server Load Problem

Similar Threads

Server load problem.

Server load at 40.00! Could this be the the problem?

Server Load Problem

Server Load problem

server load problem

cPanel & WHM

Enkompass

Help

Community

Company

Connect with Us