The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server Load Problem

Discussion in 'General Discussion' started by celliott, Jan 30, 2006.

  1. celliott

    celliott Well-Known Member

    Joined:
    Jan 2, 2006
    Messages:
    460
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    United Kingdom
    I am currently experiencing some problems with my server. Load used to be very good, at peak times it being just 0.6 or so.

    Now the load is currently 3-10 and is a bit all over the place.

    When looking at current cpu usage, I cant se any particular process which is using a lot of cpu or memory:

    Things in Top also look normal, cpu usage at 7%. A quick restart of Apache brings the load down, however I cant see that its using any excessive amount of CPU Or Memory. Could this be a possible attack?

    Code:
    Pid Owner Priority Cpu % Mem % Command 
    21471 nobody 0  1.6  1.6 /usr/local/apache/bin/httpd -DSSL  
    21474 nobody 0  1.6  1.6 /usr/local/apache/bin/httpd -DSSL  
    21472 nobody 0  1.3  1.6 /usr/local/apache/bin/httpd -DSSL  
    21483 nobody 0  1.3  1.6 /usr/local/apache/bin/httpd -DSSL  
    21488 nobody 0  1.3  1.7 /usr/local/apache/bin/httpd -DSSL  
    22029 root 0  0.7  0.1 top -n 2 -b -c  
    17531 root 0  0.3  0.2 ./sc_serv config11.conf  
    1 root 0  0.0  0.1 init [3]  
    2 root 19  0.0  0.0 [ksoftirqd/0]  
    28 root 0  0.0  0.0 [pdflush]  
    29 root 0  0.0  0.0 [pdflush]  
    19 root 0  0.0  0.0 [khubd]  
    30 root 0  0.0  0.0 [kswapd0]  
    105 root 0  0.0  0.0 [kseriod]  
    173 root 0  0.0  0.0 [kjournald]  
    1008 root 0  0.0  0.0 [shpchpd_event]  
    1316 root 0  0.0  0.0 [kjournald]  
    1858 root 0  0.0  0.1 syslogd -m 0  
    1862 root 0  0.0  0.0 klogd -x  
    1896 root 0  0.0  0.1 rpc.idmapd  
    1963 root 0  0.0  0.1 /usr/sbin/smartd  
    1972 root 0  0.0  0.0 /usr/sbin/acpid  
    3807 named 0  0.0  0.3 /usr/sbin/named -u named -t /var/named/chroot  
    3850 root 0  0.0  0.1 /usr/sbin/sshd  
    3863 root 0  0.0  0.1 xinetd -stayalive -pidfile /var/run/xinetd.pid  
    3964 root 0  0.0  1.0 /usr/sbin/clamd  
    3970 mailnull 0  0.0  0.2 /usr/sbin/exim -bd -q60m  
    3975 mailnull 0  0.0  0.1 /usr/sbin/exim -tls-on-connect -bd -oX 465  
    3980 root 0  0.0  0.1 antirelayd 
    4017 root 0  0.0  0.1 crond  
    4144 root 0  0.0  0.3 cpsrvd - waiting for connections 
    4149 root 19  0.0  0.7 cpanellogd - setting up logs for bujingai 
    4211 cpanel 0  0.0  0.1 /usr/bin/stunnel-4.04local /usr/local/cpanel/etc/stunnel/default/stunnel.conf.run  
    4229 root 0  0.0  0.4 cppop - accepting on port 110 
    4277 root 0  0.0  0.1 pure-ftpd (SERVER)  
    4280 root 0  0.0  0.1 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth  
    4334 mailman 0  0.0  0.1 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/mailmanctl -s start  
    4342 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=ArchRunner:0:1 -s  
    4343 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=BounceRunner:0:1 -s  
    4344 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=CommandRunner:0:1 -s  
    4345 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s  
    4346 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=NewsRunner:0:1 -s  
    4347 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s  
    4348 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=VirginRunner:0:1 -s  
    4349 mailman 0  0.0  0.2 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=RetryRunner:0:1 -s  
    4353 dbus 0  0.0  0.1 dbus-daemon-1 --system  
    4368 root 0  0.0  0.2 hald  
    4394 root 0  0.0  0.0 /usr/sbin/portsentry -tcp  
    4433 root 0  0.0  0.0 /sbin/mingetty tty1  
    4434 root 0  0.0  0.0 /sbin/mingetty tty2  
    4435 root 0  0.0  0.0 /sbin/mingetty tty3  
    4436 root 0  0.0  0.0 /sbin/mingetty tty4  
    4437 root 0  0.0  0.0 /sbin/mingetty tty5  
    4438 root 0  0.0  0.0 /sbin/mingetty tty6  
    4826 root 0  0.0  0.1 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid  
    4850 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    4863 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    4864 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    4902 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    5029 root 0  0.0  0.8 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5 
    5044 root 0  0.0  2.0 spamd child 
    5045 root 0  0.0  1.7 spamd child 
    5483 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    5702 jcoenen 0  0.0  0.1 imapd  
    5746 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    5750 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    6336 mailnull 0  0.0  0.1 /usr/sbin/exim -oX 26 -bd  
    6388 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    6394 mailnull 0  0.0  0.2 /usr/bin/perl /usr/local/cpanel/bin/eximstats  
    6439 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    6489 root 0  0.0  0.2 chkservd 
    8039 root 0  0.0  0.1 ./ventrilo_srv  
    15292 mysql 0  0.0  1.2 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/Serv1.3Qhost.net.pid --skip-locking  
    17218 root 19  0.0  0.2 ./server_linux  
    17219 root 0  0.0  0.2 ./server_linux  
    17220 root 0  0.0  0.2 ./server_linux  
    17221 root 0  0.0  0.2 ./server_linux  
    17222 root 0  0.0  0.2 ./server_linux  
    17223 root 0  0.0  0.2 ./server_linux  
    17224 root 0  0.0  0.2 ./server_linux  
    17225 root 0  0.0  0.2 ./server_linux  
    17226 root 0  0.0  0.2 ./server_linux  
    17227 root 0  0.0  0.2 ./server_linux  
    17228 root 0  0.0  0.2 ./server_linux  
    17229 root 0  0.0  0.2 ./server_linux  
    17339 root 0  0.0  0.2 ./sc_serv config1.conf  
    17478 root 0  0.0  0.2 ./sc_serv config10.conf  
    18389 root 0  0.0  0.2 ./sc_serv config15.conf  
    18435 root 0  0.0  0.2 ./sc_serv config2.conf  
    18479 root 0  0.0  0.2 ./sc_serv config21.conf  
    18523 root 0  0.0  0.2 ./sc_serv config20.conf  
    19203 root 0  0.0  0.2 ./sc_serv config3.conf  
    19250 root 0  0.0  0.2 ./sc_serv config5.conf  
    19302 root 0  0.0  0.2 ./sc_serv config6.conf  
    19346 root 0  0.0  0.2 ./sc_serv config7.conf  
    19438 root 0  0.0  0.2 ./sc_serv config8.conf  
    19977 root 0  0.0  0.2 ./sc_serv econfig1.conf  
    20020 root 0  0.0  0.2 ./sc_serv econfig2.conf  
    20028 qhostnet 0  0.0  0.1 pure-ftpd (IDLE)  
    20124 root 0  0.0  0.2 ./sc_serv econfig3.conf  
    20277 root 0  0.0  0.2 ./sc_serv econfig4.conf  
    20364 root 0  0.0  0.2 ./sc_serv econfig5.conf  
    20394 khalij 0  0.0  0.2 pure-ftpd (IDLE)  
    20412 mitch 0  0.0  0.1 pure-ftpd (IDLE)  
    20460 root 0  0.0  0.2 ./sc_serv econfig6.conf  
    21382 bujingai 19  0.0  0.7 cpanellogd - http logs for bujingai 
    21383 bujingai 19  0.0  0.0 /usr/local/cpanel/bin/logrunner 1.0 /usr/local/cpanel/3rdparty/bin/english/webalizer -N 10 -D /home/bujingai/tmp/webalizer/dns_cache.db -R 250 -p -n bujingai.monkeybum.net -o /home/bujingai/tmp/webalizer /usr/local/apache/domlogs/bujingai.monkeybum.net  
    21384 bujingai 19  0.0  0.0 /usr/local/cpanel/3rdparty/bin/english/webalizer -N 10 -D /home/bujingai/tmp/webalizer/dns_cache.db -R 250 -p -n bujingai.monkeybum.net -o /home/bujingai/tmp/webalizer /usr/local/apache/domlogs/bujingai.monkeybum.net  
    21464 root 0  0.0  1.4 /usr/local/apache/bin/httpd -DSSL  
    21470 root 0  0.0  0.3 /usr/bin/perl /usr/local/cpanel/bin/leechprotect  
    21473 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
    21475 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
    21477 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
    21490 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
    21510 nobody 0  0.0  1.6 /usr/local/apache/bin/httpd -DSSL  
    21602 mitch 0  0.0  0.1 pure-ftpd (IDLE)  
    21613 root 0  0.0  0.1 crond  
    21617 qhostnet 0  0.0  0.1 wget -q -O /dev/null http://pheonixhosting.co.uk/Cowtoon/socket.php  
    22007 nobody 0  0.0  1.5 /usr/local/apache/bin/httpd -DSSL  
    22018 root 0  0.0  0.5 whostmgrd - serving 86.137.110.158 
    22019 nobody 0  0.0  1.5 /usr/local/apache/bin/httpd -DSSL  
    22020 root 0  0.0  2.1 /usr/local/cpanel/whostmgr/bin/whostmgr2 ./top  
    22022 root 0  0.0  0.5 whostmgrd - serving 86.137.110.158 
    22023 root 0  0.0  0.5 whostmgrd - serving 86.137.110.158 
     
    #1 celliott, Jan 30, 2006
    Last edited: Jan 30, 2006
  2. celliott

    celliott Well-Known Member

    Joined:
    Jan 2, 2006
    Messages:
    460
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    United Kingdom
    I have tracked this down to it being multiple connections from one host to apache.

    I have banned the IP with Iptables. It is only lowering load for a few seconds and then its back.

    Any ideas?
     
  3. Dub

    Dub Member

    Joined:
    Sep 11, 2005
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    happend to me twice ( *2 diffrent servers) to me in the last 2 days server load suddenly shoots up to extreme levels then everything crashes and i lose all access to the box till its manually restarted.

    *2 diffrent servers in 2 datacenters almost 5000 miles apart two nights in a row at exactly 11:30 GMT
     
    #3 Dub, Jan 31, 2006
    Last edited: Jan 31, 2006
  4. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    You need to find out who/what is causing this high load problem. Did you install and properly configured the security patches/programs mentioned in these forums? It is very crucial to secure/protect your server. Good luck!
     
  5. celliott

    celliott Well-Known Member

    Joined:
    Jan 2, 2006
    Messages:
    460
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    United Kingdom
    After adding a few more ip's and hosts i spotted in netstat the load has sorted itself all out now.

    I run APF, BFD & Mod_Security with a decent ruleset. I have gone over the basic guide in the general section and have not missed anything.

    All good now though :)
     
  6. taotoon

    taotoon Well-Known Member

    Joined:
    Nov 14, 2004
    Messages:
    135
    Likes Received:
    0
    Trophy Points:
    16
    I face the same problem since 8-JAN-2006
     
  7. fmalekpour

    fmalekpour Well-Known Member
    PartnerNOC

    Joined:
    Dec 4, 2002
    Messages:
    85
    Likes Received:
    0
    Trophy Points:
    6
    same here, On all servers on and off. It mostly will happen after 4-6 days of a reboot. It seems something will get lots of memory and wont release it (leaking ???), It's not Apache, Exim, CPanel (pop,etc), MySQL. So what else do we have? Even if I stop all known services including network still memory usage is sky high, of cource after 4-6 days and on some of the boxes.

    F.Malekpour
     
  8. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    If you are positively sure that the high load issue you have is not software related, nor network, then it must be a hardware problem.
     
  9. celliott

    celliott Well-Known Member

    Joined:
    Jan 2, 2006
    Messages:
    460
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    United Kingdom
    My problem was pretty difficult to find. When running top the cpu and memory usage for all processes seemed about normal yet my server load still went high.

    When running in netstat in SSH I could clearly see about 4 IP's straight off, each of them having at least 10 connections to my server. Shortly after banning these IP's my server load returned to normal.

    I would advise running top just to have a quick check and then netstat. If you dont spot anything obvious then as AnyReed said, it sounds hardware related.
     
  10. fmalekpour

    fmalekpour Well-Known Member
    PartnerNOC

    Joined:
    Dec 4, 2002
    Messages:
    85
    Likes Received:
    0
    Trophy Points:
    6
    27 different servers, some RH9 and others CentOS 4.2, latest rpms, different hardware, from 512MB ram to 2GB, from 10 to 700 clients on one server. Some single processor some SMB, mostly IDE, Some SATA and a few SCSI, all on CPanel Stable tree, Perl 5.8.7, PHP 4.4.1 and PHP 5.0.5.

    but the problem is the same. It should be a software issue, When server works for about 4-6 days there are lots of disk writes which will get the speed of the HDD and that will cause a high load, Here is a sample "vmstat 1" result on a server booted 8 days ago:

    Code:
       procs                      memory      swap          io     system      cpu
     r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
     0  1  0  78980  51212  92044 489200    0    0   240   868  399   405 36 25 38
     1  1  0  78988  52668  91980 489392    0    8   284     8  353   190 27 25 47
     1  0  0  78988  52620  92168 489336    0    0   316     0  368   222 18 24 59
     1  0  0  78996  52808  92068 489568    0    8   276     8  342   191 24 26 50
     0  1  0  78996  52988  92252 489512    0    0   312     0  378   219 20 26 54
     7  0  0  79004  52460  92172 489716    0    8   252   736  388   478 36 33 30
     0  2  1  79004  52584  92216 489836    0    0   152   216  317   977 65 28  7
     2  0  0  79012  51640  92108 490072    0    8   268    12  360   188 25 29 46
     1  1  0  79012  51396  92308 490000    0    0   328     0  313   189 19 30 51
     2  0  0  79020  45684  92248 490244    0    8   236  1020  389   267 47 25 27
     4  0  0  79020  49008  92312 490484    0    0   352    84  398   185 62 15 24
     5  1  0  79028  44104  92172 490824    0    8   272   160  368   217 66 15 20
     3  0  0  79028  47700  92312 490844    0    0   224   812  450   348 64 20 16
     0  2  1  79036  50804  92180 491188    0    8   276  1108  384   247 52 22 26
     0  4  2  79036  50920  92288 491256    0    0   180   780  427   192 30 11 59
     0  1  0  79044  53308  92252 491452    0    8   316   308  423   498 29 25 45
     0  1  0  79044  53196  92452 491380    0    0   328     0  343   225 26 26 49
     0  0  0  79044  54248  92300 491828    0    8   356     8  324   381 34 24 43
     2  0  0  79052  51144  92104 492180    0    8   132   248  325   183 45 19 37

    ** lots of "bi" and wont get down **

    But when we restart the same server and let it works for a few hours result will be something like this:

    Code:
       procs                      memory      swap          io     system      cpu
     r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
     0  0  0  44984  44168 232872 686380    0    0    56     0  156    99 18  5 77
     0  0  0  44984  43744 232872 686380    0    0     0     0  137    50 13  2 85
     0  0  0  44984  42956 232872 686380    0    0     0     0  151    72 20  0 80
     1  0  0  44984  43200 232876 686400    0    0     0   444  193   117 14  4 82
     3  0  0  44984  40736 232884 686460    0    0    44   524  243   241 81  8 11
     0  0  0  44984  44092 232884 686464    0    0    16   128  152   121 13  2 85
     0  0  0  44984  44188 232884 686464    0    0     0     0  134    50  8  1 91
     0  0  0  44984  49572 232904 686468    0    0    20     0  148    66  0  0 100
     0  0  0  44984  49708 232908 686468    0    0     0   572  156    77  0  1 99
     0  0  0  44984  49996 232908 686484    0    0    16     0  180    90 17  5 78
    10  0  0  44984  50048 232908 686384    0    0     0     0  174    90  2  6 92
     0  0  0  44984  50200 232908 686404    0    0     0   356  146   150 24 15 61
     0  0  0  44984  50260 232924 686416    0    0    20   488  205   126  9  3 88
     0  0  0  44984  50252 232924 686420    0    0     0     0  126    48  8  2 90
     0  0  0  44984  50200 232928 686588    0    0   172     0  158   137  4  9 87

    ** bi going up and down which is normal **

    When hard drive getting slow even a simple "find" command will raise the load.

    F.Malekpour
     
  11. fmalekpour

    fmalekpour Well-Known Member
    PartnerNOC

    Joined:
    Dec 4, 2002
    Messages:
    85
    Likes Received:
    0
    Trophy Points:
    6
    So you mean that could be some kind of attack, That's interesting. I'll definitely check it next time one of the servers alarmed. netstat -nope should do that.



    F.Malekpour
     
  12. celliott

    celliott Well-Known Member

    Joined:
    Jan 2, 2006
    Messages:
    460
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    United Kingdom
    Im not 100% sure whether this was the same problem as yours but yes I had some form of attack on my server. Since banning these IP's I have had a nice consistant load of about 0.6 and it has not gone above since.
     
  13. Dub

    Dub Member

    Joined:
    Sep 11, 2005
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    this is happening to me every 1 day 9 hours on 2 seperate servers get a

    "cpsrvd failed @ Fri Feb 3 05:06:27 2006. A restart was attempted automagicly."

    email then it goes down hill from there. Running stable release atm.

    Server Load Averages 8.08, 7.54, 6.24 (and climbing)
    Server Uptime 1 day, 9:32

    The load normally sits at .2-.4

    it must be somthing running at that time but i cant work out what :(
     
Loading...

Share This Page