The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

High Server Load

Discussion in 'General Discussion' started by d-woo, Nov 20, 2006.

  1. d-woo

    d-woo Well-Known Member

    Joined:
    Aug 9, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    6
    PROBLEM - High Load
    ************************************************
    Loads have been around 9.0 and average around 4.0

    I have plenty of SWAP, just high server loads

    I added Mail Scanner last week. The loads were high before then, I just notice them more now.

    I want to keep MailScanner (awesome program).

    I don't mind getting more RAM if needed, or upgrading to a Pentium Processor if needed.

    I just want to be certain that I'm doing all that I can do to run the server efficiently.


    Server Specs
    ************************************************
    Processor - 1.7 Celeron
    RAM - 512
    Websites - 25

    10 use MySQL databases with messageboards with persistent connections



    DATA
    ************************************************
    WHM Memory Usage
    yeilds:
    Code:
    11-19-2006
    User.........Domain...............%CPU............%MEM..........Mysql Processes
    root..............................22.74............3.78.........2.0
    Top Process..%CPU 66.5............gzip
    Top Process..%CPU 56.0............pkgacct - americae
    Top Process..%CPU 55.8............gzip
    mailnull..........................15.92............16.35........0.0
    Top Process..%CPU 73.0............MailScanner: checking with Spam Lists
    Top Process..%CPU 72.0............MailScanner: checking with SpamAssassin
    Top Process..%CPU 53.0............MailScanner: virus scanning
    
    TOP snapshot
    Code:
     12:25pm  up 3 days,  1:42,  1 user,  load average: 3.67, 2.43, 3.38
    140 processes: 133 sleeping, 4 running, 2 zombie, 1 stopped
    CPU states:  2.4% user, 18.8% system, 65.3% nice, 13.3% idle
    Mem:   506048K av,  477336K used,   28712K free,       0K shrd,   23824K buff
    Swap: 2048276K av,  477912K used, 1570364K free                   79948K cached
    
      PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
    22559 mailnull  16   8     0    0     0 Z N  68.0  0.0   0:01 MailScanner <defu
    18829 mailnull  13   8 74708  13M  1968 D N   4.4  2.8   0:19 MailScanner: wait
    22560 root      13   8  3108 3108  2240 S N   3.9  0.6   0:00 /usr/sbin/exim -C
    22562 oakheirl  13   8  6444 6444  3424 D N   3.4  1.2   0:00 /usr/local/cpanel
    22558 mailnull   9   0  2728 2652  1876 D     2.9  0.5   0:00 /usr/sbin/exim -b
       15 root       9   0     0    0     0 DW    0.4  0.0   1:54 kjournald
    19290 nobody     9   0  9288 5524  1892 S     0.4  1.0   0:03 /usr/local/apache
    22499 mailnull  13   8 72664  31M  1892 D N   0.4  6.3   0:00 MailScanner: viru
    22556 root      10   0  1080 1080   824 R     0.4  0.2   0:00 top
    22561 oakheirl  13   8  3148 3148  2272 S N   0.4  0.6   0:00 /usr/sbin/exim -C
        1 root       8   0   160  116    92 S     0.0  0.0   0:07 init [3]
        2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd
        3 root       9   0     0    0     0 SW    0.0  0.0   0:00 kapmd
    

    Top Memory Consuming Processes

    ps aux | head -1;ps aux --no-headers| sort -rn +3 | head
    yields:
    Code:
    USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
    oakheirl 11275  0.0  6.4 70780 32852 ?       TN   07:38   0:10 /usr/local/cpanel
    mysql     5407  0.2  5.9 79060 30160 ?       SN   Nov17  10:14 /usr/sbin/mysqld
    mysql     5369  0.2  5.9 79060 30160 ?       SN   Nov17  10:07 /usr/sbin/mysqld
    mysql     3681  0.2  5.9 79060 30160 ?       SN   Nov17   8:52 /usr/sbin/mysqld
    mysql     3217  0.2  5.9 79060 30160 ?       SN   Nov17   9:05 /usr/sbin/mysqld
    mysql     3210  0.2  5.9 79060 30160 ?       SN   Nov17  10:09 /usr/sbin/mysqld
    mysql     3159  0.2  5.9 79060 30160 ?       SN   Nov17  10:39 /usr/sbin/mysqld
    mysql     3001  0.0  5.9 79060 30160 ?       SN   Nov17   0:17 /usr/sbin/mysqld
    mysql     2998  0.0  5.9 79060 30160 ?       SN   Nov17   0:00 /usr/sbin/mysqld
    mysql     2928  0.0  5.9 79060 30160 ?       SN   Nov17   0:05 /usr/sbin/mysqld
    

    Top CPU Consuming Processes

    ps aux | head -1;ps aux --no-headers | sort -rn +2
    yields:
    Code:
    root@srv1 [~]# ps aux | head -1;ps aux --no-headers | sort -rn +2
    USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
    mailnull 22707  2.4  0.0     0    0 ?        ZN   12:26   0:01 [MailScanner <def
    mailnull 22530  1.4  0.0     0    0 ?        ZN   12:25   0:01 [MailScanner <def
    mailnull 18829  1.4  2.5 78036 13036 ?       SN   12:03   0:20 MailScanner: wait
    mailnull 18784  1.3  2.2 76232 11572 ?       SN   12:02   0:19 MailScanner: wait
    mailnull 18824  1.1  2.2 76340 11580 ?       SN   12:03   0:16 MailScanner: wait
    mailnull 22743  1.0  0.2  4984 1264 ?        S    12:27   0:00 /usr/sbin/exim -b
    root     22260  0.4  0.3  3532 1700 ?        S    12:24   0:00 /usr/sbin/sshd
    nobody   19291  0.4  1.3 17812 6988 ?        S    12:06   0:06 /usr/local/apache
    nobody   19308  0.3  1.2 17268 6276 ?        S    12:06   0:03 /usr/local/apache
    .
    .
    .
    cpanel   24359  0.0  0.1  2640  700 ?        SN   09:05   0:00 /usr/bin/stunnel-
    root@srv1 [~]#
    
    
     
  2. HelloAdam

    HelloAdam Well-Known Member

    Joined:
    Nov 6, 2005
    Messages:
    145
    Likes Received:
    0
    Trophy Points:
    16
    Hey,

    Looks like you don't have a good server for that many websites. Yes I know 25 websites but with your server specs it alot. However what you can do is not install so many high programs.

    If you do a

    Code:
    top
    
    command in SSH. It will show you the current server load. gzip is used with backing up cpanel accounts. I recommend that you use something like Incermetal Backups (spelled that wrong.) Its one of the cpanel backup options. It only backups what has changed. But does not put it into a .tar.gz format. So more space is need.

    Other then that you suggest you get a bigger server.

    You can get Server Management at http://www.eastsolid.com/services/management.php..

    From,
    Adam
     
  3. hosttosucceed

    hosttosucceed Member

    Joined:
    Oct 22, 2006
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    Perhaps take off the persistent connections as they aren't recommended for a shared hosting solution.
     
  4. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    Make sure you don't have a bad/insecure script that loops indefinitely, exhausting more of your server resources. This is one of the many variables involved in server high load. I suggest you read this article about the possible causes of server high load: http://www.servertune.com/kbase/security/server_load.html
     
  5. forlinuxsupport

    forlinuxsupport Well-Known Member
    PartnerNOC

    Joined:
    Dec 22, 2004
    Messages:
    386
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Hi

    What operating system are you using ?
    See if there is a new kernel update for it.


    Try do this :
    vmstat 3

    let that run for a few minutes and post results here.

    Mailscanner seems to be using lots of cpu time. if you tail the mail logs, is there alot of mail going in and out ?
    Disable any catchall email addresses you have as they just attract spam.

    That gzip process is probably the cpanel backup stil lrunning, that pushes up load.
    Try kill it and its children (kill -9 PID)

    I often have the cpanel backup Stuck.

    Update cpanel to the latest CURRENT version and do a force update (/scripts/upcp --force )

    Regards
    Andy
     
  6. forlinuxsupport

    forlinuxsupport Well-Known Member
    PartnerNOC

    Joined:
    Dec 22, 2004
    Messages:
    386
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    us

    ps fax
    look for the scripts there. - probably hanging off http / apache or maybe cron).

    they probably have a HIGH execution time !!

    Cheers
    andy
     
  7. d-woo

    d-woo Well-Known Member

    Joined:
    Aug 9, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    6
    Thanks for the input guys.

    I believe that one client (the only one who hosts on my box other than me) has some runaway scripts.

    How can I prove it other than disabling them?

    Is there a specific log file I can check?
     
  8. d-woo

    d-woo Well-Known Member

    Joined:
    Aug 9, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    6
    I'm running:
    WHM 10.8.0 cPanel 10.9.0-S77
    RedHat 7.3 i686 - WHM X v3.1.0

    Output from vmstat2
    procs memory swap io system cpu
    r b w swpd free buff cache si so bi bo in cs us sy id
    3 6 1 500088 53548 12816 70748 492 0 504 89 249 483 39 6 55
    2 7 1 500088 50360 12916 70876 439 0 488 87 232 329 5 1 94
    0 10 0 500088 47960 13028 70876 523 0 523 83 273 365 2 1 96
    0 11 1 500088 40648 13088 70908 472 0 484 211 270 405 6 6 87
    2 9 1 500088 34240 13192 70964 488 0 508 119 267 368 2 2 97
    1 8 1 500088 24072 13296 71200 432 0 515 140 239 429 9 8 82
    1 9 1 500088 21872 13380 71260 501 0 524 96 279 352 5 3 91
    1 9 1 500088 18696 13472 71264 505 0 507 81 257 346 29 4 66
    1 10 1 500088 15240 13572 71268 520 1 523 72 255 354 40 4 56
    0 11 1 500096 12012 13680 71276 496 7 503 95 257 514 4 3 93
    1 10 1 500576 8660 13756 71288 476 180 476 292 253 326 3 4 93
    0 14 1 500932 6128 13800 71344 424 149 436 272 252 315 2 3 95
    2 8 1 501124 5292 13816 72544 288 71 836 173 316 447 2 2 96
    1 11 1 501200 5440 13860 72040 272 519 275 595 290 272 4 2 93
    3 10 1 501260 5204 13912 71976 351 280 352 357 277 295 1 2 97
    1 14 1 501304 5464 14008 71848 113 411 172 448 334 201 3 3 95
    2 13 1 501332 5424 14044 71904 215 435 309 551 333 227 2 2 96
    0 16 1 501504 5420 14064 72240 156 641 336 724 369 225 5 3 91
    4 11 2 501896 5328 14112 71908 335 536 528 639 351 340 4 3 93
    1 13 1 501964 5408 14152 71556 237 135 415 223 322 276 1 1 98
    1 13 1 502220 5308 14144 71804 349 169 547 231 342 347 4 3 94
    procs memory swap io system cpu
    Code:
    
    
     
  9. d-woo

    d-woo Well-Known Member

    Joined:
    Aug 9, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    6
    Operating system Linux
    Service Status Click to View
    Kernel version 2.4.20-28.7
    Machine Type i686
    Apache version 1.3.36 (Unix)
    PERL version 5.8.7
    Path to PERL /usr/bin/perl
    Path to sendmail /usr/sbin/sendmail
    Installed Perl Modules Click to View
    PHP version 4.4.2
    MySQL version 4.0.25-standard
    cPanel Build 10.9.0-STABLE 77
    Theme cPanel X v2.6.0
    Documentation Click to View
    cPanel Pro 1.0 (RC1)


    No catchall email addresses are running.

    I turned off gzip backup via WHM. That is, however, still running and I have to watch top to kill the processes. Should I just reboot?
     
  10. d-woo

    d-woo Well-Known Member

    Joined:
    Aug 9, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    6
    Just found this:

    21688 nobody 16 4 17460 1148 1076 R N 97.1 0.2 7:17 /usr/local/apache/bin/httpd -DSSL

    What is /usr/local/apache/bin/httpd -DSSL ?
     
  11. Fredi

    Fredi Member

    Joined:
    Apr 9, 2003
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    I have a similar problem.

    29907 nobody 0 5.0 0.3 /usr/local/apache/bin/httpd -DSSL
    9718 nobody 0 4.6 0.5 /usr/local/apache/bin/httpd -DSSL
    9781 nobody 0 3.3 0.5 /usr/local/apache/bin/httpd -DSSL and other.......
     
  12. freedog96150

    freedog96150 Well-Known Member

    Joined:
    Mar 25, 2005
    Messages:
    68
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Nevada, USA
    That is your Apache daemon.
     
  13. d-woo

    d-woo Well-Known Member

    Joined:
    Aug 9, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    6
    I'm at my wits end.

    I removed all .cgi scripts and a few PHP email test scripts that were on my box... then rebooted the server.

    The load stayed below 1.5 for over 24 hours then started back up to 12 and 13.

    I then turned off virus scan to lower the load (keeping Mail Scan for Spam on)

    I see no difference in load. Still 12 and 13.
     
  14. ramprage

    ramprage Well-Known Member

    Joined:
    Jul 21, 2002
    Messages:
    667
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    I think your problem here is MailScanner itself, it looks like its responsbile for causing the high server load on your system.

    It's possible a users account/script may be causing this but you really need to hire someone to review your server and optimze/secure it. From your posts you're running very old versions of software which can lead to hacks and root on your server.
     
  15. d-woo

    d-woo Well-Known Member

    Joined:
    Aug 9, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    6
    I agree.

    I'm in the process of getting a new box altogether and transferring everything over.

    Thanks!
     
  16. abubin

    abubin Well-Known Member

    Joined:
    Dec 7, 2004
    Messages:
    393
    Likes Received:
    1
    Trophy Points:
    18
    yeah..mailscanner does have very high requirement. Especially if you have lots of emails. With 25 domains, it should be a lot. Once you are used to mailscanner, you will not run a server without it anymore. Definitely, get higher spec system.
     
  17. yawsh

    yawsh Well-Known Member

    Joined:
    Jun 20, 2004
    Messages:
    48
    Likes Received:
    0
    Trophy Points:
    6
    I wonder if there is a script that sends a warning E-mail to the administrator or any kind of notification such as SMS when the load exceeds lets say 5 or 10. So, he could make something b4 loosing the whole server.

    I know it is not solution to the root cause but at least you manage it till you discover the root cause later. :)




    Regards,
    Mohamed
     
  18. oderland

    oderland Well-Known Member
    PartnerNOC

    Joined:
    Dec 30, 2002
    Messages:
    103
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Kungsbacka, Sweden
    install monitor software like bb4 (commercial : www.bb4.com free(?) www.bb4.org)
    We have been using this software for almost 5 years now . You can configure it to send out email/sms. Look at hobbit also it is a extension of bb4- very nice and better way to handle clients.
    We are minitoring almost 40 servers with bb4/hobbit .
     
  19. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    512mb of memory is probably your problem, unless you can actually see the runaway processes you're probably barking up the wrong tree there. MailScanner is pretty memory intensive, you might save a little by cutting down the number of MailScanner threads. I suspect you'll have problems running cpbackup overnight with that amount of memory.

    Once you start swapping you start multiplying out the time programs take to complete by literally thousands of times. Try running "vmstat 5" - if you see anything getting swapped out, or in, that's going to be the cause of your problem - too much process swapping, which I'm almost certain is your problem. The only cure is to run less, or get more memory.

    Something else that helps diagnose runaway processes is the command:
    Code:
    ps -efH | less
    This is a wonderful tool for looking for runaway processes as it groups and displays them by parent - you can see which process is related to which other process really easily, makes it much easier to see what's happening on your machine.
     
  20. d-woo

    d-woo Well-Known Member

    Joined:
    Aug 9, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    6
    Hey Brian

    You are more than correct (is that possible?) :confused:

    I actually migrated to a 2.4GHz Pentium with 1GB RAM....

    I opted not to use MailScanner just yet...although it is a lovely program

    So far...very very very little spam

    SpamAssassin is working quite well in tandem with Rules DuJour

    My average load is like 0.04....wooooo!!! :D
     
Loading...

Share This Page