High Server Load

[d-woo]

PROBLEM - High Load
************************************************
Loads have been around 9.0 and average around 4.0

I have plenty of SWAP, just high server loads

I added Mail Scanner last week. The loads were high before then, I just notice them more now.

I want to keep MailScanner (awesome program).

I don't mind getting more RAM if needed, or upgrading to a Pentium Processor if needed.

I just want to be certain that I'm doing all that I can do to run the server efficiently.


Server Specs
************************************************
Processor - 1.7 Celeron
RAM - 512
Websites - 25

10 use MySQL databases with messageboards with persistent connections



DATA
************************************************
WHM Memory Usage
yeilds:

11-19-2006
User.........Domain...............%CPU............%MEM..........Mysql Processes
root..............................22.74............3.78.........2.0
Top Process..%CPU 66.5............gzip
Top Process..%CPU 56.0............pkgacct - americae
Top Process..%CPU 55.8............gzip
mailnull..........................15.92............16.35........0.0
Top Process..%CPU 73.0............MailScanner: checking with Spam Lists
Top Process..%CPU 72.0............MailScanner: checking with SpamAssassin
Top Process..%CPU 53.0............MailScanner: virus scanning

TOP snapshot

 12:25pm  up 3 days,  1:42,  1 user,  load average: 3.67, 2.43, 3.38
140 processes: 133 sleeping, 4 running, 2 zombie, 1 stopped
CPU states:  2.4% user, 18.8% system, 65.3% nice, 13.3% idle
Mem:   506048K av,  477336K used,   28712K free,       0K shrd,   23824K buff
Swap: 2048276K av,  477912K used, 1570364K free                   79948K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
22559 mailnull  16   8     0    0     0 Z N  68.0  0.0   0:01 MailScanner <defu
18829 mailnull  13   8 74708  13M  1968 D N   4.4  2.8   0:19 MailScanner: wait
22560 root      13   8  3108 3108  2240 S N   3.9  0.6   0:00 /usr/sbin/exim -C
22562 oakheirl  13   8  6444 6444  3424 D N   3.4  1.2   0:00 /usr/local/cpanel
22558 mailnull   9   0  2728 2652  1876 D     2.9  0.5   0:00 /usr/sbin/exim -b
   15 root       9   0     0    0     0 DW    0.4  0.0   1:54 kjournald
19290 nobody     9   0  9288 5524  1892 S     0.4  1.0   0:03 /usr/local/apache
22499 mailnull  13   8 72664  31M  1892 D N   0.4  6.3   0:00 MailScanner: viru
22556 root      10   0  1080 1080   824 R     0.4  0.2   0:00 top
22561 oakheirl  13   8  3148 3148  2272 S N   0.4  0.6   0:00 /usr/sbin/exim -C
    1 root       8   0   160  116    92 S     0.0  0.0   0:07 init [3]
    2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd
    3 root       9   0     0    0     0 SW    0.0  0.0   0:00 kapmd

Top Memory Consuming Processes

ps aux | head -1;ps aux --no-headers| sort -rn +3 | head
yields:

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
oakheirl 11275  0.0  6.4 70780 32852 ?       TN   07:38   0:10 /usr/local/cpanel
mysql     5407  0.2  5.9 79060 30160 ?       SN   Nov17  10:14 /usr/sbin/mysqld
mysql     5369  0.2  5.9 79060 30160 ?       SN   Nov17  10:07 /usr/sbin/mysqld
mysql     3681  0.2  5.9 79060 30160 ?       SN   Nov17   8:52 /usr/sbin/mysqld
mysql     3217  0.2  5.9 79060 30160 ?       SN   Nov17   9:05 /usr/sbin/mysqld
mysql     3210  0.2  5.9 79060 30160 ?       SN   Nov17  10:09 /usr/sbin/mysqld
mysql     3159  0.2  5.9 79060 30160 ?       SN   Nov17  10:39 /usr/sbin/mysqld
mysql     3001  0.0  5.9 79060 30160 ?       SN   Nov17   0:17 /usr/sbin/mysqld
mysql     2998  0.0  5.9 79060 30160 ?       SN   Nov17   0:00 /usr/sbin/mysqld
mysql     2928  0.0  5.9 79060 30160 ?       SN   Nov17   0:05 /usr/sbin/mysqld

Top CPU Consuming Processes

ps aux | head -1;ps aux --no-headers | sort -rn +2
yields:

[email protected] [~]# ps aux | head -1;ps aux --no-headers | sort -rn +2
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
mailnull 22707  2.4  0.0     0    0 ?        ZN   12:26   0:01 [MailScanner <def
mailnull 22530  1.4  0.0     0    0 ?        ZN   12:25   0:01 [MailScanner <def
mailnull 18829  1.4  2.5 78036 13036 ?       SN   12:03   0:20 MailScanner: wait
mailnull 18784  1.3  2.2 76232 11572 ?       SN   12:02   0:19 MailScanner: wait
mailnull 18824  1.1  2.2 76340 11580 ?       SN   12:03   0:16 MailScanner: wait
mailnull 22743  1.0  0.2  4984 1264 ?        S    12:27   0:00 /usr/sbin/exim -b
root     22260  0.4  0.3  3532 1700 ?        S    12:24   0:00 /usr/sbin/sshd
nobody   19291  0.4  1.3 17812 6988 ?        S    12:06   0:06 /usr/local/apache
nobody   19308  0.3  1.2 17268 6276 ?        S    12:06   0:03 /usr/local/apache
.
.
.
cpanel   24359  0.0  0.1  2640  700 ?        SN   09:05   0:00 /usr/bin/stunnel-
[email protected] [~]#

 

[HelloAdam]

Hey,

Looks like you don't have a good server for that many websites. Yes I know 25 websites but with your server specs it alot. However what you can do is not install so many high programs.

If you do a

top

command in SSH. It will show you the current server load. gzip is used with backing up cpanel accounts. I recommend that you use something like Incermetal Backups (spelled that wrong.) Its one of the cpanel backup options. It only backups what has changed. But does not put it into a .tar.gz format. So more space is need.

Other then that you suggest you get a bigger server.

You can get Server Management at http://www.eastsolid.com/services/management.php..

From,
Adam

 

[hosttosucceed]

Perhaps take off the persistent connections as they aren't recommended for a shared hosting solution.

 

[AndyReed]

Loads have been around 9.0 and average around 4.0

I have plenty of SWAP, just high server loads

I added Mail Scanner last week. The loads were high before then, I just notice them more now.

Make sure you don't have a bad/insecure script that loops indefinitely, exhausting more of your server resources. This is one of the many variables involved in server high load. I suggest you read this article about the possible causes of server high load: http://www.servertune.com/kbase/security/server_load.html

 

[forlinuxsupport]

Hi

What operating system are you using ?
See if there is a new kernel update for it.


Try do this :
vmstat 3

let that run for a few minutes and post results here.

Mailscanner seems to be using lots of cpu time. if you tail the mail logs, is there alot of mail going in and out ?
Disable any catchall email addresses you have as they just attract spam.

That gzip process is probably the cpanel backup stil lrunning, that pushes up load.
Try kill it and its children (kill -9 PID)

I often have the cpanel backup Stuck.

Update cpanel to the latest CURRENT version and do a force update (/scripts/upcp --force )

Regards
Andy

 

[forlinuxsupport]

us

ps fax
look for the scripts there. - probably hanging off http / apache or maybe cron).

they probably have a HIGH execution time !!

Cheers
andy

 

[d-woo]

Thanks for the input guys.

I believe that one client (the only one who hosts on my box other than me) has some runaway scripts.

How can I prove it other than disabling them?

Is there a specific log file I can check?

 

[d-woo]

Hi

What operating system are you using ?
See if there is a new kernel update for it.


Try do this :
vmstat 3

let that run for a few minutes and post results here.


Regards
Andy

I'm running:
WHM 10.8.0 cPanel 10.9.0-S77
RedHat 7.3 i686 - WHM X v3.1.0

Output from vmstat2
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
3 6 1 500088 53548 12816 70748 492 0 504 89 249 483 39 6 55
2 7 1 500088 50360 12916 70876 439 0 488 87 232 329 5 1 94
0 10 0 500088 47960 13028 70876 523 0 523 83 273 365 2 1 96
0 11 1 500088 40648 13088 70908 472 0 484 211 270 405 6 6 87
2 9 1 500088 34240 13192 70964 488 0 508 119 267 368 2 2 97
1 8 1 500088 24072 13296 71200 432 0 515 140 239 429 9 8 82
1 9 1 500088 21872 13380 71260 501 0 524 96 279 352 5 3 91
1 9 1 500088 18696 13472 71264 505 0 507 81 257 346 29 4 66
1 10 1 500088 15240 13572 71268 520 1 523 72 255 354 40 4 56
0 11 1 500096 12012 13680 71276 496 7 503 95 257 514 4 3 93
1 10 1 500576 8660 13756 71288 476 180 476 292 253 326 3 4 93
0 14 1 500932 6128 13800 71344 424 149 436 272 252 315 2 3 95
2 8 1 501124 5292 13816 72544 288 71 836 173 316 447 2 2 96
1 11 1 501200 5440 13860 72040 272 519 275 595 290 272 4 2 93
3 10 1 501260 5204 13912 71976 351 280 352 357 277 295 1 2 97
1 14 1 501304 5464 14008 71848 113 411 172 448 334 201 3 3 95
2 13 1 501332 5424 14044 71904 215 435 309 551 333 227 2 2 96
0 16 1 501504 5420 14064 72240 156 641 336 724 369 225 5 3 91
4 11 2 501896 5328 14112 71908 335 536 528 639 351 340 4 3 93
1 13 1 501964 5408 14152 71556 237 135 415 223 322 276 1 1 98
1 13 1 502220 5308 14144 71804 349 169 547 231 342 347 4 3 94
procs memory swap io system cpu

 

[d-woo]

Operating system Linux
Service Status Click to View
Kernel version 2.4.20-28.7
Machine Type i686
Apache version 1.3.36 (Unix)
PERL version 5.8.7
Path to PERL /usr/bin/perl
Path to sendmail /usr/sbin/sendmail
Installed Perl Modules Click to View
PHP version 4.4.2
MySQL version 4.0.25-standard
cPanel Build 10.9.0-STABLE 77
Theme cPanel X v2.6.0
Documentation Click to View
cPanel Pro 1.0 (RC1)


No catchall email addresses are running.

I turned off gzip backup via WHM. That is, however, still running and I have to watch top to kill the processes. Should I just reboot?

 

[d-woo]

Just found this:

21688 nobody 16 4 17460 1148 1076 R N 97.1 0.2 7:17 /usr/local/apache/bin/httpd -DSSL

What is /usr/local/apache/bin/httpd -DSSL ?

 

[Fredi]

I have a similar problem.

29907 nobody 0 5.0 0.3 /usr/local/apache/bin/httpd -DSSL
9718 nobody 0 4.6 0.5 /usr/local/apache/bin/httpd -DSSL
9781 nobody 0 3.3 0.5 /usr/local/apache/bin/httpd -DSSL and other.......

 

[freedog96150]

Just found this:

21688 nobody 16 4 17460 1148 1076 R N 97.1 0.2 7:17 /usr/local/apache/bin/httpd -DSSL

What is /usr/local/apache/bin/httpd -DSSL ?

That is your Apache daemon.

 

[d-woo]

I'm at my wits end.

I removed all .cgi scripts and a few PHP email test scripts that were on my box... then rebooted the server.

The load stayed below 1.5 for over 24 hours then started back up to 12 and 13.

I then turned off virus scan to lower the load (keeping Mail Scan for Spam on)

I see no difference in load. Still 12 and 13.

 

[ramprage]

I think your problem here is MailScanner itself, it looks like its responsbile for causing the high server load on your system.

It's possible a users account/script may be causing this but you really need to hire someone to review your server and optimze/secure it. From your posts you're running very old versions of software which can lead to hacks and root on your server.

 

[d-woo]

I agree.

I'm in the process of getting a new box altogether and transferring everything over.

Thanks!

 

[abubin]

yeah..mailscanner does have very high requirement. Especially if you have lots of emails. With 25 domains, it should be a lot. Once you are used to mailscanner, you will not run a server without it anymore. Definitely, get higher spec system.

 

[yawsh]

I wonder if there is a script that sends a warning E-mail to the administrator or any kind of notification such as SMS when the load exceeds lets say 5 or 10. So, he could make something b4 loosing the whole server.

I know it is not solution to the root cause but at least you manage it till you discover the root cause later.




Regards,
Mohamed

 

[oderland]

I wonder if there is a script that sends a warning E-mail to the administrator or any kind of notification such as SMS when the load exceeds lets say 5 or 10. So, he could make something b4 loosing the whole server.

I know it is not solution to the root cause but at least you manage it till you discover the root cause later.




Regards,
Mohamed

install monitor software like bb4 (commercial : www.bb4.com free(?) www.bb4.org)
We have been using this software for almost 5 years now . You can configure it to send out email/sms. Look at hobbit also it is a extension of bb4- very nice and better way to handle clients.
We are minitoring almost 40 servers with bb4/hobbit .

 

[brianoz]

512mb of memory is probably your problem, unless you can actually see the runaway processes you're probably barking up the wrong tree there. MailScanner is pretty memory intensive, you might save a little by cutting down the number of MailScanner threads. I suspect you'll have problems running cpbackup overnight with that amount of memory.

Once you start swapping you start multiplying out the time programs take to complete by literally thousands of times. Try running "vmstat 5" - if you see anything getting swapped out, or in, that's going to be the cause of your problem - too much process swapping, which I'm almost certain is your problem. The only cure is to run less, or get more memory.

Something else that helps diagnose runaway processes is the command:

ps -efH | less

This is a wonderful tool for looking for runaway processes as it groups and displays them by parent - you can see which process is related to which other process really easily, makes it much easier to see what's happening on your machine.

 

[d-woo]

Hey Brian

You are more than correct (is that possible?)

I actually migrated to a 2.4GHz Pentium with 1GB RAM....

I opted not to use MailScanner just yet...although it is a lovely program

So far...very very very little spam

SpamAssassin is working quite well in tandem with Rules DuJour

My average load is like 0.04....wooooo!!! :D