274% CPU Usage (load 4.x) ???

[innsites]

This server is making me miserable. Note that it was recently optmized (Thx Chirpy)

How should this be best managed? Sometimes IOWAIT = 80%!!

The server load script just sent me this email:

Your server appears to have a high load. (WHICH IS ACTUALLY QUITE LOW THAN IF MAILMAN WAS ENABLED!)

The load averages are:
1 minute: 4.79 (4.79 per CPU)
5 minutes: 4.01 (4.01 per CPU)
15 minutes: 4.28 (4.28 per CPU)

Here is what was just recorded as the current load.

- The process ID #28381 under the user nobody is using 81.9% CPU and 2.0% Memory with the command 0 /usr/local/ap which has used 0 minutes and 05 seconds of CPU time.

- The process ID #26694 under the user nobody is using 79.1% CPU and 2.1% Memory with the command 0 /usr/local/ap which has used 0 minutes and 06 seconds of CPU time.


#####################################

The top 10 processes are using a total of 273.5% CPU and a total of 39.6% Memory. The process list will follow.

Here are the top 10 processes:

- The process ID #28381 under the user nobody is using 81.9% CPU and 2.0% Memory with the command 0 /usr/local/ap which has used 0 minutes and 05 seconds of CPU time.

- The process ID #26694 under the user nobody is using 79.1% CPU and 2.1% Memory with the command 0 /usr/local/ap which has used 0 minutes and 06 seconds of CPU time.

- The process ID #384 under the user nobody is using 77.2% CPU and 1.5% Memory with the command 0 /usr/local/ap which has used 0 minutes and 03 seconds of CPU time.

- The process ID #4850 under the user mysql is using 9.4% CPU and 4.4% Memory with the command 0 /usr/sbin/mys which has used 0 minutes and 34 seconds of CPU time.

- The process ID #6670 under the user mysql is using 8.4% CPU and 4.4% Memory with the command 0 /usr/sbin/mys which has used 0 minutes and 36 seconds of CPU time.

- The process ID #3093 under the user mysql is using 6.5% CPU and 4.4% Memory with the command 0 /usr/sbin/mys which has used 0 minutes and 23 seconds of CPU time.

- The process ID #23800 under the user mailnull is using 5.6% CPU and 6.0% Memory with the command 0 MailScanner: which has used 1 minutes and 34 seconds of CPU time.

- The process ID #4125 under the user mysql is using 1.8% CPU and 4.4% Memory with the command 0 /usr/sbin/mys which has used 0 minutes and 33 seconds of CPU time.

- The process ID #6687 under the user mysql is using 1.8% CPU and 4.4% Memory with the command 0 /usr/sbin/mys which has used 0 minutes and 30 seconds of CPU time.

- The process ID #1353 under the user mailnull is using 1.8% CPU and 6.0% Memory with the command 0 MailScanner: which has used 0 minutes and 00 seconds of CPU time.

TOP OUTPUT TO FOLLOW

 

[innsites]

Top Output #1

09:27:06 up 5:40, 1 user, load average: 4.79, 4.01, 4.28
142 processes: 137 sleeping, 4 running, 1 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 94.3% 0.0% 2.8% 0.0% 0.9% 1.8% 0.0%
Mem: 1019104k av, 948160k used, 70944k free, 0k shrd, 245280k buff
716032k actv, 144300k in_d, 14564k in_c
Swap: 2040244k av, 130408k used, 1909836k free 237532k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
28381 nobody 16 0 22772 20M 4564 S 81.9 2.0 0:05 0 /usr/local/ap
26694 nobody 16 0 24252 21M 4608 S 79.1 2.1 0:06 0 /usr/local/ap
384 nobody 16 0 18396 15M 4572 S 77.2 1.5 0:03 0 /usr/local/ap
4850 mysql 15 0 99.7M 44M 2292 S 9.4 4.4 0:34 0 /usr/sbin/mys
6670 mysql 15 0 99.7M 44M 2292 S 8.4 4.4 0:36 0 /usr/sbin/mys
3093 mysql 15 0 99.7M 44M 2292 S 6.5 4.4 0:23 0 /usr/sbin/mys
23800 mailnull 18 0 68324 60M 2608 S 5.6 6.0 1:34 0 MailScanner:
4125 mysql 15 0 99.7M 44M 2292 S 1.8 4.4 0:33 0 /usr/sbin/mys
6687 mysql 15 0 99.7M 44M 2292 S 1.8 4.4 0:30 0 /usr/sbin/mys
1353 mailnull 22 0 68356 60M 2640 R 1.8 6.0 0:00 0 MailScanner:
1354 root 24 0 3068 3068 2572 R 1.8 0.3 0:00 0 /usr/sbin/exi
2777 named 15 0 32216 30M 1516 S 0.9 3.0 2:20 0 /usr/sbin/nam
23615 mailnull 19 0 1936 1924 1576 S 0.9 0.1 0:04 0 /usr/sbin/exi
27663 nobody 15 0 24408 21M 4588 S 0.9 2.1 0:09 0 /usr/local/ap
28605 nobody 15 0 22528 19M 4788 S 0.9 1.9 0:01 0 /usr/local/ap
30134 nobody 15 0 23256 20M 4572 S 0.9 2.0 0:02 0 /usr/local/ap
1294 nobody 15 0 15144 12M 4016 S 0.9 1.2 0:00 0 /usr/local/ap
1332 mailnull 22 0 3752 3744 2628 S 0.9 0.3 0:00 0 /usr/sbin/exi
1 root 15 0 396 348 324 S 0.0 0.0 0:03 0 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd
4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
7 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
5 root 15 0 0 0 0 SW 0.0 0.0 0:09 0 kswapd
6 root 15 0 0 0 0 SW 0.0 0.0 0:19 0 kscand
8 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
16 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_0
17 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_1
20 root 15 0 0 0 0 SW 0.0 0.0 0:17 0 kjournald
72 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
1707 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
1709 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 loop0
1710 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
2709 root 15 0 476 452 384 R 0.0 0.0 0:02 0 syslogd -m 0
2713 root 15 0 384 364 312 S 0.0 0.0 0:00 0 klogd -x
2776 named 25 0 32216 30M 1516 S 0.0 3.0 0:00 0 /usr/sbin/nam
2778 named 15 0 32216 30M 1516 S 0.0 3.0 0:00 0 /usr/sbin/nam
2779 named 15 0 32216 30M 1516 S 0.0 3.0 0:47 0 /usr/sbin/nam
2791 root 15 0 636 392 388 S 0.0 0.0 0:00 0 /usr/sbin/ssh
2814 root 25 0 520 368 364 S 0.0 0.0 0:00 0 /bin/sh /usr/
2844 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
2847 mysql 25 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
2848 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
2849 mysql 25 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
2850 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
2851 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
2852 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
2853 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
2856 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:00 0 /usr/sbin/mys
3733 root 15 0 960 672 576 S 0.0 0.0 0:00 0 sshd: admin [
3857 root 15 0 804 588 508 S 0.0 0.0 0:00 0 pure-ftpd (SE
3865 root 15 0 584 436 392 S 0.0 0.0 0:00 0 /usr/sbin/pur
3891 admin 15 0 1144 848 692 S 0.0 0.0 0:01 0 sshd: [email protected]
3893 admin 23 0 572 368 364 S 0.0 0.0 0:00 0 -bash
3943 root 21 0 560 384 380 S 0.0 0.0 0:00 0 su -
3958 root 15 0 1064 1004 808 S 0.0 0.0 0:00 0 -bash
4023 root 15 0 15024 12M 3916 S 0.0 1.2 0:02 0 /usr/local/ap
4569 mysql 15 0 99.7M 44M 2292 D 0.0 4.4 0:10 0 /usr/sbin/mys
4794 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:34 0 /usr/sbin/mys
4840 cpanel 15 0 1984 1720 1056 S 0.0 0.1 0:00 0 /usr/bin/stun
4851 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:11 0 /usr/sbin/mys
5006 root 21 0 336 268 264 S 0.0 0.0 0:00 0 /sbin/mingett
5007 root 21 0 336 268 264 S 0.0 0.0 0:00 0 /sbin/mingett
5008 root 21 0 332 268 264 S 0.0 0.0 0:00 0 /sbin/mingett
5009 root 21 0 332 268 264 S 0.0 0.0 0:00 0 /sbin/mingett
5010 root 21 0 332 268 264 S 0.0 0.0 0:00 0 /sbin/mingett
5011 root 21 0 328 268 264 S 0.0 0.0 0:00 0 /sbin/mingett
6667 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:30 0 /usr/sbin/mys
13648 mysql 15 0 99.7M 44M 2292 S 0.0 4.4 0:34 0 /usr/sbin/mys
8746 root 15 0 832 780 696 S 0.0 0.0 0:00 0 xinetd -staya
8831 mailnull 15 0 5152 4052 1460 S 0.0 0.3 0:00 0 eximstats
8856 root 34 19 10376 8848 1636 S N 0.0 0.8 0:12 0 cpanellogd -

 

[innsites]

Top Output (continued)

23757 mailnull 15 0 17128 5768 1688 S 0.0 0.5 0:00 0 MailScanner:
23758 mailnull 15 0 69608 59M 2624 S 0.0 6.0 1:31 0 MailScanner:
23765 mailnull 15 0 18244 8464 1848 S 0.0 0.8 0:00 0 MailWatch SQL
24014 root 15 0 6724 6256 1256 S 0.0 0.6 0:01 0 cpsrvd - wait
4830 root 22 0 8364 6692 1576 S 0.0 0.6 0:00 0 chkservd
5347 root 15 0 5500 2676 1484 S 0.0 0.2 3:11 0 lfd - sleepin
8158 root 15 0 1032 1028 652 S 0.0 0.1 0:00 0 crond
23610 mailnull 15 0 1924 1668 1572 S 0.0 0.1 0:00 0 /usr/sbin/exi
23622 mailnull 15 0 1916 1916 1568 S 0.0 0.1 0:00 0 /usr/sbin/exi
23628 mailnull 25 0 1904 1904 1556 S 0.0 0.1 0:00 0 /usr/sbin/exi
23634 root 15 0 1500 1500 1180 S 0.0 0.1 0:00 0 antirelayd
6197 root 21 0 3480 3480 1588 S 0.0 0.3 0:00 0 /usr/bin/perl
19698 nobody 15 0 22264 19M 4592 S 0.0 1.9 0:02 0 /usr/local/ap
25132 mailman 24 0 5940 5936 2552 S 0.0 0.5 0:00 0 /usr/local/bi
25135 mailman 15 0 6544 5160 2580 S 0.0 0.5 0:00 0 /usr/local/bi
25136 mailman 15 0 5860 5856 2480 S 0.0 0.5 0:00 0 /usr/local/bi
25137 mailman 15 0 5852 4452 2480 S 0.0 0.4 0:00 0 /usr/local/bi
25138 mailman 15 0 8400 6808 2572 S 0.0 0.6 0:00 0 /usr/local/bi
25139 mailman 15 0 5880 5332 2480 S 0.0 0.5 0:00 0 /usr/local/bi
25140 mailman 15 0 9520 7564 2628 S 0.0 0.7 0:09 0 /usr/local/bi
25141 mailman 15 0 8308 5376 2548 S 0.0 0.5 0:00 0 /usr/local/bi
25142 mailman 15 0 7860 6708 2544 S 0.0 0.6 0:00 0 /usr/local/bi
25491 root 15 0 8096 6928 3580 S 0.0 0.6 0:00 0 cppop - accep
25948 nobody 15 0 24372 21M 4592 S 0.0 2.1 0:09 0 /usr/local/ap
26343 nobody 15 0 24312 21M 4508 S 0.0 2.1 0:02 0 /usr/local/ap
26346 nobody 15 0 24132 21M 4596 S 0.0 2.1 0:01 0 /usr/local/ap
26347 nobody 15 0 24536 20M 4604 S 0.0 2.1 0:04 0 /usr/local/ap
27552 nobody 15 0 24204 21M 4580 S 0.0 2.1 0:02 0 /usr/local/ap
28603 nobody 15 0 18732 16M 4472 S 0.0 1.6 0:00 0 /usr/local/ap
29096 mailnull 15 0 3796 3788 2676 S 0.0 0.3 0:00 0 /usr/sbin/exi
29755 mailnull 20 0 2148 2136 1752 S 0.0 0.2 0:00 0 /usr/sbin/exi
29869 nobody 16 0 24072 21M 4488 S 0.0 2.1 0:01 0 /usr/local/ap
29930 mailnull 15 0 3800 3792 2676 S 0.0 0.3 0:00 0 /usr/sbin/exi
30012 mailnull 15 0 3784 3776 2668 S 0.0 0.3 0:00 0 /usr/sbin/exi
30935 mailnull 15 0 3788 3780 2656 S 0.0 0.3 0:00 0 /usr/sbin/exi
31134 mailnull 15 0 2164 2152 1760 S 0.0 0.2 0:00 0 /usr/sbin/exi
32324 mailnull 15 0 2148 2136 1752 S 0.0 0.2 0:00 0 /usr/sbin/exi
32664 mailnull 15 0 3796 3788 2676 S 0.0 0.3 0:00 0 /usr/sbin/exi
1007 mailnull 15 0 3800 3792 2676 S 0.0 0.3 0:00 0 /usr/sbin/exi
1024 nobody 15 0 15144 12M 4016 S 0.0 1.2 0:00 0 /usr/local/ap
1101 root 20 0 7048 6772 1492 S 0.0 0.6 0:00 0 whostmgrd - s
1182 mailnull 20 0 2164 2152 1760 S 0.0 0.2 0:00 0 /usr/sbin/exi
1292 nobody 16 0 16740 14M 4336 S 0.0 1.4 0:00 0 /usr/local/ap
1295 nobody 17 0 15144 12M 4016 S 0.0 1.2 0:00 0 /usr/local/ap
1308 mailnull 22 0 0 0 0 Z 0.0 0.0 0:00 0 MailScanner <
1311 nobody 18 0 15144 12M 4016 S 0.0 1.2 0:00 0 /usr/local/ap
1312 nobody 17 0 16772 14M 4332 S 0.0 1.4 0:00 0 /usr/local/ap
1313 nobody 17 0 15144 12M 4016 S 0.0 1.2 0:00 0 /usr/local/ap
1314 nobody 17 0 15144 12M 4016 S 0.0 1.2 0:00 0 /usr/local/ap
1317 mailnull 19 0 2148 2136 1752 S 0.0 0.2 0:00 0 /usr/sbin/exi
1319 mailnull 19 0 2028 2016 1656 S 0.0 0.1 0:00 0 /usr/sbin/exi
1329 mailnull 20 0 2036 2024 1656 S 0.0 0.1 0:00 0 /usr/sbin/exi
1331 mailnull 21 0 2024 2012 1652 S 0.0 0.1 0:00 0 /usr/sbin/exi
1333 mailnull 21 0 2164 2152 1760 S 0.0 0.2 0:00 0 /usr/sbin/exi
1335 root 15 0 1248 1248 808 S 0.0 0.1 0:00 0 crond
1338 root 15 0 1312 1312 1032 S 0.0 0.1 0:00 0 /usr/bin/perl
1341 root 24 0 1072 1072 788 R 0.0 0.1 0:00 0 top -n1 -bc
1342 nobody 19 0 15632 13M 4292 S 0.0 1.3 0:00 0 /usr/local/ap
1344 mailnull 21 0 2016 2004 1644 S 0.0 0.1 0:00 0 /usr/sbin/exi
1345 mailnull 19 0 1952 1940 1592 S 0.0 0.1 0:00 0 /usr/sbin/exi
1346 mailnull 19 0 1952 1940 1592 S 0.0 0.1 0:00 0 /usr/sbin/exi
1348 mailnull 22 0 2016 2004 1644 S 0.0 0.1 0:00 0 /usr/sbin/exi
1349 mailnull 22 0 2016 2004 1644 S 0.0 0.1 0:00 0 /usr/sbin/exi
1350 mailnull 22 0 2016 2004 1644 S 0.0 0.1 0:00 0 /usr/sbin/exi
1351 mailnull 19 0 1952 1940 1592 S 0.0 0.1 0:00 0 /usr/sbin/exi
1352 mailnull 22 0 2016 2004 1644 S 0.0 0.1 0:00 0 /usr/sbin/exi
1355 mailnull 19 0 1952 1940 1592 S 0.0 0.1 0:00 0 /usr/sbin/exi
1356 mailnull 19 0 1952 1940 1592 S 0.0 0.1 0:00 0 /usr/sbin/exi
1357 mailnull 19 0 1952 1940 1592 S 0.0 0.1 0:00 0 /usr/sbin/exi

 

[Manuel_accu]

Hi,

First let us know what is the process 'ap' and why it is taking too much load on the server,

Yes, iowait is more than 80% means your disk IO is not responding to the server processes I have also found high swap usage ,

First I suggest you to upgrade the RAM on this server or provide the output of below mentioend commands:

# netstat -ntu | awk '{print $5}'| cut -d: -f1 | sort | uniq -c | sort -nr | more

# ps aux | head -1; ps aux | sort -rn +2 | head -10

And I suggest you to check the below mentioned URL and then check your server for any diagnosis

http://forums.linuxwebadmin.info/index.php/topic,51.0.html

Thanks,