high load -- few/normal processes?

[Jorel]

I have a load of 30 but nothing I see in "top" seems to indicate why. Everything seems pretty normal except the processor load. Any ideas? Thanks.


15:37:22 up 30 days, 1:58, 1 user, load average: 36.22, 35.82, 28.30
385 processes: 381 sleeping, 2 running, 1 zombie, 1 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 2.8% 0.0% 0.4% 0.1% 0.0% 96.3% 0.0%
cpu00 3.7% 0.0% 0.5% 0.0% 0.0% 95.6% 0.0%
cpu01 1.9% 0.0% 0.3% 0.3% 0.1% 97.0% 0.0%
Mem: 1026628k av, 1012568k used, 14060k free, 0k shrd, 18588k buff
766400k actv, 145364k in_d, 15692k in_c
Swap: 1052248k av, 1008632k used, 43616k free 148628k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
14021 nobody 15 0 21648 15M 2376 S 0.4 1.5 0:00 1 httpd
18424 mailnull 19 0 3072 2928 1892 S 0.3 0.2 0:00 0 exim
17243 nobody 15 0 21548 14M 2360 S 0.2 1.4 0:00 0 httpd
18327 root 15 0 1436 1436 880 R 0.2 0.1 0:00 0 top
18426 mailnull 16 0 3820 3768 2580 S 0.2 0.3 0:00 0 exim
3075 named 15 0 48584 40M 880 D 0.1 4.0 4:34 0 named
17630 nobody 15 0 19072 12M 2288 D 0.1 1.2 0:00 1 httpd
18267 spoon1 15 0 4392 3100 1004 D 0.1 0.3 0:00 1 cppop
1 root 15 0 372 372 308 S 0.0 0.0 0:36 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 keventd
5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
6 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
9 root 15 0 0 0 0 SW 0.0 0.0 0:06 1 bdflush
7 root 15 0 0 0 0 SW 0.0 0.0 20:49 1 kswapd
8 root 15 0 0 0 0 SW 0.0 0.0 47:46 1 kscand
10 root 15 0 0 0 0 SW 0.0 0.0 2:43 0 kupdated
11 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
18 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_0
19 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_1
20 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_2
21 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_3

[Sheldon]

http://www.chkrootkit.org

go download that see if it detects hidden processes, if so youve been hacked. (well atleast theres a good chance.)

for now kill off anything that you DONT absolutely need running while you look into it. such as httpd and mysql. if you cant gain control, force a reboot then login as soon as it comes back online and shut off apache and mysql.

download chkrootkit, compile it and run it.

-Sheldon

[Sheldon]

oh another thing..

dont rely on top... its useless..

#ps auxf

thats more helpful, that way you can see all the child processes running under each parent.

[Jorel]

I ran chkrootkit (I already had it) and nothing was detected. So I tried a reboot. A few minutes after the reboot, the load went down to about .5 (normal for my 1,000-domain dual opteron). Now it's about at about a 2.0, which I guess is okay. Here is the result of the ps auxf:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 3 0.0 0.0 0 0 ? SW 15:41 0:00 [migration/1]
root 2 0.0 0.0 0 0 ? SW 15:41 0:00 [migration/0]
root 1 0.5 0.0 1532 504 ? S 15:41 0:04 init
root 4 0.0 0.0 0 0 ? SW 15:41 0:00 [keventd]
root 5 0.0 0.0 0 0 ? SWN 15:41 0:00 [ksoftirqd/0]
root 6 0.0 0.0 0 0 ? SWN 15:41 0:00 [ksoftirqd/1]
root 9 0.0 0.0 0 0 ? SW 15:41 0:00 [bdflush]
root 7 0.0 0.0 0 0 ? SW 15:41 0:00 [kswapd]
root 8 0.0 0.0 0 0 ? SW 15:41 0:00 [kscand]
root 10 0.1 0.0 0 0 ? SW 15:41 0:00 [kupdated]
root 11 0.0 0.0 0 0 ? SW 15:41 0:00 [mdrecoveryd]
root 18 0.0 0.0 0 0 ? SW 15:41 0:00 [scsi_eh_0]
root 19 0.0 0.0 0 0 ? SW 15:41 0:00 [scsi_eh_1]
root 20 0.0 0.0 0 0 ? SW 15:41 0:00 [scsi_eh_2]
root 21 0.0 0.0 0 0 ? SW 15:41 0:00 [scsi_eh_3]
root 24 0.1 0.0 0 0 ? DW 15:41 0:01 [kjournald]
root 79 0.0 0.0 0 0 ? SW 15:41 0:00 [khubd]
root 532 0.0 0.0 0 0 ? SW 15:41 0:00 [kjournald]
root 619 0.0 0.0 0 0 ? SW 15:41 0:00 [kjournald]
root 1356 0.0 0.0 1592 580 ? S 15:42 0:00 syslogd -m 0
root 1360 0.0 0.0 1532 460 ? S 15:42 0:00 klogd -x
root 1370 0.0 0.0 1528 440 ? S 15:42 0:00 irqbalance
root 1382 0.0 0.0 83564 600 ? S 15:42 0:00 /sbin/auditd
named 1440 0.3 0.8 52560 8240 ? S 15:42 0:02 /usr/sbin/named -u named
root 1454 0.0 0.1 3668 1548 ? S 15:42 0:00 /usr/sbin/sshd
root 3161 0.0 0.2 7044 2172 ? S 15:49 0:00 \_ sshd: root@pts/0
root 3168 0.0 0.1 4256 1356 pts/0 S 15:49 0:00 \_ -bash
root 5403 0.0 0.0 2744 792 pts/0 R 15:56 0:00 \_ ps auxf
root 1468 0.0 0.0 2148 832 ? S 15:42 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
rcm 5392 0.3 0.1 2540 1028 ? D 15:56 0:00 \_ imapd
root 1486 0.0 0.2 6060 2700 ? S 15:42 0:00 chkservd
root 1547 0.0 0.8 20232 8776 ? S 15:42 0:00 /usr/sbin/clamd
mailnull 1551 0.0 0.1 6592 1836 ? S 15:42 0:00 /usr/sbin/exim -bd -oX 26
mailnull 1557 0.0 0.1 6628 1832 ? S 15:42 0:00 /usr/sbin/exim -bd -q60m
mailnull 5139 0.1 0.4 7716 4344 ? S 15:55 0:00 \_ /usr/sbin/exim -bd -q60m
mailnull 5178 0.2 0.4 7716 4344 ? S 15:55 0:00 \_ /usr/sbin/exim -bd -q60m
mailnull 5366 0.5 0.3 7568 3760 ? D 15:56 0:00 \_ /usr/sbin/exim -bd -q60m
mailnull 1565 0.0 0.1 6588 1796 ? S 15:42 0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
root 1572 0.0 0.1 2932 1404 ? S 15:42 0:00 antirelayd
root 1587 0.0 1.9 21916 20088 ? S 15:42 0:00 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/s
root 1728 0.0 2.0 22528 20808 ? S 15:42 0:00 \_ spamd child
root 1731 0.0 2.0 22408 20628 ? S 15:42 0:00 \_ spamd child
root 1732 0.0 2.0 22420 20684 ? S 15:42 0:00 \_ spamd child
root 1733 0.0 2.0 22556 20816 ? S 15:42 0:00 \_ spamd child
root 1735 0.0 2.0 22280 20552 ? S 15:42 0:00 \_ spamd child
root 1597 0.2 1.8 26480 19112 ? S 15:42 0:02 /usr/local/apache/bin/httpd -DSSL
nobody 1712 0.3 2.3 31164 24224 ? S 15:42 0:02 \_ /usr/local/apache/bin/httpd -DSSL
nobody 1720 0.5 2.3 31312 24368 ? S 15:42 0:04 \_ /usr/local/apache/bin/httpd -DSSL
nobody 2022 0.6 2.3 31520 24492 ? S 15:43 0:05 \_ /usr/local/apache/bin/httpd -DSSL
nobody 3218 0.1 2.1 29036 22056 ? S 15:49 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5152 1.7 2.3 30816 23860 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5153 1.4 2.2 30444 23500 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5154 0.7 2.2 30396 23444 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5155 0.8 2.1 29416 22480 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5156 0.6 2.2 29868 22772 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5160 0.7 2.1 29216 22264 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5161 0.3 2.3 32912 23820 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5164 0.5 2.1 29024 22068 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5168 1.2 2.2 30508 23556 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5169 0.4 2.0 28480 21504 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5170 1.4 2.3 30744 23772 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5181 0.5 2.2 30336 23396 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5232 0.0 1.8 26484 19196 ? S 15:56 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5297 2.4 2.1 29016 22052 ? S 15:56 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5386 4.0 2.1 29068 22084 ? S 15:56 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 5401 0.0 1.8 26480 19120 ? S 15:56 0:00 \_ /usr/local/apache/bin/httpd -DSSL
root 1611 0.0 0.0 1604 648 ? S 15:42 0:00 crond
root 1740 0.0 0.6 10228 6796 ? S 15:42 0:00 cppop - accepting on port 110
root 1763 0.0 0.5 7180 5412 ? SN 15:42 0:00 /usr/bin/perl /usr/local/cpanel/cpanellogd
root 1778 0.0 0.6 8168 6736 ? S 15:42 0:00 cppop - accepting on port 110
nobody 1782 0.0 0.1 3392 2032 ? S 15:42 0:00 entropychat
nobody 1786 0.0 0.0 1728 612 ? S 15:42 0:00 /usr/local/cpanel/bin/startmelange
cpanel 1809 0.0 0.1 3684 1628 ? S 15:42 0:00 /usr/bin/stunnel-4.04local /usr/local/cpanel/etc/stunnel/defau
root 1831 0.0 0.0 1528 488 ? S 15:42 0:00 /usr/sbin/portsentry -tcp
root 1850 0.0 0.0 0 0 ? SW 15:42 0:00 [loop0]
root 1851 0.0 0.7 11596 8156 ? S 15:42 0:00 cpsrvd - waiting for connections
lemonla 5099 0.0 0.8 11984 8408 ? S 15:55 0:00 \_ cpaneld - serving 83.160.19.223
lemonla 5100 0.1 1.3 23708 14200 ? S 15:55 0:00 | \_ /usr/local/cpanel/cpanel ./frontend/x/subdomain/doaddd
root 5104 0.0 0.0 1544 552 ? S 15:55 0:00 | \_ /usr/local/cpanel/bin/domainwrap ADD bbs heart-thr
root 5105 0.3 0.4 5672 4180 ? S 15:55 0:00 | \_ /usr/local/cpanel/bin/domainadmin
root 5134 0.5 0.2 3520 2168 ? D 15:55 0:00 | \_ /usr/bin/perl /scripts/convertemails --qui
cpanel 5387 1.0 0.8 11612 8256 ? S 15:56 0:00 \_ cpaneld - serving 66.169.140.25
cpanel 5388 7.6 1.0 17504 10400 ? S 15:56 0:00 \_ /usr/local/cpanel/3rdparty/bin/php /usr/local/cpanel/b
root 1860 0.0 0.0 0 0 ? SW 15:42 0:00 [kjournald]
root 1866 0.0 0.0 1520 420 tty1 S 15:42 0:00 /sbin/mingetty tty1
root 1867 0.0 0.0 1524 420 tty2 S 15:42 0:00 /sbin/mingetty tty2
root 1868 0.0 0.0 1512 420 tty3 S 15:42 0:00 /sbin/mingetty tty3
root 1869 0.0 0.0 1508 420 tty4 S 15:42 0:00 /sbin/mingetty tty4
root 1870 0.0 0.0 1512 416 tty5 S 15:42 0:00 /sbin/mingetty tty5
root 1871 0.0 0.0 1512 420 tty6 S 15:42 0:00 /sbin/mingetty tty6
root 1921 0.0 0.1 6056 1816 ? S 15:43 0:00 pure-ftpd (SERVER)
lightup 3369 0.0 0.1 6076 1984 ? S 15:50 0:00 \_ pure-ftpd (IDLE)
holliday 4170 0.0 0.1 6076 1992 ? S 15:51 0:00 \_ pure-ftpd (IDLE)
root 1925 0.0 0.0 5588 968 ? S 15:43 0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureau
root 2054 0.0 0.1 4248 1164 ? S 15:43 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-fi
mysql 2075 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
mysql 2077 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
mysql 2078 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
mysql 2079 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
mysql 2080 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
mysql 2081 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
mysql 2084 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
mysql 2085 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --

[Sheldon]

Try turning off uneccesary services in cpanel such as melange and entropychat. I know they can eat server resources like nothing.

Next best thing would be to download and install latest version of rkhunter and run that.

For me to tell you much more id have to sit there and babysit it for awhile to know whats going on. Also make sure you werent running any important high level crons during that time, such as cpanel backups.

-Sheldon

[AndyReed]

There are many variables involved in server high load. To mention a few, you need to track down the resources eater including insecure/bad script and UDP/TCP attaks. Upgrade and then optimize your server services and applications.

[Myacen]

Your using a ton of swap, put some extra ram in the machine and you should be fine.

[chirpy]

Indeed. As Myacen ha said, the server is showing the classic symptoms of memory thrashing, where there's too little physical RAM available for the servers needs. It's evident from the huge swap usage and the high IOWAIT %age. It's possible that you may be able to do some tuning to relieve that load (presuming that nothing has a memory leak) with mysql and apache and spamd which are the usual culprits. If not, then only more RAM will help.

[Jorel]

Thanks to everyone for the advice. I will try to track down unnecisary processes and such. I will also try and scrounge up the cash for a RAM upgrade. I mean, what's the point of a Dual Opteron 244 box when you only have 1GB of RAM? Seems like all the pros get at least 4GB and anywhere up to 16GB of RAM for their Dual Opteron machines. I'm leasing a server though, so unfortunately I don't have the benefit of buying cheap RAM for it on my own.