High CPU loading & troubleshooting.

[SuperBaby]

My server's CPU has been overloaded at 1.0~2.0 for more than 24 hours. I created an outage report and got a few "replies". Unfortunately none of them is useful. I am disappointed that the ISP tech did not look into the problem and simply throw back the ball to me.

I hope someone can help me out. It seems that the overloading is caused by some Perl scripts processing the domain log files. I know rebooting the server might solve the problem for a while but the high loading will come back again as usual.

Under WHM >> Server Status >> CPU/Memory Usage:

Top Process %CPU 95.0 /usr/local/apache/bin/smb -start

What is "smb"? I cd to /usr/local/apache/bin but "smb" does not exist.

Using "top" under shell:
6359 nobody 25 0 2072 2016 500 R 98.9 0.1 2176m 0 perl
30744 root 15 0 1912 416 84 S 0.7 0.0 4:08 0 httpd
12376 root 15 0 1076 1076 800 R 0.1 0.1 0:00 0 top

root@server01 [~]# /usr/sbin/lsof -p 6359

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
perl 6359 nobody cwd DIR 7,0 1024 40001 /var/tmp/...
perl 6359 nobody rtd DIR 3,3 4096 2 /
perl 6359 nobody txt REG 3,3 1003193 31012 /usr/bin/perl
perl 6359 nobody mem REG 3,3 106400 99206 /lib/ld-2.3.2.so
perl 6359 nobody mem REG 3,3 19241 41127 /usr/lib/perl5/5.8.1/i686-linux/auto/IO/IO.so
perl 6359 nobody mem REG 3,3 91624 99235 /lib/libnsl-2.3.2.so
perl 6359 nobody mem REG 3,3 15900 99231 /lib/libdl-2.3.2.so
perl 6359 nobody mem REG 3,3 212020 99211 /lib/tls/libm-2.3.2.so
perl 6359 nobody mem REG 3,3 23688 99229 /lib/libcrypt-2.3.2.so
perl 6359 nobody mem REG 3,3 12716 99257 /lib/libutil-2.3.2.so
perl 6359 nobody mem REG 3,3 26377 41317 /usr/lib/perl5/5.8.1/i686-linux/auto/Socket/Socket.so
perl 6359 nobody mem REG 3,3 52492 99241 /lib/libnss_files-2.3.2.so
perl 6359 nobody mem REG 3,3 18460 99239 /lib/libnss_dns-2.3.2.so
perl 6359 nobody mem REG 3,3 76608 99251 /lib/libresolv-2.3.2.so
perl 6359 nobody mem REG 3,3 1539996 99259 /lib/tls/libc-2.3.2.so
perl 6359 nobody 0r CHR 1,3 15 /dev/null
perl 6359 nobody 1w FIFO 0,5 39558550 pipe
perl 6359 nobody 2w REG 3,3 205050865 77843 /usr/local/apache/logs/error_log
perl 6359 nobody 3u REG 7,0 0 22 /tmp/ZCUDT3gZ5x (deleted)
perl 6359 nobody 4u IPv4 39573226 TCP server01.abcdef.info:35705->211.43.14.121:ircd (ESTABLISHED)
perl 6359 nobody 5w FIFO 0,5 35383801 pipe
perl 6359 nobody 6r FIFO 0,5 35383802 pipe
perl 6359 nobody 7u unix 0xf6412080 39558548 socket
perl 6359 nobody 8r FIFO 0,5 35383803 pipe
perl 6359 nobody 15w REG 3,3 205050865 77843 /usr/local/apache/logs/error_log
perl 6359 nobody 18w REG 3,3 2323 72197 /usr/local/apache/domlogs/aaaaaa.com-bytes_log
perl 6359 nobody 19w REG 3,3 19309 76420 /usr/local/apache/domlogs/bbbbb.org.my-bytes_log
perl 6359 nobody 20w REG 3,3 27901 72209 /usr/local/apache/domlogs/ccccccc.com-bytes_log
perl 6359 nobody 21w REG 3,3 264861 72316 /usr/local/apache/domlogs/dddddd.com-bytes_log
perl 6359 nobody 22w REG 3,3 5948 76751 /usr/local/apache/domlogs/eeeee.info-bytes_log
and other domains' log files ....
...
..
.

[kmpanilla]

figure it out yet? I just had a similar problem and ended up killing the 'smb' processes since they were hogging CPU for the past day.

-k

[HostMerit]

You both need help with server security. Contact me at kris@hostmerit.com, I can help you out - Those are most likely bots / shell processes started via PHP, for now, type ps -u nobody at shell, and anything that isnt httpd, kill -9 the process, then restart httpd. Also suggest mod_security, I can give you a link for that also with a good conf set.

[kmpanilla]

Appears it's the phpBB worm (Perl.Santy) that one of my users did not patch. Patching his copy manually fixed the problem.. What fun.