Zombie Hazard!

[cyberultra]

Hi all, I recently got a new server but some how it's having problem starting from yesterday.

There are a lot of zombie processes and keep increasing. It was once more than 5000 before the kernel was upgraded and rebooted. However after the server reboot the same problem happened again.

Other than that, process "init" is taking 100% of CPU and causing high load.

Server specs is as below:
Dual Xeon 2.80 w/ HT
2GB RAM
CentOS 3.3 (kernel v2.4.29)
cPanel


Here is the copy of "top"

05:19:46 up 54 min, 3 users, load average: 1.09, 1.08, 1.03
414 processes: 98 sleeping, 2 running, 314 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 12.8% 0.0% 94.4% 0.0% 0.0% 0.0% 292.4%
cpu00 11.9% 0.0% 87.1% 0.0% 0.0% 0.0% 0.9%
cpu01 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
cpu02 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
cpu03 0.9% 0.0% 7.3% 0.0% 0.0% 0.0% 91.7%
Mem: 2069208k av, 411020k used, 1658188k free, 0k shrd, 47784k buff
115096k active, 82908k inactive
Swap: 2048276k av, 0k used, 2048276k free 150100k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1 root 15 0 8692 8692 452 R 100.5 0.4 53:11 0 init
4789 root 14 0 1472 1472 912 R 7.3 0.0 0:03 3 top
2 root 9 0 0 0 0 SW 0.0 0.0 0:00 2 keventd
3 root 18 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd_CPU0
4 root 18 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd_CPU1
5 root 19 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd_CPU2
6 root 18 19 0 0 0 SWN 0.0 0.0 0:00 3 ksoftirqd_CPU3
7 root 9 0 0 0 0 SW 0.0 0.0 0:00 2 kswapd
8 root 9 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
9 root 9 0 0 0 0 SW 0.0 0.0 0:00 1 kupdated
10 root 9 0 0 0 0 SW 0.0 0.0 0:01 1 kjournald
519 root 9 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
520 root 9 0 0 0 0 SW 0.0 0.0 0:00 2 kjournald
949 root 9 0 572 572 492 S 0.0 0.0 0:00 0 syslogd
953 root 9 0 472 472 408 S 0.0 0.0 0:00 0 klogd
963 root 9 0 440 440 384 S 0.0 0.0 0:00 1 irqbalance
992 root 9 0 404 404 348 S 0.0 0.0 0:00 3 mdadm
1056 nobody 9 0 2128 2128 1944 S 0.0 0.1 0:00 0 proftpd
1069 root 9 0 1912 1912 1420 S 0.0 0.0 0:00 2 cupsd
1128 root 8 0 1572 1572 1432 S 0.0 0.0 0:00 3 sshd
1142 root 9 0 836 836 724 S 0.0 0.0 0:00 3 xinetd
1160 root 8 0 2712 2712 1428 S 0.0 0.1 0:00 1 chkservd
1220 mailnull 8 0 1908 1908 1568 S 0.0 0.0 0:00 0 exim
1225 mailnull 9 0 1896 1896 1560 S 0.0 0.0 0:00 0 exim
1229 mailnull 9 0 1872 1872 1536 S 0.0 0.0 0:00 0 exim
1234 root 9 0 1424 1424 1148 S 0.0 0.0 0:00 1 antirelayd
1249 root 9 0 2180 2180 1884 S 0.0 0.1 0:01 3 sshd
1259 root 9 0 1356 1356 1132 S 0.0 0.0 0:00 0 bash
1309 root 9 0 1428 1428 896 S 0.0 0.0 0:43 2 top

Any help would be much appreciated! Thanks in advance!

[K_aneda]

Dosen't show which processes are going zombie, could you paste a full ps auxw ?

[cyberultra]

I typed "ps -aux | grep Z", and below is part of what I got

mailnull 11201 0.0 0.0 0 0 ? Z 07:13 0:00 [exim <defunct>]
root 11222 0.0 0.0 0 0 ? Z 07:14 0:00 [mysql <defunct>]
root 11226 0.0 0.0 0 0 ? Z 07:14 0:00 [spamc <defunct>]
mailnull 11230 0.0 0.0 0 0 ? Z 07:14 0:00 [exim <defunct>]
root 10533 0.0 0.0 0 0 ? Z 07:01 0:00 [whostmgr2 <defunct>]
mailnull 6991 0.0 0.0 0 0 ? Z 06:09 0:00 [exim <defunct>]
mailnull 6999 0.0 0.0 0 0 ? Z 06:09 0:00 [exim <defunct>]
mailnull 7001 0.0 0.0 0 0 ? Z 06:09 0:00 [exim <defunct>]
mailnull 7017 0.0 0.0 0 0 ? Z 06:09 0:00 [exim <defunct>]
mailnull 7019 0.0 0.0 0 0 ? Z 06:09 0:00 [exim <defunct>]

90% of the zombie processes are
"mailnull 7019 0.0 0.0 0 0 ? Z 06:09 0:00 [exim <defunct>]"

Anyone have any idea why process "init" is taking up that high load?

Thanks.

[K_aneda]

You getting a lot of e-mail lately (abnormal traffic) ?

[cyberultra]

You getting a lot of e-mail lately (abnormal traffic) ?

No, I guess the problem is for some reason zombie processes are not cleared and stack it up. As you can see there are a lot of zombie processes, normally is less than 5 or none.

[K_aneda]

Yeah but not as many as you've stated.... and especially within the first hour of the machine being 'up'.

[Donna]

Look for a file called links.cgi on your server. I had this issue with one of my servers and it turned out someone had used an insecure phplinks install to put a cgi with spam software on the server.

They were literally using it to send thousands of spam emails per minute to the same account on the server (the one they'd gotten into) and it kept giving me a lot of exim zombies like you have.

I found the file in question inside a "links" folder in his cgi-bin

tail -f /var/log/exim_mainlog

will give you an idea of the mail accounts being hit, I'd start my search with that persons domain account.