Help: httpd started heavily, high cpu load, slow ftpupdate, etc

Hi folks,

This is my worst day. I got my box in a lot of trouble. They are:

1. HTTPD is starting heavily, it takes 70-90% CPU and 1-3 minutes to get up. Even, somtimes it can't be started, and ended with message: "Terminated" I didn't press CTRL-C though. Many warnings showed up too when doing "httpd configtest"

2. Slow ftpupdate (vhost passwd sync), and i found many duplicate lines in /etc/proftpd/xxxx_logs

3. Can't do terminate account (a process called "grpck" ate 90% CPU and Running in top), i have fixed this temporarily with "chmod 0 /usr/sbin/grpck" some guys said it's a fedora bug

4. Now the server runs sloooowwly and get halt sometimes (on peak time), so i have to do hard reset

5. When i remove all VirtualHost entry in httpd.conf OR reduce about 300 vhost entry, the server run smoothly.

They are all happen about 1 days after cpanel update The box running smoothly the day before.

The server hosts 1007 accounts. CPU is dual P4 2.8 GHz, 1GB RAM, 2x80 GB IDE, Fedora Core 2

[root@centaur root]# uname -a
Linux centaur.idwebhost.com 2.6.10-1.9_FC2smp #1 SMP Thu Jan 13 18:15:45 EST 2005 i686 i686 i386 GNU/Linux

WHM version, STABLE branch (it was a RELEASE, i turned it to STABLE after the problems appeared): WHM 9.9.9 cPanel 9.9.9-S15 Fedora i686 - WHM X v3.1.0

============== when httpd is going up ==================

top - 05:54:53 up 10 min, 2 users, load average: 7.22, 4.18, 2.15
Tasks: 79 total, 4 running, 75 sleeping, 0 stopped, 0 zombie
Cpu(s): 96.2% us, 3.8% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1025892k total, 295316k used, 730576k free, 36984k buffers
Swap: 2048276k total, 0k used, 2048276k free, 69096k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2105 root 25 0 35856 25m 3512 R 72.7 2.6 0:46.39 httpd
2304 root 25 0 36980 25m 3512 R 62.6 2.6 0:11.39 httpd
2692 root 25 0 7316 4468 1756 R 34.8 0.4 0:01.10 cpanellogd
2023 root 17 0 2392 900 724 R 5.4 0.1 0:08.80 top

==================================================

httpd configtest output:
...... (many warning lines before)
Warning: DocumentRoot [/home/elitii/public_html/chat] does not exist
Warning: DocumentRoot [/home/asvia/public_html/azuwir] does not exist
Warning: DocumentRoot [/home/azuwiri/public_html/bupunsu] does not exist
Warning: DocumentRoot [/home/missing/public_html/parokiskw] does not exist
[Tue Jan 25 06:13:12 2005] [warn] NameVirtualHost 64.247.16.143:80 has no VirtualHosts
[Tue Jan 25 06:13:12 2005] [warn] NameVirtualHost 64.247.16.143:80 has no VirtualHosts
[Tue Jan 25 06:13:12 2005] [warn] NameVirtualHost 64.247.16.143:80 has no VirtualHosts
[Tue Jan 25 06:13:12 2005] [warn] NameVirtualHost 64.247.16.143:80 has no VirtualHosts
[Tue Jan 25 06:13:12 2005] [warn] NameVirtualHost 64.247.16.143:80 has no VirtualHosts
[Tue Jan 25 06:13:12 2005] [warn] NameVirtualHost 64.247.16.143:80 has no VirtualHosts
..... (many more identical warning...)

My httpd.conf size is: [root@centaur root]# stat /etc/httpd/conf/httpd.conf
File: `/etc/httpd/conf/httpd.conf'
Size: 1111425

Lines Count: [root@centaur root]# wc -l /etc/httpd/conf/httpd.conf
38231 /etc/httpd/conf/httpd.conf

 

Still got the same problem

My server hang up 6 times this afternoon, all because load averaga > 15. And httpd, mkvhostpasswd, ftpupdate, are the top processes.

what about you tdreams? have already got a solution?

top - 18:08:44 up 1:42, 3 users, load average: 2.99, 4.05, 3.41
Tasks: 132 total, 5 running, 126 sleeping, 1 stopped, 0 zombie
Cpu0 : 67.4% us, 2.4% sy, 0.0% ni, 29.0% id, 0.6% wa, 0.3% hi, 0.0% si
Cpu1 : 75.6% us, 2.4% sy, 0.0% ni, 22.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1025892k total, 663004k used, 362888k free, 110328k buffers
Swap: 2048276k total, 0k used, 2048276k free, 274732k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16634 root 25 0 35920 26m 3512 R 91.2 2.6 1:05.34 httpd
15476 root 17 0 2940 948 724 R 7.7 0.1 0:41.19 top

 

Most process get on top of "top" especially those which require access to cpanel account database (plaintext or what?)

[root@centaur hartono]# time /scripts/ftpupdate
Ftp Password Files synced
Vhost Passwords synced

real 4m53.390s <-- usually much less than this
user 4m17.342s
sys 0m5.544s

at 1st day, i saw there is total 40000 lines when i exec cat /etc/proftpd/* | wc -l and many of them are identical lines. so i removed them by using array_unique() from PHP. the result is now only about 10000 lines. But it did not fix the problem.

When i log in into user cpanel (using xfercpanel), there is a top process called "cpanel", ate 95% CPU and when i'm viewing the source of blank unfinished cpanel index page in the web browser, the html code stopped in ...manage another account: .... <-- here should be the drop down list of user.

So, i conclude, that my WHM/Cpanel is so strange. especially when processing user/accounting.

One more, when the httpd started, it took about 5 minutes, and ate 90% of CPU too (running state)

My total user is 1001 now. it was 1014 then i termintae unused 13 unused account. What confused me is that 4 days ago, it is running smoothly with 1000 account too.

Is this kernel issue?
My kernel is: Linux centaur.idwebhost.com 2.6.10-1.9_FC2smp #1 SMP Thu Jan 13 18:15:45 EST 2005 i686 i686 i386 GNU/Linux.