HELP HELP!!!! Server Wide cPanel/WHM/Apache/MySQL Problems :(

mOdY

Well-Known Member
Dec 25, 2004
80
0
156
Hello all,

I have massive flood of problems from one of my VPS servers running on Cent OS 4.2 x86_64 & having cPanel/WHM installed on it and Mysql 4.1.14 and Apache 1.3.34....

First of all my main problems are with the loads on this server , it's too high even though it doesn't have more than 20 accounts on it, 2 of them are heavy loaded forums which have high MySQL usage but i'll list you my problems more technically down here.

Apache seems like it's not closing the connection after it sent the user the requested data, here's my httpd.conf and /etc/my.cnf & below it some simple lines of my 'ps aux' as i can't paste the whole list here cause it's too large.

[---------- Start httpd.conf -----------]
RLimitMEM 355722581
RLimitCPU 240
ServerType standalone
ServerRoot /usr/local/apache
Timeout 22
KeepAlive Off
MaxKeepAliveRequests 1
KeepAliveTimeout 1
MinSpareServers 6
MaxSpareServers 24
StartServers 6
MaxClients 768
MaxRequestsPerChild 200
[---------- End httpd.conf -----------]

[---------- Start my.cnf -----------]
[mysqld]
set-variable = max_connections=500
safe-show-database
connect_timeout=8
interactive_timeout=100
max_connect_errors=8
tmp_table_size=64M
wait_timeout=15
skip-innodb
[mysqldump]
quick
max_allowed_packet=10M
query_cache_size=128M
query_cache_limit=2M
query_cache_type=1
key_buffer_size = 1G
[---------- End my.cnf -----------]


[---------- Start Some lines of 'ps aux' ----------]
nobody 25197 1.0 0.5 123452 10376 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25198 6.7 1.1 131424 20868 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
nobody 25199 0.3 0.5 123316 10340 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25201 1.5 0.5 123316 10352 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25202 0.6 0.5 123316 10316 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25203 0.6 0.5 123312 10300 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25204 0.3 0.5 123316 10332 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25208 0.4 0.5 123316 10344 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25209 2.3 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
mysql 25217 15.1 0.8 48176 15908 pts/2 S 14:49 0:01 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
mysql 25219 10.1 0.8 47980 15888 pts/2 R 14:49 0:01 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
nobody 25220 10.5 0.9 126556 16708 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
nobody 25222 1.1 0.5 127320 10528 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25223 8.2 0.9 127468 16760 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
mysql 25224 3.2 0.8 47980 15888 pts/2 S 14:49 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
mysql 25226 6.1 0.8 47980 15888 pts/2 R 14:49 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
nobody 25231 4.9 0.5 123452 10388 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
mysql 25232 4.0 0.8 47980 15888 pts/2 R 14:49 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
nobody 25233 7.1 0.9 127716 17484 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25234 15.0 1.1 131084 21468 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
nobody 25235 3.4 0.5 123452 10412 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25239 2.0 0.5 123452 10380 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25240 3.7 0.6 123720 11100 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25241 13.2 1.1 131184 21544 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
nobody 25242 0.7 0.5 123316 10296 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25243 10.1 0.9 126712 16844 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25245 11.3 0.9 127736 17420 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25246 6.5 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
nobody 25247 15.7 0.8 125976 15548 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
nobody 25254 4.7 0.6 123744 11140 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25256 2.8 0.5 123316 10368 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25257 8.2 0.9 127728 17456 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25260 2.0 0.5 123452 10368 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25261 19.2 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
nobody 25262 8.2 0.5 123452 10380 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25263 1.6 0.5 123316 10340 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25264 11.8 0.9 127716 17440 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25265 6.0 0.5 123316 10348 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25266 1.8 0.5 123316 10336 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25273 4.2 0.5 123316 10336 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25278 7.4 0.6 123808 11208 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25279 6.5 0.5 123316 10340 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25281 15.5 0.5 123448 10380 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25282 38.5 1.1 131264 21072 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25283 19.0 0.5 123316 10380 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
mysql 25284 23.0 0.0 0 0 pts/2 Z 14:49 0:00 [mysqld] <defunct>
nobody 25289 3.5 0.5 123316 10336 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25290 24.0 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
nobody 25291 10.0 0.5 123312 10368 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
mysql 25292 23.0 0.8 47980 15888 pts/2 S 14:49 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
nobody 25293 24.0 0.5 123452 10388 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25294 23.0 0.6 123716 10996 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25295 9.0 0.5 123312 10344 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25296 34.0 0.5 123452 10388 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25297 49.0 0.5 123316 10384 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25298 8.0 0.5 123316 10332 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25299 43.0 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
nobody 25300 9.0 0.6 123708 11004 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25301 0.0 0.5 123316 10332 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25302 0.0 0.5 123316 10328 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25303 0.0 0.5 123316 10332 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
nobody 25304 0.0 0.5 123312 10360 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
[---------- End Some lines of 'ps aux' ----------]


[---------- Start Server's Uptime ----------]
14:53:03 up 5 days, 20:42, 2 users, load average: 26.56, 24.22, 24.39
[---------- End Server's Uptime ----------]


[---------- Start Top ----------]
top - 15:14:00 up 5 days, 21:03, 2 users, load average: 32.34, 25.73, 24.34
Tasks: 419 total, 22 running, 372 sleeping, 1 stopped, 24 zombie
Cpu(s): 34.7% us, 16.3% sy, 0.0% ni, 40.9% id, 6.8% wa, 0.3% hi, 1.0% si
Mem: 1812452k total, 1751924k used, 60528k free, 23812k buffers
Swap: 3534260k total, 207504k used, 3326756k free, 849544k cached
[---------- End Top ----------]


[---------- Start some netstat info ----------]
[email protected] [~]# netstat -nt | grep :80 | wc -l
1390
[---------- End some netstat info ----------]


Note: I have Zend Optimizer 2.6.2 for x86_64 & have eaccelerator-0.9.4-rc1 & have FastCGI Apache module.

Also i've suspended the most using account as to see if there's a change and not a clue....

Please if someone think he could share finding this problem it'll be most welcomed :)
 

webignition

Well-Known Member
Jan 22, 2005
1,880
0
166
The ps output shows a fair amount of Apache processes - is this normal for this server, or is it more than you would expect?

I recently had a problem with a user who was using Mambo with a bit of a dodgy modified template. The symptoms were loads of Apache processes and looking at the Apache Status page in WHM revealed that the IP of the requesting machine was the server itself.

Each request to the Mambo-based site was spawning a request by Mambo itself for something within Mambo, and this request spawned another identical request and so on.

It's quite easy to see how one request to the Mambo-based site was nearly killing the server in less than a second.
 

mOdY

Well-Known Member
Dec 25, 2004
80
0
156
Well, this ps output isn't even half of the real ps output im having although pasting it here will be loose of time & i don't have any mambo hosted on this vps although it's a good thing to paste here an httpd status maybe it could help in tracing the core of the problem...

Server Built: Feb 13 2006 20:26:46
_________________________________________________________________

Current Time: Wednesday, 15-Feb-2006 15:49:34 EST
Restart Time: Wednesday, 15-Feb-2006 15:40:15 EST
Parent Server Generation: 5
Server uptime: 9 minutes 19 seconds
Total accesses: 8595 - Total Traffic: 224.9 MB
CPU Usage: u116.47 s24.52 cu.09 cs.11 - 25.3% CPU load
15.4 requests/sec - 411.9 kB/second - 26.8 kB/request
219 requests currently being processed, 7 idle servers
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWW
_WWWWWWWWW_WWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_LWWWWWWW
WWWWWWWWWWWWWW_WWWSWWWW....W..W...W.WWW.W.WW.....WW.............
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
 

mOdY

Well-Known Member
Dec 25, 2004
80
0
156
webignition said:
Well, I can see that there are a fair few active requests.

Is there anything common about them? Look for patterns to find the cause!

Look for patterns ! how ?
 

acenetryan

Well-Known Member
PartnerNOC
Aug 21, 2005
197
1
168
In apache status, you can look for a large amount of requests from a single IP. If you see an abnormal amount, use iptables to block the IP. You may also want to look into "netstat". It can show you current network traffic and is helpful if you're trying to find the cause of a DOS.