very strange problem

[mher]

This problem is driving me crazy for the last 12 hours.

i dont' know if this is ddos attack or not but every thirty minutes or so the number of httpd processes on the server is spiking dramatically. The top command shows 250 sleeping processes and httpd stops working.

the apache status in whm shows this just before httpd crashes:

RRRWRRRRRRR_RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRWRR___W
___W............................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process


Srv PID Acc M CPU SS Req Conn Child Slot Host VHost Request
0-3 23632 0/91/91 R 4.86 94 28 0.0 0.41 0.41 ? ? ..reading..
1-3 23633 0/87/87 R 4.90 94 27 0.0 0.67 0.67 ? ? ..reading..
2-3 23634 0/87/87 R 4.55 94 30 0.0 0.36 0.36 ? ? ..reading..
3-3 23635 0/50/50 W 1.86 202 0 0.0 0.15 0.15 80.5.160.19 www.*******.com POST /forums/vbshout.php HTTP/1.1
4-3 23636 0/86/86 R 3.37 94 36 0.0 0.17 0.17 ? ? ..reading..
5-3 23643 0/85/85 R 3.96 94 26 0.0 0.53 0.53 ? ? ..reading..
6-3 23645 0/84/84 R 3.99 94 43 0.0 1.94 1.94 ? ? ..reading..
7-3 23646 0/45/45 R 2.37 94 28 0.0 0.17 0.17 ? ? ..reading..
8-3 23648 0/85/85 R 4.08 94 34 0.0 0.40 0.40 ? ? ..reading..
9-3 23649 0/85/85 R 4.95 94 25 0.0 0.37 0.37 ? ? ..reading..
10-3 23749 0/72/73 R 4.19 94 28 0.0 0.43 0.43 ? ? ..reading..
11-3 23651 0/5/5 _ 0.26 6 2330 0.0 0.15 0.15 218.102.187.146 www.*******.net GET /fop/viewforum.php?f=6 HTTP/1.1
12-3 23896 0/6/19 R 0.21 94 26 0.0 0.00 0.03 ? ? ..reading..
13-3 23904 0/0/1 R 0.03 93 0 0.0 0.00 0.00 ? ? ..reading..
14-3 23905 0/0/0 R 0.00 92 0 0.0 0.00 0.00 ? ? ..reading..
15-3 23906 0/1/1 R 0.03 92 28 0.0 0.00 0.00 ? ? ..reading..
16-3 23907 0/1/1 R 0.03 91 32 0.0 0.00 0.00 ? ? ..reading..


I have to restart httpd several times and block some ips from netstat to make httpd back to work.

 

[celliott]

Im guessing your server load also gets quite high when Apache fails?

This kind of activity is usually caused by Outdated/Insecure scripts. I was amazed at how much difference this kind of thing can make but its a main cause for http related load.

 

[AndyReed]

This kind of activity is usually caused by Outdated/Insecure scripts. I was amazed at how much difference this kind of thing can make but its a main cause for http related load.

In addition to celliott said, and just in case you need to learn more about security and different attacks, go to:
http://servertune.com/kbase/security/attacks.html
http://servertune.com/kbase/security/concept.html

Overall, you need to track down these malicious httpd processes and find out who is killing your server:
/usr/sbin/lsof | grep WHATEVER
ls -l /proc/xxxx
where xxxx is the PID number which may tell you where the script is located.