Apache keeps crashing - Ton of "READING" requests

**Myacen** · 01-24-2005 01:37 PM

Stable nothing has changed on our end, another is running 9.9.9 e 36 so I dont know.

**timdorr** · 01-24-2005 01:38 PM

By the way, this has nothing to do with cPanel. It appears to be some sort of worm going around.

**fishfreek** · 01-24-2005 01:39 PM

Originally Posted by linux-image

what does #ipcs on shell as root give u ?

I get this.

root@alpha (/)#ipcs

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 1769472 root 600 524288 251 dest
0x00000000 1802241 root 600 368644 251 dest
0x0052e2c1 229379 postgres 600 1466368 1

------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 0 nobody 600 1
0x0052e2c1 131073 postgres 600 17
0x0052e2c2 163842 postgres 600 17
0x0052e2c3 196611 postgres 600 17
0x00000000 262148 nobody 600 1
0x00000000 589829 nobody 600 1
0x00000000 622598 nobody 600 1
0x00000000 753671 nobody 600 1
0x00000000 786440 nobody 600 1
0x00000000 819209 nobody 600 1

------ Message Queues --------
key msqid owner perms used-bytes messages

Im running WHM 10.0.0 cPanel 10.0.0-C6

I have been running the C6 release of cpanel since at least Thursday of last week.

**fishfreek** · 01-24-2005 01:42 PM

An example of what I am seeing in Apache Status.

I have the modsecurity rule %27 installed to catch the majority of the php worm hits but this must be something new or a new varient.

Main >> Server Status >> Apache Status
Server Version: Apache/1.3.33 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.10 FrontPage/5.0.2.2635 mod_ssl/2.8.22 OpenSSL/0.9.7a
Server Built: Dec 17 2004 20:32:54

--------------------------------------------------------------------------------
Current Time: Monday, 24-Jan-2005 13:39:21 EST
Restart Time: Monday, 24-Jan-2005 13:37:14 EST
Parent Server Generation: 0
Server uptime: 2 minutes 7 seconds
Total accesses: 1024 - Total Traffic: 5.7 MB
CPU Usage: u5.49 s.8 cu0 cs.03 - 4.98% CPU load
8.06 requests/sec - 45.8 kB/second - 5.7 kB/request
202 requests currently being processed, 1 idle servers
RRRRRRRRRRRRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRKRRRRRRRRRRRRRRRRRRRRRRRRRRRRRKRKRRRRRRRRRRRRRRRR
RKRRRRRRWRRRRRRRRRRRKKRKKRRRRRRRKRRRRRKRRRRKKKRRRRRRRKKKRRKRKRRR
RRRKKRKRRR_.....................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process

Srv PID Acc M CPU SS Req Conn Child Slot Host VHost Request
0-0 3523 0/0/0 R 0.00 123 0 0.0 0.00 0.00 ? ? ..reading..
1-0 3524 0/13/13 R 0.02 49 1 0.0 0.07 0.07 ? ? ..reading..
2-0 3525 0/1/1 R 0.00 123 0 0.0 0.00 0.00 ? ? ..reading..
3-0 3526 0/4/4 R 0.00 104 1 0.0 0.05 0.05 ? ? ..reading..
4-0 3527 0/29/29 R 0.03 20 1 0.0 0.12 0.12 ? ? ..reading..
5-0 3530 0/6/6 R 0.01 86 1 0.0 0.03 0.03 ? ? ..reading..
6-0 3531 0/5/5 R 0.01 121 12 0.0 0.00 0.00 ? ? ..reading..
7-0 3532 0/3/3 R 0.01 121 1 0.0 0.00 0.00 ? ? ..reading..
8-0 3533 0/2/2 R 0.02 103 185 0.0 0.03 0.03 ? ? ..reading..
9-0 3534 0/0/0 R 0.00 120 0 0.0 0.00 0.00 ? ? ..reading..
10-0 3535 0/1/1 R 0.00 120 2 0.0 0.00 0.00 ? ? ..reading..
11-0 3536 0/0/0 R 0.00 120 0 0.0 0.00 0.00 ? ? ..reading..
12-0 3546 0/3/3 R 0.00 88 1112 0.0 0.03 0.03 ? ? ..reading..
13-0 3547 0/5/5 R 0.01 85 1 0.0 0.01 0.01 ? ? ..reading..

**linux-image** · 01-24-2005 01:42 PM

if thatz a worm; then the solution was given in this thread to populate the iptables:

http://forums.cpanel.net/showthread....lients+setting

Could anyone of u guys try that ?

**timdorr** · 01-24-2005 01:45 PM

I've been doing something similar but the number of incoming machines is too numerous to make a difference. For every IP I've blocked, there are 2 more right behind it...

**fishfreek** · 01-24-2005 01:53 PM

I do the netstat -n and the resulting list of IP's doesnt really turn up one or a few that are hitting the server a bunch of times.

**fishfreek** · 01-24-2005 02:09 PM

In my error_log I have enteries like this.

[Mon Jan 24 14:02:57 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:02:57 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:02:57 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:02:57 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:02:57 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:02:58 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:02:58 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:02:58 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:02:58 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml
[Mon Jan 24 14:03:08 2005] [error] [client 66.153.120.118] File does not exist: /home/shiolac/public_html/403.shtml

Of course that IP is very likly hitting us causing these problems. LIke was posted above if I IPTABLES that ip and then tail the error log again I have just as many enteries from a new IP. I could sit here all day long and put new IP's in iptables with no signs of a letup.

Everything points to one specific user (shiolac) yet when I put that user in suspended status the issues dont stop.

**easyhoster1** · 01-24-2005 02:15 PM

Originally Posted by fishfreek

In my error_log I have enteries like this.

Of course that IP is very likly hitting us causing these problems. LIke was posted above if I IPTABLES that ip and then tail the error log again I have just as many enteries from a new IP. I could sit here all day long and put new IP's in iptables with no signs of a letup.

Everything points to one specific user (shiolac) yet when I put that user in suspended status the issues dont stop.

You could try commenting out this user in httpd.conf and restart apache to see if that helps.

Just wondering, are any of these sites running phpbb or some other software. Seems like these attacks have the look of a worm. Also, start looking in your /tmp /var/tmp and see if anything has been droped in there.

**Stream** · 01-24-2005 02:17 PM

I have several phpbb-installations but all are up to date.

**bornonline** · 01-24-2005 02:17 PM

I have been seeing - ? ..reading.. in Apache status for about two weeks, but have not found anything on it either.

**fishfreek** · 01-24-2005 02:23 PM

That account is using VB 3.0.6 I dont know if they are using postnuke/phpnuke or some other site creation tool.

If I comment them out of httpd.conf the traffic just continues to come in. Its just requests are going to /usr/local/apache/ vs /home/username/public_html/

**cpanelnick** · 01-24-2005 02:24 PM

We believe this is caused by a new worm/attack. Adjusting the Timeout value in your httpd.conf to something lower then the default 300 will help with this. I wouldn't recommend setting this to below 90 seconds, but if you are really getting hit you may have no choice.

Make sure to restart apache after changing this value
/usr/local/cpanel/bin/safeaprestart

**Myacen** · 01-24-2005 02:28 PM

To fix this temporarily simply edit your httpd.conf file and edit the timeout from the default 300 seconds to 10 and restart apache

**fishfreek** · 01-24-2005 02:32 PM

How can we see what the URL request is so we can develop a modsecurity rule around it?

I set my timeout value to 30. I was having apache crash every few minutes so even a value of 90 I feared would not save me. This of course doent fix the issue of the attack but just keeps apache from dieing.

LinkBack

Thread Tools

"Leave DNS on server" option missing from reseller center "terminate accounts" page.

Apache Status and HTTP Requests of "OPTIONS * HTTP/1.0"

Apache requests stuck in reading state

Apache Rebuild: "Use System Mysql" and "Mysql Module"