strange HTTPD DoS attack ...

[student]

Sometimes, my apache web server is getting overloaded by unknown requests. During one of this strange overload, using WHM -> 'Server Status'->'Apache Status' I've noticed enormous amount of unusual connections:
eg.
51-7 | 3408 | 0/97/2961 | R | 0.47 | 43 | 1 |0.0 | 0.93 | 80.44 | ? | ? | ..reading..
...
so we have in the request:
Host: ? Vhost: ? Request: ..reading..

and because of that connections, normal sites are unavailable (very, very long loading time, or timeout)

any suggestions how to deal with this?
btw. after apache restart (service httpd restart) all is ok again, I see usual connecions by normal users, without strange '..reading..' requests...

Is there any way to see the output of WHM -> 'Server Status'->'Apache Status' in the shell? I mean is there some kind of shell script that does that and producing output not to WHM, but my shell console?

Ok, and the last question, I would like to ask about SWAP space. Seeing lots of sudden 'load average' overload jumpings, and [kswapd0] process eating all of my cpu%. I've decided to turn off SWAP space. Anyone has experience with non SWAP space server working? Is it good solution? I've 2GB of RAM, and in my opinion, SWAP is not needed in such ampunt of RAM (that is max. used in ~50%)

Appreciate any help, thanks

 

[Manuel_accu]

Hi,

Check the below mentioned URL, which may help you to find the culprit:

http://forums.linuxwebadmin.info/index.php/topic,95.0.html

Thx

 

[driverC]

I had the same type of attack quite often. They connect to Apache to create a process and then they don't do anything leaving the process open for as long as it takes to time out. Often they connect to the server's IP so it is impossible to stop the attack by suspending a specific account. As a result of this Apache may reach the process limit and reject new connections or the server may break down overloaded just because of some 14 year old little ****.

Here is how to deal with it:
1) SSH into your server
2) Enter this command: pico /usr/local/apache/conf/httpd.conf
3) Scroll down to where it reads "Timeout 300"
4) Change it to "Timeout 5"
5) Scroll down to "MaxClients 150"
6) Change it to "MaxClients 500"
7) Type "service httpd stop"
8) Wait for around 10 seconds
9) Type "service httpd start"

(you may want to create a backup of your httpd.conf file first just in case you make a mistake)

Now requests will time out after 5 seconds. That means if there is no traffic between the server and the visitor for 5 seconds the connection will time out. This will improve the server's load dramatically and the DDoS processes will probably not reach Apache's maxClients value anymore. The timeout setting can be a problem if customers are uploading files using Perl or PHP scripts but apart from that it shouldn't cause any issues (i.e. it will not interrupt downloads) and so far no customer has complained.

Now that Apache is working again you can try to find a way to lock the attacking IPs out which can be very tricky and you may be better off waiting until the attack ends.

 

[NightStorm]

Wouldn't this type of attack (Where zombies connect dozens of times to a single site and overload httpd) that mod_evasive was designed to block?

 

[driverC]

Wouldn't this type of attack (Where zombies connect dozens of times to a single site and overload httpd) that mod_evasive was designed to block?

No ! Not if the attack is comming from various different IPs where each IP does only access the server once every couple of seconds. And my experience is that such attacks do never come from a single IP.

Enter the following command to find out if all accesses are comming from the same IP:

netstat -na |grep :80 |sort

Note: If you see the same IP multiple times (i.e. 10 times) it doesn't mean that it is attacking your server.

 

[student]

Hi,
driverC, great thanks for help, after applying your steps, this kind of DoS has been stopped

Speaking about strange requests, also i see (quite often) lots of connections (in Apache Status) such as:
136-14 | - | 0/0/45 | . | 0.01 | 33780 | 1 | 0.0 | 0.00 | 0.90 | 80.182.50.66 | (unavailable) | GET /img/Biedrona.gif HTTP/1.1

as you can see, te 'Vhost' is '(unavailable)'. These connections sometimes are existing for a few hours and comming from single IP (server is not dropping them), is it normal? Should i consider this as some kind of DoS attempt?

 

[driverC]

Hi,
driverC, great thanks for help, after applying your steps, this kind of DoS has been stopped

Speaking about strange requests, also i see (quite often) lots of connections (in Apache Status) such as:
136-14 | - | 0/0/45 | . | 0.01 | 33780 | 1 | 0.0 | 0.00 | 0.90 | 80.182.50.66 | (unavailable) | GET /img/Biedrona.gif HTTP/1.1

as you can see, te 'Vhost' is '(unavailable)'. These connections sometimes are existing for a few hours and comming from single IP (server is not dropping them), is it normal? Should i consider this as some kind of DoS attempt?

This may be a keep alive request or something. You could disable keep alive in httpd.conf (but there is no need to do that unless your servers are overloaded). I am not sure why it says unavailable in the VHost section. Maybe it is an access to your server's IP (i.e. someone accessing http://IP/~username/index.htm. You can disable that in Cpanel's security settings (you should really disable this for security reasons and because it allows people to steal bandwidth from other users).

 

[jols]

This may be a keep alive request or something. You could disable keep alive in httpd.conf (but there is no need to do that unless your servers are overloaded). I am not sure why it says unavailable in the VHost section. Maybe it is an access to your server's IP (i.e. someone accessing http://IP/~username/index.htm. You can disable that in Cpanel's security settings (you should really disable this for security reasons and because it allows people to steal bandwidth from other users).

I would love to disable this, but then how would our newly hosted customers be able to see their web pages before their domain becomes active at the server?

 

[dexus]

This may be a keep alive request or something. You could disable keep alive in httpd.conf (but there is no need to do that unless your servers are overloaded). I am not sure why it says unavailable in the VHost section. Maybe it is an access to your server's IP (i.e. someone accessing http://IP/~username/index.htm. You can disable that in Cpanel's security settings (you should really disable this for security reasons and because it allows people to steal bandwidth from other users).

No, that is not the case, I was testing that strenge thing on a server that have only two domains and I always have such 'Vhost' is '(unavailable)' connections to existing files on those domains. When I trie to access through IP adress, like you suggested, that is always shown as:

(unavailable) | GET /~user/path/file.php

end this "strange" connections are always shown as:

(unavailable) | GET /path/file.php

as you notice without "~" and without username, just path on the unavailable host, and that just doesn't have any sense.

Does anyone have some logical explanation for this. I am just lost here.

 

[Spiral]

I would love to disable this, but then how would our newly hosted customers be able to see their web pages before their domain becomes active at the server?

That is ridiculously easy !!!!

Just setup wildcard DNS and assign all new accounts a temporary subdomain address!

We have mod_userdir disabled in all our servers (actually not even compiled) so
it is not even possible to use and "/~username" at all whatsoever!

In place of that though, we have a global server domain setup and through
the use of wildcard DNS and a one line brainless easy modification to Cpanel's apache
templates in /usr/local/cpanel/etc/httptemplates, any new accounts that we create
through WHM or the Cpanel API automatically have a temporary subdomain setup.

Users would have something like: http://login.domain.com that they can
use until their own domain finishes transferring.

How?

Let's say your domain used for user subdomains is "mydomain.com" and
your shared IP address for your server is "1.2.3.4" then you would do this ....

1. Login to your server via SSH as root

2. Edit /var/named/mydomain.com.db and add the following line:

* IN A 1.2.3.4

3. Restart your DNS server

/etc/rc.d/init.d/named restart

4. Edit /usr/local/cpanel/httptemplates/apache1/standard ('apache2' if you use apache2)

Under the ServerAlias lines in the template, add a new line:

ServerAlias %user%.mydomain.com

5. Restart your apache server

/usr/local/apache/bin/apachectl restart

Existing accounts created before this change won't be effected but any new accounts
you create would automatically have an alternate subdomain address of
http://login.mydomain.com where "login" is the user's login name.

 

[ramprage]

Here is how to deal with it:

5) Scroll down to "MaxClients 150"
6) Change it to "MaxClients 500"

That's not exactly good advice..... you increased the allowed number of connections which in a real DDoS attack will end up crashing yoru server becaue the hardware can't handle Apache with that many connections going.

Instead raise your MaxClients value slightly. You're using the default value anyways, try something like 200.

You should use a DDoS tool combined with your firewall as well. This will help block IPs who are connecting too much from the usual automatically. I've seen attacks where some IPs are connected over 4,000 times (per single IP)

Viewing Apache status from shell:

lynx localhost/whm-server-status

 

[Arsenico]

Try limiting the connections that one ip address can make on the port 80

with IPFW:
ipfw add allow tcp from any to me dst-port 80 in limit src-addr 15


Where 15 is the max of connections that an ip addres can create.

also try with this.

ipfw add 65534 allow tcp from any to any limit src-addr 50


Where 50 is the max of connections that 1 ip address can make to the server.

You can change it. (dont know how to do it on IPTABLES, sorry , im married with UNIX and ipfw :P)


also in the HTTPD.CONF

RLimitMEM
RLimitCPU

http://www.apacheref.com/ref/http_core/RLimitCPU.html
http://www.apacheref.com/ref/http_core/RLimitMEM.html


u'll need to set those to not get overloaded all the time.

 

[CaMer0n]

I'd just like to thank driverC for his http.conf configuration.
I was getting massive CPU usage from Apache, mostly due to Yahoo and MSN bots it seems.
Every process was using 10-30% of the CPU.
After making the modifications below, the problem has vanished.

Thanks again
Cameron

1) SSH into your server
2) Enter this command: pico /usr/local/apache/conf/httpd.conf
3) Scroll down to where it reads "Timeout 300"
4) Change it to "Timeout 5"
5) Scroll down to "MaxClients 150"
6) Change it to "MaxClients 500"
7) Type "service httpd stop"
8) Wait for around 10 seconds
9) Type "service httpd start"