apache ddos attack anyone help?

[katmai]

i got a ton of those requests, and i really have no idea what to do right now. mod_security will not help because this is not a valid httpd request. any clues?

[Tue Apr 10 00:01:06 2007] [error] [client 82.81.92.220] request failed: erroneous characters after protocol string: $MyNick (adsl)Yw|$Lock EXTENDEDPROTOCOLABCABCABCABCABCABC Pk=DCPLUSPLUS0.690ABCABC|
[Tue Apr 10 00:01:06 2007] [error] [client 82.81.92.220] request failed: erroneous characters after protocol string: $MyNick (adsl)Yw|$Lock EXTENDEDPROTOCOLABCABCABCABCABCABC Pk=DCPLUSPLUS0.690ABCABC|
[Tue Apr 10 00:01:06 2007] [error] [client 82.81.92.220] request failed: erroneous characters after protocol string: $MyNick (adsl)Yw|$Lock EXTENDEDPROTOCOLABCABCABCABCABCABC Pk=DCPLUSPLUS0.690ABCABC|
[Tue Apr 10 00:01:06 2007] [error] [client 82.81.92.220] request failed: erroneous characters after protocol string: $MyNick (adsl)Yw|$Lock EXTENDEDPROTOCOLABCABCABCABCABCABC Pk=DCPLUSPLUS0.690ABCABC|
[Tue Apr 10 00:01:06 2007] [error] [client 82.81.92.220] request failed: erroneous characters after protocol string: $MyNick (adsl)Yw|$Lock EXTENDEDPROTOCOLABCABCABCABCABCABC Pk=DCPLUSPLUS0.690ABCABC|
[Tue Apr 10 00:01:06 2007] [error] [client 82.81.92.220] request failed: erroneous characters after protocol string: $MyNick (adsl)Yw|$Lock EXTENDEDPROTOCOLABCABCABCABCABCABC Pk=DCPLUSPLUS0.690ABCABC|

 

[katmai]

nobody came close to this issue ?

 

[brendanrtg]

Setup a firewall and ban that ip address?

 

[chae]

Hi Yah,

Do a search on here for $MyNick you'll see two post from me regarding this. The only option I had was to null route the IP until the attack was over.

 

[Parahosting]

Banning the IP will not fix the problem, because there might be a chance that the IP address WILL change and render it useless. Possibilities are, you're dealing with an actual hacker.

 

[jayh38]

try CSF, it will do a nice job keeping up with the IP changes as needed. set the connection limit to 200 which is default I believe. There are many other tools in there that may help deal with a situation like this.

 

[brianoz]

I second the comments on csf - it will detect the flood and ban that IP, and if the IP changes it will also ban the new IP. It won't cope well with a true DDOS - nothing does, but it will help a lot.

www.configserver.com/cp/csf.html

 

[ramprage]

I wrote a custom DDoS script just for these type of attacks. It runs on cron and detects IPs which have to many connections and can block them using APF or CSF or just iptables.

Works well, I used it with a client who was getting a major attack. Some ips had over 4,000 connections. In less than half hour it blocked them all and saved his server. He tried everything for days and nothing worked. Half hour after my script was on it saved his server.

So yeah it works well

 

[pirrup]

could you post the script? would be very helpfull for others

 

[katmai]

guys . for 1 thing.

the attack is run by admins of big dc++ hubs, they make a redirect on their users to the apache, and overload it with requests.

mod_security - no use because it's not a valid http request.

one thing if you run iptables is to setup:

iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 10 --hitcount 5 -j DROP


this would help in dropping part of the illegal connections, doesn't solve the issue, but at least the apache doesn't crash and still is accessible.

another solution would be to recompile the kernel, which i don't quite like it right now screwing my 140 days uptime i know sounds silly, but i still would love some help on this.

btw regarding csf.

pretty useless, imagine having to drop like 1000 ip addresses, but what do you do if the hub ddosing you is 5000 users or 8000 users ? you make 16000 rules in csf? your server will overload even without any ddos going on at that point. unless you have some quad processors and hell of ram to do it.

second of all. the connections don't stay established. for example, 1 ip gets like 2 3 connections, it takes only 350 ips to fill up your apache (i run myself 1000 maxclients) so csf will not pick up anything.

 

[ramprage]

could you post the script? would be very helpfull for others

My script isn't free, this one is $25 if interested.

 

[Cristi4n]

Also remember, when using:
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 10 --hitcount 5 -j DROP

enable KeepAlive in apache, set Timeout to something small like 10 set keepalive timeout more than 10 seconds, and you will block a lot of repetead connections and allow legit clients to connect.
I personally used seconds 4 and hitcount 2. (DO NOT drop hitcount below 2).
One other problem remains, the flood will still continue, at lower rate. If you can keep up it's ok. Remember not to end up in swap during that attack, increase apache limit until everything works ok.
If you want to block the ip's permanently (I blocked like 20.000 ips, a small number) you can use snort with snortsam and add a rule to block everuthing containing $MyNick, or simply use snort-inline and block everything containing $MyNick or DCPLUSPLUS. Snort Inline will work better as packets will not get to apache.
Are you sure you only see lines with $MyNick or you see also a lot more lines with a simple "-" ?

Remember also, there is no way to block a ddos, you can only mitigate the attack.
CSF and APF in my case were no use. I had a lot of IP's (DDOS not DOS). Also consider that hardware firewalls will not help in such attack. DO NOT THROW your money on hardware firewalls. You can instead buy 1 or 2 GB of ram and increase apache limit even more and at lower price.

I managed to mitigate the attack at my server (was not easy) but I did it and after 48 hours it was gone.

 

[AndyReed]

Also consider that hardware firewalls will not help in such attack. DO NOT THROW your money on hardware firewalls.

I'll have to disagree with you on this. If your host doesn't have a good hardware firewall, then you are out of luck. One of our clients hosting his dedicated server with us was under heavy DDoS attack. Since we use the latest hardware firewall, Cisco ASA 5520, although was a little slow from time to time during that nasty attack, his server never went down and was/is accessible all the time.

 

[Spiral]

the attack is run by admins of big dc++ hubs, they make a redirect on their users to the apache, and overload it with requests.

Just a tip for everyone ....

As one hosting server generally doesn't need to talk to another other
than for DNS and email traffic, I will often block entire data centers
from my servers for inbound web access.

This is very easy to do with IPTABLES and basically would allow any visitor
coming from a home cable or DSL connection to reach your web server
but block web access from any server machine out there.

iptables -A INPUT -s 0.0.0.0/0 -d 1.1.1.1/1 -p tcp --dport 80:2097 -j REJECT

*** 0.0.0.0/0 would be replaced by the CIDR range to be blocked ****

*** 1.1.1.1/1 would be replaced by your server's IP or CIDR range ****

(I put the destination into the iptables block so that outbound connections
from your server to some other server would still work while incoming
connections from the banned IP range would be blocked)

This does a nice job of blocking proxies too since nearly all proxies
run off of server machines hosted at data centers.

You can find out the IP ranges assigned to data centers with simple
WHOIS inquiries against sample IPs at http://www.arin.net

 

[Cristi4n]

Ok, MABY there are some good hardware firewalls. Still most of them do nothing in case of a ddos.
You do not need a hardware firewall that blocks spyware in this case, so for anyone that even thinks of buying one please ask someone with high experience.
IT DEPENDS IF IT IS A DDOS OR DOS. Many people talk about dos and ddos as they are the same thing.
In the case presented above, I see only one IP, and this can be consider a DOS. Still I believe the guy pasted only a few lines and he is dealing with the same problem I had. A ddos not a dos.
In this case:
One ip connects 3~4 times and then waits for 5~10 minutes and connects again.
SO:
CSF, APF and others DO NOT HELP
Scripts like the above that costs 25$ DOES NOT HELP AT ALL. CSF and APFdoes the same thing and maby even better (Is good for DOS, not DDOS)
mod_evasive a.s.o does not help either.
The rules for iptables presented above are a good start. After that, you are on your own.
You can do a simple cat in the error log and grep for all the ips that flood untill now and ban them. That is also good as they will try to reconnect.


About blocking a range of IPs, can be a solution but was not a good one in my case. My server has clients from country X for example and the ddos came from 3 major networks in that country. Blocking range of ips is a bad ideea as you will block almost all your legit visitors.
It can be a good ideea in some cases.
Also I saw in error log lines with $MyNick and DCPLUSPLUS but I do not think they are real DC clients. Real DC clients do not try to reconnect that fast and do not cause a ddos for 24 hours.
In my case, a client sent a line with $MyNick then started to send "-" and only that. They are not real DC clients, they are just computers that got a virus. I just unblocked a client of mine that was flooding my server without knowing. He doesn't even have a DC client installed.

 

[Spiral]

About blocking a range of IPs, can be a solution but was not a good one in my case. My server has clients from country X for example and the ddos came from 3 major networks in that country. Blocking range of ips is a bad ideea as you will block almost all your legit visitors.

Apparently you didn't read anything I wrote above!

I wasn't talking about blindly "blocking IP ranges" per say ....

I was talking about blocking IP ranges that ORIGINATE from server data centers
specifically and of those only blocking inbound web traffic specifically.

What I am talking about doing will have absolutely no impact whatsoever on
human visitors trying to reach your server no matter where they are located
because it is not internet connection IP addresses that are being blocked.

Users coming from home and business internet connections would be able to
reach all your web site just fine without any trouble.

However, other server machines out there housed at big data centers would
be blocked from talking to your server other than general email, dns, ping,
and SSH / rsync type access. Web access would be blocked.

This will eliminate the vast majority of attacks as very nearly all automated
attacks originate from compromised servers housed in large scale data centers.




PS: From your other comments, it seems you think that mod_security requires a
request to be a valid request which is incorrect. ModSecurity will actually block
invalid and incomplete requests as well. Make it your first loaded module.


.

 

[katmai]

the datacenter blocks ... make no sense here, my log was just a part from my logs, which showed around 1000 ip addresses trying to hose the apache. Server was reachable during this period, a bit slower because legit people were being blocked at the same time.

the issue we are talking here is dc++ hub users being redirected, unless those dc++ hub users sleep and surf the net from the datacenter you block, your arguments are pretty useless.

mod security will not block that kind of request, and i have proven it last night by seeing nothing getting blocked, with rulesets on (wadda ya want i gave it a try)


next solution is easy but i'm lazy, would be kernel recompilation with iptables+strings support, and then all would come to an end. (big FU goes to all lamerz out there trying to test sysadmin skills)

if anyone had another suggestion ... cool, if not i will wait till some serious problem would come in and i would have to reboot that server.

 

[Cristi4n]

PS: From your other comments, it seems you think that mod_security requires a
request to be a valid request which is incorrect. ModSecurity will actually block
invalid and incomplete requests as well. Make it your first loaded module.


.

I did not say anything about mod_security and it will not help at all in this case.
The problem here is: apache reaches it's limit and will not be able to serve legit clients and it does that even if you have mod_security.


kernel with stringmatch for iptables will help only if you really are getting ddos from legit DC clients. As in my case I had errors with "$MyNick" and DCPLUSPLUS and I really thougut I was getting flooded by a DC HUB admin but most of the other errors contained only one line "-" in the request. And that is why I am saying the computers that attacked my server are infected with a virus. Try to connect with a real DC client and you will see the difference.Real DC clients do not reconnect that fast. I think the virus was ment for flooding a DC HUB and some stupid guy thought at the ideea of using them to attack webservers.
Anyway, what I did was using iptables and snort to mitigate the attack. But if your server works now, you can wait and it will be gone in 24 max 48 hours.

 

[chae]

But if your server works now, you can wait and it will be gone in 24 max 48 hours.

The attack that we had lasted for over a week, we null routed the IP then after a week we brought it back online again & watched top & apache error logs...within minutes the load shot up to 200+ & the error logs starting churning out the $MyNick.

In the end we just dropped the IP altogether from the server. Touch wood haven't had any problem since.

 

[useradmin]

$MyNick

HI,


we are getting this problem every evening for more then 4 hours... how to prevent from $MyNick problem. we are able to see 408 request in access_logs while $MyNick is showing in error logs....




we have ban many for while.. as well as we tried to ban $MyNick manually using mod_security too.


please help to fix such attack....