Monster load spikes due to hostile spider

[bhd]

We have just blocked customscoop.com. It appears they run a new gathering service and have a spider running across several servers which intermittently opens a bazillion threads at a time to a single source (not very polite!) killing the server it is trying to spider in the process. They were opening about 500 threads at a time to a message board and pushing the load average up to 100+ for periods of 10-15 minutes.

If you have had any massive load spikes in the past few weeks with no logical explanation, you may want to search your log files for any of the following IP addresses: 64.49.241.192 - 64.49.241.223

Better still stick 64.49.241.192/27 into your APF deny_hosts

We've had this on 3 servers already ... there's a hostile spider on the loose so be warned!

[gorilla]

Have you contacted rackspace regarding this ? as it seem its one of their IPs http://www.whois.sc/64.49.241.192

[bhd]

I saw. It is Rackspace. However this is not really an abuse issue which is why I never reported it -- although the consequences of what these guys are doing is just as bad as a DOS attack I guess.

[RickG]

Not to defend these guys, but customscoop.com looks like a legit operation that I'm certain would want to know we are starting to block their IP's because of the "tiger" in their robot. I would hope this is not what they intended, as it would have a serious impact on their credibility and business model (maybe we should let someone from news.com know about this - a little negative press press can do wonders). If you have the inclination, you may want to drop them a note.

[bhd]

Hehe, a tiger eh. That's about the size of it. Frankly, I don't want to give anyone bad publicity ... just wanted to stop the tiger from killing our servers

The upside is, I was motivated to write a perl script to track events just like this. What I mean by "just like this" is simply that it is very difficult (for me at least) to find the source of a 100+ server load when it's only there for a few minutes.

1. Scanning log files don't help.
2. Doing things like top > filename generate massive files and, in my case, were never run at the right time so I never got to see what I was looking for.
3. When the load average is so high, logging in with SSH is impossible anyways.

The script I wrote can run several commands (of your choice ... like ps, netstat etc) at once and capture to a text file. It sits in a loop monitoring load and only begins logging when the load hits a preset level ... that way it only logs when a spike is ocurring. That's how I found this spider with the tiger in it's tank.

If anyone is interested, I can post a link to the zip file.

[gorilla]

love to have a look at your script

[bhd]

You can download it here