The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Monster load spikes due to hostile spider

Discussion in 'General Discussion' started by bhd, Apr 10, 2005.

  1. bhd

    bhd Well-Known Member

    Joined:
    Sep 20, 2003
    Messages:
    149
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    JNB ZA
    cPanel Access Level:
    Root Administrator
    We have just blocked customscoop.com. It appears they run a new gathering service and have a spider running across several servers which intermittently opens a bazillion threads at a time to a single source (not very polite!) killing the server it is trying to spider in the process. They were opening about 500 threads at a time to a message board and pushing the load average up to 100+ for periods of 10-15 minutes.

    If you have had any massive load spikes in the past few weeks with no logical explanation, you may want to search your log files for any of the following IP addresses: 64.49.241.192 - 64.49.241.223

    Better still stick 64.49.241.192/27 into your APF deny_hosts

    We've had this on 3 servers already ... there's a hostile spider on the loose so be warned! ;)
     
  2. gorilla

    gorilla Well-Known Member

    Joined:
    Feb 3, 2004
    Messages:
    699
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Sydney / Australia
    #2 gorilla, Apr 10, 2005
    Last edited: Apr 10, 2005
  3. bhd

    bhd Well-Known Member

    Joined:
    Sep 20, 2003
    Messages:
    149
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    JNB ZA
    cPanel Access Level:
    Root Administrator
    I saw. It is Rackspace. However this is not really an abuse issue which is why I never reported it -- although the consequences of what these guys are doing is just as bad as a DOS attack I guess.
     
  4. RickG

    RickG Well-Known Member

    Joined:
    Feb 28, 2005
    Messages:
    238
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    North Carolina
    Not to defend these guys, but customscoop.com looks like a legit operation that I'm certain would want to know we are starting to block their IP's because of the "tiger" in their robot. I would hope this is not what they intended, as it would have a serious impact on their credibility and business model (maybe we should let someone from news.com know about this - a little negative press press can do wonders). If you have the inclination, you may want to drop them a note.
     
  5. bhd

    bhd Well-Known Member

    Joined:
    Sep 20, 2003
    Messages:
    149
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    JNB ZA
    cPanel Access Level:
    Root Administrator
    Hehe, a tiger eh. That's about the size of it. Frankly, I don't want to give anyone bad publicity ... just wanted to stop the tiger from killing our servers ;)

    The upside is, I was motivated to write a perl script to track events just like this. What I mean by "just like this" is simply that it is very difficult (for me at least) to find the source of a 100+ server load when it's only there for a few minutes.

    1. Scanning log files don't help.
    2. Doing things like top > filename generate massive files and, in my case, were never run at the right time so I never got to see what I was looking for.
    3. When the load average is so high, logging in with SSH is impossible anyways.

    The script I wrote can run several commands (of your choice ... like ps, netstat etc) at once and capture to a text file. It sits in a loop monitoring load and only begins logging when the load hits a preset level ... that way it only logs when a spike is ocurring. That's how I found this spider with the tiger in it's tank.

    If anyone is interested, I can post a link to the zip file.
     
  6. gorilla

    gorilla Well-Known Member

    Joined:
    Feb 3, 2004
    Messages:
    699
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Sydney / Australia
    love to have a look at your script :)
     
  7. bhd

    bhd Well-Known Member

    Joined:
    Sep 20, 2003
    Messages:
    149
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    JNB ZA
    cPanel Access Level:
    Root Administrator
    You can download it here
     
Loading...

Share This Page