The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Bot Protection

Discussion in 'General Discussion' started by lloyd_tennison, Jul 12, 2005.

  1. lloyd_tennison

    lloyd_tennison Well-Known Member

    Joined:
    Mar 12, 2004
    Messages:
    698
    Likes Received:
    1
    Trophy Points:
    18
    One of my clients has been being hit with a bot that is a form filler. The bot is called "www.textron.com/testBot" and it fills out forms like crazy. I have tried using a robots.txt file - but since this is one that probably changes name all the time (textron makes highly protected armored vehicles used in Irq) so that's why their name is being used. IP addresses change all the time on them - so cannot block by IP.

    I tried:

    Code:
    User-agent: testBot
    Disallow: /
    
    User-agent: www.textron.com/testBot
    Disallow: /
    
    but any thoughts as they have received almost a thousand this week.

    As a last resort - maybe one of those submissions that use a verify graphic- that is cheap or open source?
     
  2. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    robots.txt is irrelevant here. That file is only an advisory file for search engines which they can completely ignore - it doesn't actually do anything.

    Something like mod_dosevasive might help.
     
  3. lloyd_tennison

    lloyd_tennison Well-Known Member

    Joined:
    Mar 12, 2004
    Messages:
    698
    Likes Received:
    1
    Trophy Points:
    18
    Already installed at default settings..

    Hmm.
     
  4. lloyd_tennison

    lloyd_tennison Well-Known Member

    Joined:
    Mar 12, 2004
    Messages:
    698
    Likes Received:
    1
    Trophy Points:
    18
    Anyone have any other ideas - as the bot is so fast less that 1 minute - that mod_dosevasive has not had time to react. Then all different IP's..
     
  5. Izzee

    Izzee Well-Known Member

    Joined:
    Feb 6, 2004
    Messages:
    469
    Likes Received:
    0
    Trophy Points:
    16
    Try this in a .htaccess in the web root of the sites that are being hit by this bothersome bot.
    These are only examples.
    You can use these entries as a guide to add your own bothersome bots.
    Code:
    RewriteEngine on 
    RewriteBase /
    # User-Agents with no privileges (mostly spambots/spybots/offline downloaders that ignore robots.txt)
    RewriteCond %{REMOTE_ADDR} "^63\.148\.99\.2(2[4-9]|[3-4][0-9]|5[0-5])$" [OR] # Cyveillance spybot
    RewriteCond %{HTTP_USER_AGENT} HostItCheap [NC,OR] # spambot
    RewriteCond %{HTTP_USER_AGENT} PersonaPilot [NC,OR] # rude bot
    RewriteCond %{HTTP_USER_AGENT} "Indy Library" [NC] # spambot
    RewriteRule .* - [F,L]
    I have tried to give a cross section of what can be achieved using the Rewrite Engine.
    Some reading at the Apache site might help.

    For your particular bot you might be able to use this:
    Code:
    RewriteEngine on 
    RewriteBase /
    RewriteCond %{HTTP_USER_AGENT} testBot [NC] # rude and resource hog bot
    RewriteRule .* - [F,L]
    My .htacces file is some 7k in size now.
    I constantly keep adding to it when I find a misbehaving bot.
    My logs provide the source of those bots that I don't particularly want.
    HTH
     
Loading...

Share This Page