The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Block Unwanted Bots Agent and Proxys

Discussion in 'Security' started by Bidi, Feb 8, 2016.

  1. Bidi

    Bidi Well-Known Member

    Joined:
    Oct 3, 2012
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    Romania, Transilvania
    cPanel Access Level:
    DataCenter Provider
    Hy guys, so after 4h of google-ing and testing i quit :) the think is i made a file .htaccess in /home/ into my server and i added the way it is into it. But i think is not working, i used User Agent Switch to make the tests but still nothink.

    The think is i whant to block this sort of bad agents and proxy traffic to ower websites, i think nobody whants bad traffic right ? Like proxys ones, witch there is a software it grabs 100k proxys and after start sending trafic to a website and makes masive traffic and a masive usage on the server. So here is it, did i made sompting wrong ? :D

    Code:
    <IfModule mod_rewrite.c>
    RewriteEngine On
    
    RewriteCond %{HTTP:VIA}                 !^$ [OR]
    RewriteCond %{HTTP:FORWARDED}           !^$ [OR]
    RewriteCond %{HTTP:USERAGENT_VIA}       !^$ [OR]
    RewriteCond %{HTTP:X_FORWARDED_FOR}     !^$ [OR]
    RewriteCond %{HTTP:PROXY_CONNECTION}    !^$ [OR]
    RewriteCond %{HTTP:XPROXY_CONNECTION}   !^$ [OR]
    RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR]
    RewriteCond %{HTTP:HTTP_CLIENT_IP}      !^$ [OR]
    RewriteCond %{HTTP_USER_AGENT} ^$        [OR]
    RewriteCond %{HTTP_REFERER} amazonaws\.com [OR]
    RewriteCond %{HTTP_USER_AGENT} ^AISearchBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^woriobot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^heritrix [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetSeer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Nutch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^aipbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Anarchie [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ASPSeek [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^attach [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^autoemailspider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Custo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^DISCo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^eCatch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^FlashGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^GetRight [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^GrabNet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Grafula [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^HMView [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^HTTrack [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^iblog [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^InterGET [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^JetCar [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^larbin [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Linkwalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^nameprotect [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Navroad [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NearSite [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetAnts [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Octopus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^pavuk [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^RealDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ReGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^searchestate [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SuperBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Surfbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^TurnitinBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebAuto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebCopier [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebFetch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebReaper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebSauger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebStripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Wget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Widow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Xenu [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Zeus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^curl/ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^HTMLParser [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Jakarta\ Commons [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Java [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^libcurl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LWP::Simple [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^lwp-request [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Microsoft\ Data\ Access [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Microsoft\ URL\ Control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MS\ Web\ Services\ Client\ Protocol [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^PECL::HTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^POE-Component-Client-HTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^PycURL [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Snoopy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^VB\ Project [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WWW::Mechanize [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} RPT-HTTPClient [NC,OR]
    
    RewriteCond %{HTTP_USER_AGENT} ^.*(<|>|'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(HTTrack|Wordpress|wp|emailwolf|clshttp|archiver|loader|email|nikto|miner|python).* [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|libwww\-perl|emailwolf|curl|wget|harvest|scan|grab|extract).* [NC]
    RewriteRule ^(.*)$ - [F,L]
    </IfModule>
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,834
    Likes Received:
    672
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  3. quizknows

    quizknows Well-Known Member

    Joined:
    Oct 20, 2009
    Messages:
    942
    Likes Received:
    57
    Trophy Points:
    28
    cPanel Access Level:
    DataCenter Provider
    This would be my recommendation.

    There are a couple good ways to do this with ModSecurity.

    One would be individual rules like:

    Code:
    SecRule REQUEST_HEADERS:User-Agent "^NetSeer" "deny,id:12345"
    
    Or, you could make a list of user agents in a file like /usr/local/apache/conf/my_custom_list.conf and then make a modsec rule like:
    Code:
    SecRule REQUEST_HEADERS:User-Agent "@pmFromFile /usr/local/apache/conf/my_custom_list.conf" "deny,id:12346"
    
    This method is detailed in the modsecurity manual here Reference Manual · SpiderLabs/ModSecurity Wiki · GitHub
     
  4. Bidi

    Bidi Well-Known Member

    Joined:
    Oct 3, 2012
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    Romania, Transilvania
    cPanel Access Level:
    DataCenter Provider
    i will try to make mod_sec way :) i hope not braking to many thinks, but there is another one, witch drives me crazy is this (on the pic) and i dont know what agent to write to block them, if no agent or sompting like this.

    They ar some sort of proxys, or mirc bots i think.

    [Removed]
     
    #4 Bidi, Feb 10, 2016
    Last edited by a moderator: Feb 10, 2016
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,834
    Likes Received:
    672
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  6. Bidi

    Bidi Well-Known Member

    Joined:
    Oct 3, 2012
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    Romania, Transilvania
    cPanel Access Level:
    DataCenter Provider
    I fix it, i made all the thinks in mod_sec rules, even for proxys, bad agents, fake hits ...etc thank you, works like a charm :D
     
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,834
    Likes Received:
    672
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  8. Bidi

    Bidi Well-Known Member

    Joined:
    Oct 3, 2012
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    Romania, Transilvania
    cPanel Access Level:
    DataCenter Provider
    Whell i`m back :( cuz of this rulls witch brakes my websites and i dont understand whant i`m doing bad, dose anyone got any ideas ?



    Code:
     # Block empty User-Agents.
    SecRule REQUEST_HEADERS:User-Agent "@eq 0" \
    "id:'13009',phase:2,t:none,deny,status:406,log,msg:'Fake Agent - Detectat'"
    
    and
    
    SecRule REQUEST_HEADERS:User-Agent "^$" \
    "id:'13006',phase:2,t:none,deny,status:406,log,msg:'Fake Agent - Detectat'" 
     
    #8 Bidi, Feb 17, 2016
    Last edited: Feb 17, 2016
  9. quizknows

    quizknows Well-Known Member

    Joined:
    Oct 20, 2009
    Messages:
    942
    Likes Received:
    57
    Trophy Points:
    28
    cPanel Access Level:
    DataCenter Provider
    Try this instead of those two:

    Code:
    # Block empty User-Agents.
    SecRule &REQUEST_HEADERS:User-Agent "@eq 0" \
    "id:'13009',phase:2,t:none,deny,status:406,log,msg:'Fake Agent - Detectat'"
    
    If you're using an operator like @eq sometimes you need to add the & before the matched var.
     
  10. Bidi

    Bidi Well-Known Member

    Joined:
    Oct 3, 2012
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    Romania, Transilvania
    cPanel Access Level:
    DataCenter Provider
    Hy i forget to mention :( i had like theath before but the problem is when i had theath rule enabled and on WHM i go to Apache Status i got this error.

    Do i have to whitelist sompting ?


    Apache server status for ************
    Failed to receive status information from Apache.

    And i dont understand why.

    Thank you for your reply
     
  11. quizknows

    quizknows Well-Known Member

    Joined:
    Oct 20, 2009
    Messages:
    942
    Likes Received:
    57
    Trophy Points:
    28
    cPanel Access Level:
    DataCenter Provider
    That is the correct syntax (with the &) however I don't think WHM provides a User Agent when it queries server status.

    The log looks like this:
    Code:
    127.0.0.1 - - [17/Feb/2016:21:20:01 -0500] "GET /whm-server-status HTTP/1.0" 200 6153
    
    Try this:

    Code:
    # Block empty User-Agents.
    SecRule &REQUEST_HEADERS:User-Agent "@eq 0" \
    "id:'13009',phase:2,t:none,deny,chain,status:406,log,msg:'Fake Agent - Detectat'"
    SecRule REMOTE_ADDR "!@ipMatch 127.0.0.1"
    
    This will allow 127.0.0.1 to query the server without a user agent specified but other IPs will not be allowed to. That should fix WHM server status for you.
     
  12. Bidi

    Bidi Well-Known Member

    Joined:
    Oct 3, 2012
    Messages:
    51
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    Romania, Transilvania
    cPanel Access Level:
    DataCenter Provider
    I had added lest see how it works :D thank you
     
Loading...

Share This Page