The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Mod Security Rules Incorrectly Blocking Googlebot

Discussion in 'Security' started by Another Blogger, Jan 6, 2013.

  1. Another Blogger

    Another Blogger Member

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    Hello everyone

    I'm using CSF Firewall in my blog and the problem is the Mod_Security rules incorrectly blocks Googlebot regularly. It is happening almost every week.

    In past I have removed a Mod Security rule which was blocking Googlebot and now a new Mod Security rule 950005 is blocking Googlebot.

    Whenever it blocks Googlebot, I manually remove Googlebot IP address from deny list using WHM.

    Is there any way to permanently stop this? How can I set Mod Security to not block Googlebot?

    My hosting staff suggested me to add following rule at the end of Mod Security configuration:

    Code:
    #Allow googlebots
SecRule REMOTE_HOST googlebot.com$ allow,pass
    But I'm not sure whether I should add it or not. Is there any security risk in adding this rule or is it ok? Is there any other recommended way to allow Googlebot?

    Any kind of suggestions will be highly appreciated.
     
    #1 Another Blogger, Jan 6, 2013
    Last edited: Jan 6, 2013
  2. Indianets

    Indianets Well-Known Member

    Joined:
    Jun 13, 2008
    Messages:
    65
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Root Administrator
    Add this to your config. The right way to identify Google bot is it's User-Agent.

    Code:
    SecRule REQUEST_HEADERS:User-Agent "Googlebot" phase:1,nolog,allow,ctl:ruleEngine=off
     
    #2 Indianets, Jan 6, 2013
  3. Another Blogger

    Another Blogger Member

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Thanks for your reply but if someone edits browser's user-agent using add-on or configurations and make it look-like Googlebot's useragent then? Will it cause a security risk?
     
    #3 Another Blogger, Jan 6, 2013
  4. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    168
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    I would say that the right way to correct that issue is to remove (or maybe edit) the rule blocking Googlebot.

    Where are the rules from? Atomicorp rules don't seem to have rule #950005.
     
    #4 quietFinn, Jan 6, 2013
  5. Another Blogger

    Another Blogger Member

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ I'm using CSF firewall. Following is the email alert content:

    Code:
    Time: Fri Jan 4 13:47:06 2013
IP: 66.249.73.150 (US/United States/crawl-66-249-73-150.googlebot.com)
Failures: 5 (mod_security)
Interval: 300 seconds
Blocked: Permanent Block

Log entries:

[Fri Jan 04 13:44:02 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

[Fri Jan 04 13:44:03 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

[Fri Jan 04 13:45:03 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

[Fri Jan 04 13:46:04 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

[Fri Jan 04 13:47:04 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]
     
    #5 Another Blogger, Jan 6, 2013
  6. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    168
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    I would suggest that you disable LF_MODSEC, or at least change it to temporary block.
     
    #6 quietFinn, Jan 6, 2013
  7. Another Blogger

    Another Blogger Member

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Will it not disable Mod security?
     
    #7 Another Blogger, Jan 6, 2013
  8. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    168
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    No, it will only disable CSF/LFD checking of mod_security logs.
    You can leave it on (LF_MODSEC = 1), but make the block temporary (LF_MODSEC_PERM = 3600, 1 hour block).
     
    #8 quietFinn, Jan 6, 2013
  9. Another Blogger

    Another Blogger Member

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Thanks. But will it solve the problem? I mean it'll block Googlebot IP temporary and after 1 hour when Googlebot will try to crawl again, firewall will again block it temporary. Is it right?

    PS: Can you please clarify what's the difference between enabling and disabling CSF checking of mod_security logs? I mean if we disable the checking, what will happen in that situation? How will the malicious IP address blocked by the server if we disable CSF checking? I'm a little bit confused between Mod security and CSF.
     
    #9 Another Blogger, Jan 6, 2013
  10. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    168
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    I don't know what really is the best solution for this issue.
    Mod_security blocking Googlebot is a "false positive", and usually those are fixed by creating an exception. As has been suggested you could "whitelist" Googlebot by it's User-Agent, but that would open a hole for anyone using a fake Googlebot User-Agent.

    If you want to know what mod_security is and how it works you should start reading for ex. here:
    https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-Frequently-Asked-Questions-(FAQ)#wiki-What_exactly_is_ModSecurity

    CSF/LFD reads, if configured so, mod_security log file, and when it sees that one IP has been blocked LF_MODSEC times in 5 minutes it blocks the IP, either permanently or temporarily.
    So it works so that mod_security is blocking something, in this case Googlebot, CSF/LFD sees those blocks in mod_security log, and blocks the used IP in the firewall.
    In this case, mod_security is not blocking Googlebot because it is Googlebut, but because Googlebot is doing something that seems suspicious. If you can't prevent mod_security from doing that, you should at least make sure that Googlebot is not blocked permanently.

    In our servers we don't have LF_MODESC enabled, but if it was, it would not do permanent blocking.
     
    #10 quietFinn, Jan 7, 2013
  11. Another Blogger

    Another Blogger Member

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Thanks again for your help. So if I disable LF_MODSEC, Mod_Security will still check IP addresses based upon its rules and if it finds something malicious, it'll block it. Right?
     
    #11 Another Blogger, Jan 7, 2013
  12. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    168
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    Yes that's right.
     
    #12 quietFinn, Jan 7, 2013
  13. Another Blogger

    Another Blogger Member

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Thanks for all your help. :)
     
    #13 Another Blogger, Jan 7, 2013
  14. Indianets

    Indianets Well-Known Member

    Joined:
    Jun 13, 2008
    Messages:
    65
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Root Administrator
    Sorry for my delayed arrival.

    If you are using CSF, you can add .googlebot.com to "/etc/csf/csf.rignore"

    Code:
    .googlebot.com
    - Vijay
     
    #14 Indianets, Jan 9, 2013
    Last edited: Jan 9, 2013
  15. crazyaboutlinux

    crazyaboutlinux Well-Known Member

    Joined:
    Nov 3, 2007
    Messages:
    942
    Likes Received:
    0
    Trophy Points:
    66
    It's a nice idea
     
    #15 crazyaboutlinux, Feb 22, 2013
(You must log in or sign up to post here.)
Loading...
Similar Threads - Security Rules Incorrectly
  1. zoner

    SecRemoteRules - ModSecurity Vendor

    zoner, Jul 19, 2017, in forum: Security
    Replies:
    4
    Views:
    140
    quizknows
    Jul 25, 2017
  2. rinkleton

    ModSecurity - Edit Custom Rules

    rinkleton, May 2, 2017, in forum: Security
    Replies:
    11
    Views:
    476
    rarod
    May 5, 2017
  3. caisc

    Mod_security rules to prevent spam registrations and comments on wordpress sites

    caisc, Feb 7, 2017, in forum: Security
    Replies:
    12
    Views:
    642
    caisc
    Feb 10, 2017
  4. oSM

    OWASP ModSecurity v3 - Best Rulesets to use?

    oSM, Dec 26, 2016, in forum: Security
    Replies:
    2
    Views:
    718
    jfall123
    Dec 27, 2016
  5. rs-freddo

    New Mod_Security Ruleset after EA4 upgrade

    rs-freddo, Dec 25, 2016, in forum: EasyApache
    Replies:
    1
    Views:
    334
    cPanelMichael
    Jan 5, 2017

Share This Page