The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Mod Security Rules Incorrectly Blocking Googlebot

Discussion in 'Security' started by Another Blogger, Jan 6, 2013.

  1. Another Blogger

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    Hello everyone

    I'm using CSF Firewall in my blog and the problem is the Mod_Security rules incorrectly blocks Googlebot regularly. It is happening almost every week.

    In past I have removed a Mod Security rule which was blocking Googlebot and now a new Mod Security rule 950005 is blocking Googlebot.

    Whenever it blocks Googlebot, I manually remove Googlebot IP address from deny list using WHM.

    Is there any way to permanently stop this? How can I set Mod Security to not block Googlebot?

    My hosting staff suggested me to add following rule at the end of Mod Security configuration:

    Code:
    #Allow googlebots
    SecRule REMOTE_HOST googlebot.com$ allow,pass
    
    But I'm not sure whether I should add it or not. Is there any security risk in adding this rule or is it ok? Is there any other recommended way to allow Googlebot?

    Any kind of suggestions will be highly appreciated.
     
    #1 Another Blogger, Jan 6, 2013
    Last edited: Jan 6, 2013
  2. Indianets

    Indianets Well-Known Member

    Joined:
    Jun 13, 2008
    Messages:
    65
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Add this to your config. The right way to identify Google bot is it's User-Agent.

    Code:
    SecRule REQUEST_HEADERS:User-Agent "Googlebot" phase:1,nolog,allow,ctl:ruleEngine=off
     
  3. Another Blogger

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Thanks for your reply but if someone edits browser's user-agent using add-on or configurations and make it look-like Googlebot's useragent then? Will it cause a security risk?
     
  4. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    18
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    I would say that the right way to correct that issue is to remove (or maybe edit) the rule blocking Googlebot.

    Where are the rules from? Atomicorp rules don't seem to have rule #950005.
     
  5. Another Blogger

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ I'm using CSF firewall. Following is the email alert content:

    Code:
    Time: Fri Jan 4 13:47:06 2013
    IP: 66.249.73.150 (US/United States/crawl-66-249-73-150.googlebot.com)
    Failures: 5 (mod_security)
    Interval: 300 seconds
    Blocked: Permanent Block
    
    Log entries:
    
    [Fri Jan 04 13:44:02 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]
    
    [Fri Jan 04 13:44:03 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]
    
    [Fri Jan 04 13:45:03 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]
    
    [Fri Jan 04 13:46:04 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]
    
    [Fri Jan 04 13:47:04 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]
    
     
  6. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    18
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    I would suggest that you disable LF_MODSEC, or at least change it to temporary block.
     
  7. Another Blogger

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Will it not disable Mod security?
     
  8. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    18
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    No, it will only disable CSF/LFD checking of mod_security logs.
    You can leave it on (LF_MODSEC = 1), but make the block temporary (LF_MODSEC_PERM = 3600, 1 hour block).
     
  9. Another Blogger

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Thanks. But will it solve the problem? I mean it'll block Googlebot IP temporary and after 1 hour when Googlebot will try to crawl again, firewall will again block it temporary. Is it right?

    PS: Can you please clarify what's the difference between enabling and disabling CSF checking of mod_security logs? I mean if we disable the checking, what will happen in that situation? How will the malicious IP address blocked by the server if we disable CSF checking? I'm a little bit confused between Mod security and CSF.
     
  10. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    18
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    I don't know what really is the best solution for this issue.
    Mod_security blocking Googlebot is a "false positive", and usually those are fixed by creating an exception. As has been suggested you could "whitelist" Googlebot by it's User-Agent, but that would open a hole for anyone using a fake Googlebot User-Agent.

    If you want to know what mod_security is and how it works you should start reading for ex. here:
    https://github.com/SpiderLabs/ModSe...stions-(FAQ)#wiki-What_exactly_is_ModSecurity

    CSF/LFD reads, if configured so, mod_security log file, and when it sees that one IP has been blocked LF_MODSEC times in 5 minutes it blocks the IP, either permanently or temporarily.
    So it works so that mod_security is blocking something, in this case Googlebot, CSF/LFD sees those blocks in mod_security log, and blocks the used IP in the firewall.
    In this case, mod_security is not blocking Googlebot because it is Googlebut, but because Googlebot is doing something that seems suspicious. If you can't prevent mod_security from doing that, you should at least make sure that Googlebot is not blocked permanently.

    In our servers we don't have LF_MODESC enabled, but if it was, it would not do permanent blocking.
     
  11. Another Blogger

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Thanks again for your help. So if I disable LF_MODSEC, Mod_Security will still check IP addresses based upon its rules and if it finds something malicious, it'll block it. Right?
     
  12. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    18
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    Yes that's right.
     
  13. Another Blogger

    Joined:
    Jan 4, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    ^^ Thanks for all your help. :)
     
  14. Indianets

    Indianets Well-Known Member

    Joined:
    Jun 13, 2008
    Messages:
    65
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Sorry for my delayed arrival.

    If you are using CSF, you can add .googlebot.com to "/etc/csf/csf.rignore"

    Code:
    .googlebot.com
    - Vijay
     
    #14 Indianets, Jan 9, 2013
    Last edited: Jan 9, 2013
  15. crazyaboutlinux

    crazyaboutlinux Well-Known Member

    Joined:
    Nov 3, 2007
    Messages:
    938
    Likes Received:
    0
    Trophy Points:
    16
    It's a nice idea
     
Loading...

Share This Page