The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

OWASP Blocking Google

Discussion in 'Security' started by keat63, Mar 18, 2015.

  1. keat63

    keat63 Well-Known Member

    Joined:
    Nov 20, 2014
    Messages:
    765
    Likes Received:
    20
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    It seems every few days, i'm seeing Goolge being blocked by one OWASP rule after the other.
    Some of the errors look quite alarming.
    Of course blocking good bots is not good for SEO, so rather than opening all these rules, is there a way to configure this to allow the good bots.
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello,

    I suggest using "WHM Home » Security Center » Report ModSecurity Hit" to report the rule if it's not working as intended. Or, you may want to disable the rule completely. I'll leave this thread open for other users to add feedback regarding adding a custom rule to allow specific bots.

    Thank you.
     
  3. mariusfv

    mariusfv Member

    Joined:
    Mar 24, 2013
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    cPanelMichael: from cpanel documentation: https://documentation.cpanel.net/display/ALD/ModSecurity+Tools#ModSecurityTools-Reportarule

    Report a rule
    If you find a problem with a vendor's rule, perform the following steps to report the issue to the rule's vendor:
    Locate the hit that the rule generated in the Hits List and click More.
    Click Report this hit.
    Note: This option does not appear if the vendor does not accept reports

    So OWASP does not accept to report rule(latest WHM installed).

    I confirm OWASP Blocking Google


    And over 90% use CSF Firewall so they have ban Googlebot, in /var/log/lfd.log:

    Mar 19 03:12:55 server1 lfd[29570]: (mod_security) mod_security (id:981138) triggered by 66.249.67.124 (US/United States/crawl-66-249-67-124.googlebot.com): 5 in the last 3600 secs - *Blocked in csf* [LF_MODSEC]
    Mar 19 03:37:11 server1 lfd[1181]: (mod_security) mod_security (id:981140) triggered by 66.249.67.66 (US/United States/crawl-66-249-67-66.googlebot.com): 5 in the last 3600 secs - *Blocked in csf* [LF_MODSEC]

    We must check Googlebot IP's if they are banned in CSF:

    csf -g 66.249.67.124 (is one example of Google IP, you must check all)

    The output will display the Google bot IP is banned(as long in csf settings LF_MODSEC - READ MOD SECURITY LOGS) is activated.

    Remove Googlebot ip from banned list:

    csf -dr 66.249.67.124 (is one example of Google IP, you must check all)

    As long as CSF is set to read mod security log([LF_MODSEC]) they have for sure banned Google Bot on their servers.(and tons of websites from Google crawl)

    Temporary we can edit csf.rignore from CSF

    nano /etc/csf/csf.rignore

    and add line:
    .googlebot.com (In that CSF file is specified this example!)

    Save & Restart CSF.

    Disadvantage with /etc/csf/csf.rignore: The DNS lookups will increase server load (from my test).

    The problem must be solved on Mod Security side to allow Googlebot, Yahoobot and MSNbot.




     
    #3 mariusfv, Mar 19, 2015
    Last edited: Mar 19, 2015
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  5. mariusfv

    mariusfv Member

    Joined:
    Mar 24, 2013
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    cPanelMichael you have right, now working to submit.(I've submitted...)

    The big problem is that CSF + Mod Security still ban Google bot even if I add in /etc/csf/csf.rignore
    the line: .googlebot.com

    Mod security ID who ban GOOGLE BOT:
    mod_security (id:981138)
    mod_security (id:981140)

    Till problem is solved by OWASP know someone a Mod Security rule to accept GOOGLE BOT?
     
  6. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  7. quizknows

    quizknows Well-Known Member

    Joined:
    Oct 20, 2009
    Messages:
    942
    Likes Received:
    57
    Trophy Points:
    28
    cPanel Access Level:
    DataCenter Provider
    Depends what rules are being tripped. While disabling them will work, it might not be the best idea. That said, anything google is tripping is probably an erroneous or bad rule and relatively safe to whitelist.

    Anyway, whitelisting the googlebot user agent is a bad idea; lots of attack scripts use that hoping people have it whitelisted.

    What you would want to do is find a list of IPs that google bot is known to use, and a rule could be made from that to whitelist them. However, it is better to report or disable any rules google is tripping (AFTER making sure it's a legit google IP and not some bot stealing their user agent, since that's extremely common).
     
    cPanelDon likes this.
  8. vicos

    vicos Well-Known Member

    Joined:
    Apr 18, 2003
    Messages:
    62
    Likes Received:
    0
    Trophy Points:
    6
    I just installed OWASP and have the same issue with Rule 981138 blocking Googlebots.

    I did some checking and it is an error in the way the rule is written. 981138 checks the IP address against the Project HoneyPot database (a DNS blacklist). So, I contacted them to ask if they knew if they Google listed. They said yes, but they clearly identify it as a search crawler:

    Any time the last octet of the response is a 0, the IP is indicated as a search engine crawler.

    You can find more information here: http://www.projecthoneypot.org/httpbl_api.php

    So, anyway, the rules needs to be fixed to check for '0' in the last octet and let these IPs pass...

    Will reporting this here get this fixed or do I need to go somewhere else? Checking the DNS blacklist is very valuable, so having to disable it because of this glitch is a shame.
     
  9. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You will need to use the "WHM Home » Security Center » Report ModSecurity Hit" option to report the rule if it's not working as intended.

    Thank you.
     
Loading...

Share This Page