The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

OWASP blocking seo bots?

Discussion in 'Security' started by keat63, Feb 17, 2015.

  1. keat63

    keat63 Well-Known Member

    Joined:
    Nov 20, 2014
    Messages:
    765
    Likes Received:
    20
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Does OWASP block SEO bots.
    I've seen repeated blocks today, and upon further investigation some have been google bot, MSN Bot and Bing Bot, all on rule 981138
     
  2. keat63

    keat63 Well-Known Member

    Joined:
    Nov 20, 2014
    Messages:
    765
    Likes Received:
    20
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    I guess the following email answers that then.

    Over the last 24 hours, Googlebot encountered 66 errors while attempting to access your robots.txt.
    To ensure that we didn't crawl any pages listed in that file, we postponed our crawl.
    Your site's overall robots.txt error rate is 64.1%.

    Before i disable yet another rule, are there any tweaks to allow good bots ?
     
  3. xanubi

    xanubi Well-Known Member

    Joined:
    Jun 28, 2006
    Messages:
    86
    Likes Received:
    1
    Trophy Points:
    8
    OWASP have alot of false-positives, and some rules are a little broken.
    Honeyproject is one of them, and that's why you're seeing that.

    What i did was disable that rule and create a new one on cpanel.user.conf that really works.
     
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,807
    Likes Received:
    667
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    You can disable the rule if it's a false positive. Please note the following quote from Brian on the other OWASP forums thread:

    Thank you.
     
  5. keat63

    keat63 Well-Known Member

    Joined:
    Nov 20, 2014
    Messages:
    765
    Likes Received:
    20
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Logging in to Google WebMaster tools and the two domains that i'm currently interested in couldn't be crawled, so I had to firstly disable that particular rule, and then delete all the ModSecurity entries listed against that rule.
    I dare say i could have isolated GoogleBot down to a particular entry, however, how many other good bots were blocked.

    I'm guessing there are about 100 rules, 5 of which i've disabled for one reason or another.
    if 95 others are helping to keep me safe than i can't grumble
     
Loading...

Share This Page