Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Whitelisting the Search Engine Spiders

Discussion in 'Security' started by Avalon, May 21, 2015.

  1. Avalon

    Avalon Member

    Joined:
    Apr 27, 2015
    Messages:
    19
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United States
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Today I noticed that mod_security blocked valid Google spiders from accessing some of my servers and while it was relatively easy to unblock them. Then I proceed to add the "big name" search engines to the CSF ignore list but I know this list is incomplete:

    Code:
    .googlebot.com
    .crawl.yahoo.net
    .search.msn.com
    Does anyone have a listing of what domains are used for other popular search engines like DuckDuckGo, Yandex, Baidu, etc.

    I haven't had false positives with any of the other engines yet but better to be prepared.

    I didn't think it would be so hard to find a listing of crawler domains to be honest.
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,711
    Likes Received:
    1,791
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    Avalon likes this.
  3. Avalon

    Avalon Member

    Joined:
    Apr 27, 2015
    Messages:
    19
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United States
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Thank you. It's a shame they don't have just a text list but this will allow me to find [some] of the more prominent ones and make sure they don't get blocked by firewalls.
     
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice