The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Whitelisting the Search Engine Spiders

Discussion in 'Security' started by Avalon, May 21, 2015.

  1. Avalon

    Avalon Member

    Joined:
    Apr 27, 2015
    Messages:
    19
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United States
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Today I noticed that mod_security blocked valid Google spiders from accessing some of my servers and while it was relatively easy to unblock them. Then I proceed to add the "big name" search engines to the CSF ignore list but I know this list is incomplete:

    Code:
    .googlebot.com
    .crawl.yahoo.net
    .search.msn.com
    Does anyone have a listing of what domains are used for other popular search engines like DuckDuckGo, Yandex, Baidu, etc.

    I haven't had false positives with any of the other engines yet but better to be prepared.

    I didn't think it would be so hard to find a listing of crawler domains to be honest.
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,852
    Likes Received:
    675
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  3. Avalon

    Avalon Member

    Joined:
    Apr 27, 2015
    Messages:
    19
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United States
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Thank you. It's a shame they don't have just a text list but this will allow me to find [some] of the more prominent ones and make sure they don't get blocked by firewalls.
     

Share This Page