Whitelisting the Search Engine Spiders

[Avalon]

Today I noticed that mod_security blocked valid Google spiders from accessing some of my servers and while it was relatively easy to unblock them. Then I proceed to add the "big name" search engines to the CSF ignore list but I know this list is incomplete:

.googlebot.com
.crawl.yahoo.net
.search.msn.com

Does anyone have a listing of what domains are used for other popular search engines like DuckDuckGo, Yandex, Baidu, etc.

I haven't had false positives with any of the other engines yet but better to be prepared.

I didn't think it would be so hard to find a listing of crawler domains to be honest.

 

[cPanelMichael]

Hello,

You may find this third-party URL helpful:

http://www.robotstxt.org/db.html

Thank you.

 

[Avalon]

Hello,

You may find this third-party URL helpful:

http://www.robotstxt.org/db.html

Thank you.

Thank you. It's a shame they don't have just a text list but this will allow me to find [some] of the more prominent ones and make sure they don't get blocked by firewalls.