Blocking Googlebot & Other Spiders

Spiral

BANNED
Jun 24, 2005
2,018
8
193
Yes you can ....

1. For a development site, stick a password on the front

2. Setup a "robots.txt" to voluntarily limit access for "good" bots

3. You can block the "User Agent" for the bots in .htaccess or httpd.conf

4. You can block the CIDR ranges for the bots in /etc/hosts.deny, iptables, csf / apf, httpd.conf, and / or .htaccess