The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking Googlebot & Other Spiders

Discussion in 'General Discussion' started by grayloon, Apr 30, 2010.

  1. grayloon

    grayloon Well-Known Member

    Joined:
    Oct 31, 2007
    Messages:
    98
    Likes Received:
    2
    Trophy Points:
    8
    Location:
    Evansville, IN
    cPanel Access Level:
    Root Administrator
    Twitter:
    I'm running WHM on my development server, and Google has indexed several sites there. Can I do something server-wide to block access from the search engine spider user agents like googlebot, msnbot, etc.?
     
  2. Spiral

    Spiral BANNED

    Joined:
    Jun 24, 2005
    Messages:
    2,023
    Likes Received:
    7
    Trophy Points:
    0
    Yes you can ....

    1. For a development site, stick a password on the front

    2. Setup a "robots.txt" to voluntarily limit access for "good" bots

    3. You can block the "User Agent" for the bots in .htaccess or httpd.conf

    4. You can block the CIDR ranges for the bots in /etc/hosts.deny, iptables, csf / apf, httpd.conf, and / or .htaccess
     
Loading...

Share This Page