Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Blocking Googlebot & Other Spiders

Discussion in 'General Discussion' started by grayloon, Apr 30, 2010.

  1. grayloon

    grayloon Well-Known Member

    Joined:
    Oct 31, 2007
    Messages:
    107
    Likes Received:
    2
    Trophy Points:
    68
    Location:
    Evansville, IN
    cPanel Access Level:
    Root Administrator
    Twitter:
    I'm running WHM on my development server, and Google has indexed several sites there. Can I do something server-wide to block access from the search engine spider user agents like googlebot, msnbot, etc.?
     
  2. Spiral

    Spiral BANNED

    Joined:
    Jun 24, 2005
    Messages:
    2,023
    Likes Received:
    8
    Trophy Points:
    193
    Yes you can ....

    1. For a development site, stick a password on the front

    2. Setup a "robots.txt" to voluntarily limit access for "good" bots

    3. You can block the "User Agent" for the bots in .htaccess or httpd.conf

    4. You can block the CIDR ranges for the bots in /etc/hosts.deny, iptables, csf / apf, httpd.conf, and / or .htaccess
     
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice