Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

SOLVED Development URL indexed on Google

Discussion in 'General Discussion' started by MichaelW., Feb 13, 2019.

  1. MichaelW.

    MichaelW. Member

    Joined:
    May 31, 2018
    Messages:
    24
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    Sweden
    cPanel Access Level:
    Root Administrator
    Hello!

    In our WHM/cPanel environments we work as follows:
    1. Set up development account on WHM.
    2. Build the site using cPanel and Wordpress.
    3. When the website is done, we use "Park Domain" in WHM to publish it with the right domain.

    This all works great, but we have noticed one problem.
    All the development URLs are indexed by Google. We do not want this.

    They are not supposed to be shown anywhere outside of the organisation.

    I need some advice on how to fix this. :)
     
  2. rpvw

    rpvw Well-Known Member

    Joined:
    Jul 18, 2013
    Messages:
    1,088
    Likes Received:
    441
    Trophy Points:
    113
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    You realistically only have a couple of options:
    • Password protect the root folder of your development installation cPanel > Files > Directory Privacy
    • Add a 'robots.txt' file (will work for Google, but not all robots/crawlers respect it)
    • Use an .htaccess file entry to only allow access from certain IPs (will only work if the authorised IPs are static)

    Example robots.txt file content to block all robots/crawlers that respect it:
    Code:
    User-agent: *
    Disallow: /
    
    or to disallow them from a specific folder
    Code:
    User-agent: *
    Disallow: /yourfolder/
    To learn more about robots.txt - just Google for it :-D
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. MichaelW.

    MichaelW. Member

    Joined:
    May 31, 2018
    Messages:
    24
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    Sweden
    cPanel Access Level:
    Root Administrator
    Hello!

    Thank you for the information.

    Correct me if I am wrong, but our published websites, the ones that has been parked, use the same files as the development websites.
    So if I go in and add robots.txt, won't it affect both the live website and the development site?

    Same thing with adding a password protection, won't it also affect the live website or is there some kind of workaround?
     
  4. rpvw

    rpvw Well-Known Member

    Joined:
    Jul 18, 2013
    Messages:
    1,088
    Likes Received:
    441
    Trophy Points:
    113
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    When you have finished development of the site and it is ready to go 'live', just remove whatever protection you used.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. MichaelW.

    MichaelW. Member

    Joined:
    May 31, 2018
    Messages:
    24
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    Sweden
    cPanel Access Level:
    Root Administrator
    OK, so Google won't index it while it has protection, and when it is live, it won't be seen by Google?

    Also, about the robots.txt, did I get it right or am I thinking wrong here?
     
  6. rpvw

    rpvw Well-Known Member

    Joined:
    Jul 18, 2013
    Messages:
    1,088
    Likes Received:
    441
    Trophy Points:
    113
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    When you have finished the development, change the content of the robots.txt file to read
    Code:
    User-agent: *
    Disallow:
    Note the Disallow line line is empty which means that all URLs can be indexed

    See www.robotstxt.org/orig.html#format for full details and syntax
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,815
    Likes Received:
    443
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    I agree with @rpvw in this one the robots.txt would be the best solution disallow any bots from indexing until you're ready to put the site live.
    Password protecting isn't a bad idea either and neither would affect the live site.

    Another solution might be to use a domain that isn't live and modify your local hosts file.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    MichaelW. likes this.
  8. MichaelW.

    MichaelW. Member

    Joined:
    May 31, 2018
    Messages:
    24
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    Sweden
    cPanel Access Level:
    Root Administrator
    Thanks for explaining, and thanks for the help. :)
    Will take this answer as the solution.
     
    cPanelLauren likes this.
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice