Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

IP Address Crawlable

Discussion in 'Bind / DNS / Nameserver Issues' started by tribulant, Sep 18, 2017.

  1. tribulant

    tribulant Member

    Joined:
    Mar 3, 2014
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Website Owner
    Good day,

    It seems like Google is crawling our website on it's IP address, hostname, etc. at Amazon EC2 eg:

    example.com/blog/
    ec2-54-88-xx-xxx.compute-1.amazonaws.com/blog/
    54.88.xx.xxx/blog/

    How can we make sure only the domain is crawled and accessible?
     
    #1 tribulant, Sep 18, 2017
    Last edited by a moderator: Sep 18, 2017
  2. rpvw

    rpvw Well-Known Member

    Joined:
    Jul 18, 2013
    Messages:
    334
    Likes Received:
    95
    Trophy Points:
    28
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    If your domain name resolves to an IP (and it must for anyone to be able to call the website using the domain name) the DNS is probably correct.

    I am intrigued as to how the DNS is resolving to the /blog website using the hostname and the IP in the first place. Perhaps this would better be addressed by your web host.

    There is no way of really influencing what the various robots can crawl other than by using a robots.txt file (or include crafted meta tags into the headers of the parsed web page) to exclude certain folders/files from being searched.

    As far as I know, if you put an IP into this file, it would be ignored, and you need to remember that not all the robots respect the robots.txt or meta tag conventions anyway !

    Hope this helps.
     
    #2 rpvw, Sep 18, 2017
    Last edited: Sep 18, 2017
    tribulant likes this.
  3. tribulant

    tribulant Member

    Joined:
    Mar 3, 2014
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Website Owner
    Thanks I will continue to investigate and post back here once I have a solution to this.

    In the meanwhile, I hope that Google doesn't penalize for this.
     
  4. rpvw

    rpvw Well-Known Member

    Joined:
    Jul 18, 2013
    Messages:
    334
    Likes Received:
    95
    Trophy Points:
    28
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    #4 rpvw, Sep 19, 2017
    Last edited: Sep 19, 2017
  5. kssuhesh

    kssuhesh Member

    Joined:
    Jan 11, 2009
    Messages:
    13
    Likes Received:
    3
    Trophy Points:
    128
    Location:
    Kochi
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    You should add a 301 redirect to one url, once you 301 redirect everything, all the urls currently indexed with the IP will eventually drop out in favor of the same urls on the domain to which you redirected them. You can use the redirect in the following link for doing the same.
    - Removed -
     
    #5 kssuhesh, Sep 19, 2017
    Last edited by a moderator: Sep 19, 2017
  6. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,425
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    You can follow the steps referenced below (partly taken from our SSL FAQ document) to prevent access to your domain name over it's IP address.

    1. Navigate to WHM's Include Editor interface (WHM Home >> Service Configuration >> Apache Configuration >> Include Editor).

    2. Select the Pre Virtual Host Include option.

    3. Select the Apache version from the menu. We recommend that you select All Versions.

    4. Enter the following text in the available text box:

    Code:
    <VirtualHost IPADDRESS:80>
      ServerName HOSTNAME
      DocumentRoot /usr/local/apache/htdocs
      ServerAdmin EMAIL
      <IfModule mod_suphp.c>
        suPHP_UserGroup nobody nobody
      </IfModule>
     </VirtualHost>
    
    <VirtualHost IPADDRESS:443>
      ServerName HOSTNAME
      DocumentRoot /usr/local/apache/htdocs
      ServerAdmin EMAIL
      <IfModule mod_suphp.c>
        suPHP_UserGroup nobody nobody
      </IfModule>
        SSLEngine on
        SSLCertificateFile SSLCERTIFICATEFILE
        SSLCertificateKeyFile YOUR-SSLCERTIFICATEKEYFILE
     </VirtualHost>
    Note:

    This example uses the following values:
    • IPADDRESS represents your account's dedicated IP address.
    • HOSTNAME represents your server's hostname.
    • EMAIL represents your contact email address.
    • SSLCERTIFICATEFILE represents the full file path to your SSL certificate.
    • SSLCERTIFICATEKEYFILE represents the full file path to your SSL certificate's key.
    5. Click Proceed
    6. Click Update.

    Thank you.
     
Loading...

Share This Page