Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

IP Address Crawlable

Discussion in 'Bind/DNS/Nameserver' started by tribulant, Sep 18, 2017.

  1. tribulant

    tribulant Member

    Joined:
    Mar 3, 2014
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Website Owner
    Good day,

    It seems like Google is crawling our website on it's IP address, hostname, etc. at Amazon EC2 eg:

    example.com/blog/
    ec2-54-88-xx-xxx.compute-1.amazonaws.com/blog/
    54.88.xx.xxx/blog/

    How can we make sure only the domain is crawled and accessible?
     
    #1 tribulant, Sep 18, 2017
    Last edited by a moderator: Sep 18, 2017
  2. rpvw

    rpvw Well-Known Member

    Joined:
    Jul 18, 2013
    Messages:
    839
    Likes Received:
    302
    Trophy Points:
    113
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    If your domain name resolves to an IP (and it must for anyone to be able to call the website using the domain name) the DNS is probably correct.

    I am intrigued as to how the DNS is resolving to the /blog website using the hostname and the IP in the first place. Perhaps this would better be addressed by your web host.

    There is no way of really influencing what the various robots can crawl other than by using a robots.txt file (or include crafted meta tags into the headers of the parsed web page) to exclude certain folders/files from being searched.

    As far as I know, if you put an IP into this file, it would be ignored, and you need to remember that not all the robots respect the robots.txt or meta tag conventions anyway !

    Hope this helps.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #2 rpvw, Sep 18, 2017
    Last edited: Sep 18, 2017
    tribulant likes this.
  3. tribulant

    tribulant Member

    Joined:
    Mar 3, 2014
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Website Owner
    Thanks I will continue to investigate and post back here once I have a solution to this.

    In the meanwhile, I hope that Google doesn't penalize for this.
     
  4. rpvw

    rpvw Well-Known Member

    Joined:
    Jul 18, 2013
    Messages:
    839
    Likes Received:
    302
    Trophy Points:
    113
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #4 rpvw, Sep 19, 2017
    Last edited: Sep 19, 2017
  5. kssuhesh

    kssuhesh Member

    Joined:
    Jan 11, 2009
    Messages:
    16
    Likes Received:
    5
    Trophy Points:
    128
    Location:
    Kochi
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    You should add a 301 redirect to one url, once you 301 redirect everything, all the urls currently indexed with the IP will eventually drop out in favor of the same urls on the domain to which you redirected them. You can use the redirect in the following link for doing the same.
    - Removed -
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #5 kssuhesh, Sep 19, 2017
    Last edited by a moderator: Sep 19, 2017
  6. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,803
    Likes Received:
    1,898
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    You can follow the steps referenced below (partly taken from our SSL FAQ document) to prevent access to your domain name over it's IP address.

    1. Navigate to WHM's Include Editor interface (WHM Home >> Service Configuration >> Apache Configuration >> Include Editor).

    2. Select the Pre Virtual Host Include option.

    3. Select the Apache version from the menu. We recommend that you select All Versions.

    4. Enter the following text in the available text box:

    Code:
    <VirtualHost IPADDRESS:80>
      ServerName HOSTNAME
      DocumentRoot /usr/local/apache/htdocs
      ServerAdmin EMAIL
      <IfModule mod_suphp.c>
        suPHP_UserGroup nobody nobody
      </IfModule>
     </VirtualHost>
    
    <VirtualHost IPADDRESS:443>
      ServerName HOSTNAME
      DocumentRoot /usr/local/apache/htdocs
      ServerAdmin EMAIL
      <IfModule mod_suphp.c>
        suPHP_UserGroup nobody nobody
      </IfModule>
        SSLEngine on
        SSLCertificateFile SSLCERTIFICATEFILE
        SSLCertificateKeyFile YOUR-SSLCERTIFICATEKEYFILE
     </VirtualHost>
    Note:

    This example uses the following values:
    • IPADDRESS represents your account's dedicated IP address.
    • HOSTNAME represents your server's hostname.
    • EMAIL represents your contact email address.
    • SSLCERTIFICATEFILE represents the full file path to your SSL certificate.
    • SSLCERTIFICATEKEYFILE represents the full file path to your SSL certificate's key.
    5. Click Proceed
    6. Click Update.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice