tribulant

Member
Mar 3, 2014
12
1
3
cPanel Access Level
Website Owner
Good day,

It seems like Google is crawling our website on it's IP address, hostname, etc. at Amazon EC2 eg:

example.com/blog/
ec2-54-88-xx-xxx.compute-1.amazonaws.com/blog/
54.88.xx.xxx/blog/

How can we make sure only the domain is crawled and accessible?
 
Last edited by a moderator:

rpvw

Well-Known Member
Jul 18, 2013
1,088
446
113
Spain
cPanel Access Level
Root Administrator
If your domain name resolves to an IP (and it must for anyone to be able to call the website using the domain name) the DNS is probably correct.

I am intrigued as to how the DNS is resolving to the /blog website using the hostname and the IP in the first place. Perhaps this would better be addressed by your web host.

There is no way of really influencing what the various robots can crawl other than by using a robots.txt file (or include crafted meta tags into the headers of the parsed web page) to exclude certain folders/files from being searched.

As far as I know, if you put an IP into this file, it would be ignored, and you need to remember that not all the robots respect the robots.txt or meta tag conventions anyway !

Hope this helps.
 
Last edited:
  • Like
Reactions: tribulant

tribulant

Member
Mar 3, 2014
12
1
3
cPanel Access Level
Website Owner
If your domain name resolves to an IP (and it must for anyone to be able to call the website using the domain name) the DNS is probably correct.

I am intrigued as to how the DNS is resolving to the /blog website using the hostname and the IP in the first place. Perhaps this would better be addressed by your web host.

There is no way of really influencing what the various robots can crawl other than by using a robots.txt file (or include crafted meta tags into the headers of the parsed web page) to exclude certain folders/files from being searched.

As far as I know, if you put an IP into this file, it would be ignored, and you need to remember that not all the robots respect the robots.txt or meta tag conventions anyway !

Hope this helps.
Thanks I will continue to investigate and post back here once I have a solution to this.

In the meanwhile, I hope that Google doesn't penalize for this.
 

rpvw

Well-Known Member
Jul 18, 2013
1,088
446
113
Spain
cPanel Access Level
Root Administrator
Last edited:

kssuhesh

Member
Jan 11, 2009
19
5
128
Kochi
cPanel Access Level
Root Administrator
Twitter
Hello,

You should add a 301 redirect to one url, once you 301 redirect everything, all the urls currently indexed with the IP will eventually drop out in favor of the same urls on the domain to which you redirected them. You can use the redirect in the following link for doing the same.
- Removed -
 
Last edited by a moderator:

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,911
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Hello,

You can follow the steps referenced below (partly taken from our SSL FAQ document) to prevent access to your domain name over it's IP address.

1. Navigate to WHM's Include Editor interface (WHM Home >> Service Configuration >> Apache Configuration >> Include Editor).

2. Select the Pre Virtual Host Include option.

3. Select the Apache version from the menu. We recommend that you select All Versions.

4. Enter the following text in the available text box:

Code:
<VirtualHost IPADDRESS:80>
  ServerName HOSTNAME
  DocumentRoot /usr/local/apache/htdocs
  ServerAdmin EMAIL
  <IfModule mod_suphp.c>
    suPHP_UserGroup nobody nobody
  </IfModule>
 </VirtualHost>

<VirtualHost IPADDRESS:443>
  ServerName HOSTNAME
  DocumentRoot /usr/local/apache/htdocs
  ServerAdmin EMAIL
  <IfModule mod_suphp.c>
    suPHP_UserGroup nobody nobody
  </IfModule>
    SSLEngine on
    SSLCertificateFile SSLCERTIFICATEFILE
    SSLCertificateKeyFile YOUR-SSLCERTIFICATEKEYFILE
 </VirtualHost>
Note:

This example uses the following values:
  • IPADDRESS represents your account's dedicated IP address.
  • HOSTNAME represents your server's hostname.
  • EMAIL represents your contact email address.
  • SSLCERTIFICATEFILE represents the full file path to your SSL certificate.
  • SSLCERTIFICATEKEYFILE represents the full file path to your SSL certificate's key.
5. Click Proceed
6. Click Update.

Thank you.