Search engine crawl +Virtual directories [moved]

[sirloin]

I have my main site with 30 or so virtual directories under it. I have a robots.txt file in each one of the virtual directories.

What I noticed on some search engines is that some listings are showing up under the main sites as a sub folder instead of its actual www.virtualdirectory.com name.

If i put a robots.txt file in the main sites root and tell it not to crawl folder(s) /virtualdirectory... /virtualdirectory1... /virtualdirectory2 etc., will that block any search engine crawl from going to actual names?:
www.virtualdirectory.com www.virtualdirectory1.com www.virtualdirectory2.com.

Right now www.mainsite.com/virtualdirectory/index.html stinks because all of the image links are broken and it might be turing off some visitors from coming back again.

is this a robots.txt issue or a .htaccess issue? and whichever how can i overcome this and fix it so that search engines see the correct site and not through the main account name.

Cheers
Cliff
Los Angeles
Bros before Ho's

[Izzee]

...is this a robots.txt issue or a .htaccess issue? and whichever how can i overcome this and fix it so that search engines see the correct site and not through the main account name...

Robots.txt file is not much use when you want these bots to do as they are told. It only needs to be placed in the web root directory anyway. Some bots read the file and others make out they read it and others don't even bother to read it.

.htaccess is where you can add directives that will redirect visitors to the pages you intended them to see after the bots have not done as you have requested them to do.

Check out this page Apache Directives for an explanation of some of these directives that might suit your particular needs. Great reading and resource.

Redirect permanent /virtualdirectory/index.htm http://virtualdirectory.com/index.htm
Redirect permanent /virtualdirectory/ http://virtualdirectory.com/

The above might be what you should look at and test to see if it works. Trial and error will be part of the .htaccess learning curve.
HTH

[sirloin]

Izzy, thanks for the Quick reply..(17 hours ahead tomorrow) everyone in US is asleep now. I'll test it.
BTW used to live in Surfers Paradise QLD, hope to return...for good. used ion.com.au back in early 90's.
Cliff

[Izzee]

Cliff, Central Qld - Capricorn Coast iprimus.com.au and Surfers is a bonza spot. Good luck with your plans to return. You won't regret it
Hope the above works out for your problem, let us know here. There may be others on the board that could suggest alternatives also.

You can use regex in the redirect directive if you wish.
Example:

RedirectMatch /virtualdirectory/(.*)\.html$ http://virtualdirectory.com

You could use relative links for your images etc. instead of direct links.
eg. virtualdirectory/images/image.png instead of http://virtualdirectory.com/images/image.png

The advantage of using .htaccess is the proper url will be showing in the visitors browser and if they choose to bookmark then they will have the correct address not a bot botch up.

These are a few of the ways to have things happen the way you want them. Being in control is a nice warm fuzzy feeling

[sirloin]

Used the following
RedirectMatch /virtualdirectory/(.*)\.html$ http://virtualdirectory.com

Works like a charm

Maybe they could incorporate this into cPanels scripts? when defining a new site.

Cliff
Figured your third suggestion was the best since you had the most time to think about it.

Owe you a case of V.B.

Cheers!
http://www.fosters.com.au/enjoy/beer...ria_bitter.htm

[sirloin]

Hmmm seems that I'm now not getting any traffic since the change? I've deleted the changes in my htacccess file but cPanel seems to keep the htaccess changes even tho they do not show up when editing htaccess from the cpanel file manager.

My Hosting provider is looking into it.

The problem with those redirects is that they only redirect to the home page.

Would there be any suggestions that would keep the end user on the same path redicecting to the exact original file requested.

also tried with trailing /$1

RedirectMatch /virtualdirectory/(.*)\.html$ http://virtualdirectory.com/$1

Day Number of visits Pages Hits Bandwidth

17 Jul 2005 13 19 424 1.24 MB
18 Jul 2005 17 20 293 1.10 MB
19 Jul 2005 0 0 0 0
20 Jul 2005 0 0 0 0
21 Jul 2005 1 1 1 9.03 KB
22 Jul 2005 0 0 0 0
23 Jul 2005 0 0 0 0
24 Jul 2005 0 0 0 0
25 Jul 2005 0 0 0 0