The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Search engine crawl +Virtual directories [moved]

Discussion in 'General Discussion' started by sirloin, Jul 16, 2005.

  1. sirloin

    sirloin Registered

    Joined:
    Jul 15, 2005
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Search engine crawl +Virtual directories

    I have my main site with 30 or so virtual directories under it. I have a robots.txt file in each one of the virtual directories.

    What I noticed on some search engines is that some listings are showing up under the main sites as a sub folder instead of its actual www.virtualdirectory.com name.

    If i put a robots.txt file in the main sites root and tell it not to crawl folder(s) /virtualdirectory... /virtualdirectory1... /virtualdirectory2 etc., will that block any search engine crawl from going to actual names?:
    www.virtualdirectory.com www.virtualdirectory1.com www.virtualdirectory2.com.

    Right now www.mainsite.com/virtualdirectory/index.html stinks because all of the image links are broken and it might be turing off some visitors from coming back again.

    is this a robots.txt issue or a .htaccess issue? and whichever how can i overcome this and fix it so that search engines see the correct site and not through the main account name.

    Cheers
    Cliff
    Los Angeles
    Bros before Ho's
     
  2. Izzee

    Izzee Well-Known Member

    Joined:
    Feb 6, 2004
    Messages:
    469
    Likes Received:
    0
    Trophy Points:
    16
    Robots.txt file is not much use when you want these bots to do as they are told. It only needs to be placed in the web root directory anyway. Some bots read the file and others make out they read it and others don't even bother to read it.

    .htaccess is where you can add directives that will redirect visitors to the pages you intended them to see after the bots have not done as you have requested them to do.

    Check out this page Apache Directives for an explanation of some of these directives that might suit your particular needs. Great reading and resource.
    Code:
    Redirect permanent /virtualdirectory/index.htm http://virtualdirectory.com/index.htm
    Redirect permanent /virtualdirectory/ http://virtualdirectory.com/
    The above might be what you should look at and test to see if it works. Trial and error will be part of the .htaccess learning curve. :)
    HTH
     
    #2 Izzee, Jul 16, 2005
    Last edited: Jul 16, 2005
  3. sirloin

    sirloin Registered

    Joined:
    Jul 15, 2005
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Izzy, thanks for the Quick reply..(17 hours ahead tomorrow) everyone in US is asleep now. I'll test it.
    BTW used to live in Surfers Paradise QLD, hope to return...for good. used ion.com.au back in early 90's.
    Cliff
     
  4. Izzee

    Izzee Well-Known Member

    Joined:
    Feb 6, 2004
    Messages:
    469
    Likes Received:
    0
    Trophy Points:
    16
    Cliff, Central Qld - Capricorn Coast :) iprimus.com.au and Surfers is a bonza spot. Good luck with your plans to return. You won't regret it :D
    Hope the above works out for your problem, let us know here. There may be others on the board that could suggest alternatives also.

    You can use regex in the redirect directive if you wish. ;)
    Example:
    Code:
    RedirectMatch /virtualdirectory/(.*)\.html$ http://virtualdirectory.com
    You could use relative links for your images etc. instead of direct links.
    eg. virtualdirectory/images/image.png instead of http://virtualdirectory.com/images/image.png

    The advantage of using .htaccess is the proper url will be showing in the visitors browser and if they choose to bookmark then they will have the correct address not a bot botch up. ;)

    These are a few of the ways to have things happen the way you want them. Being in control is a nice warm fuzzy feeling :)
     
    #4 Izzee, Jul 16, 2005
    Last edited: Jul 16, 2005
  5. sirloin

    sirloin Registered

    Joined:
    Jul 15, 2005
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
  6. sirloin

    sirloin Registered

    Joined:
    Jul 15, 2005
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    After a few days? things aren't so good - in Traffic Logs

    Hmmm seems that I'm now not getting any traffic since the change? I've deleted the changes in my htacccess file but cPanel seems to keep the htaccess changes even tho they do not show up when editing htaccess from the cpanel file manager.

    My Hosting provider is looking into it.

    The problem with those redirects is that they only redirect to the home page.

    Would there be any suggestions that would keep the end user on the same path redicecting to the exact original file requested.

    also tried with trailing /$1

    RedirectMatch /virtualdirectory/(.*)\.html$ http://virtualdirectory.com/$1

    Day Number of visits Pages Hits Bandwidth

    17 Jul 2005 13 19 424 1.24 MB
    18 Jul 2005 17 20 293 1.10 MB
    19 Jul 2005 0 0 0 0
    20 Jul 2005 0 0 0 0
    21 Jul 2005 1 1 1 9.03 KB
    22 Jul 2005 0 0 0 0
    23 Jul 2005 0 0 0 0
    24 Jul 2005 0 0 0 0
    25 Jul 2005 0 0 0 0
     
Loading...

Share This Page