Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

How to use httpd alias to show a robots.txt file only if robots.txt does not exist.

Discussion in 'General Discussion' started by nat, Jun 25, 2007.

  1. nat

    nat Well-Known Member

    Joined:
    Jan 16, 2003
    Messages:
    210
    Likes Received:
    0
    Trophy Points:
    166
    Is it possible to use alias or somethign else in httpd to show a different file ONLY if the file does not exist?

    Example:

    1. I count more than 50 Slurp requests from many different IPs coming in just 5 seconds. From time to time it causes a high load.

    2. ONLY for people who don't have a robots.txt file I want to setup an alias to slow slurp down just a little bit.

    Alias /robots.txt /home/myserver/public_html/robots.txt

    robots.txt on myserver contains:

    User-agent: Slurp
    Crawl-delay: 2

    The above kills other peoples robots.txt file so that cannot be used.

    3. For people who already have a robots.txt file, what is in their file is what should be used.



    Right now I can only get it one way or the other. Does anyone know a way to get apache to show a file only if the person doesn't already have a file by the same name in their directory?
     
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice