The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to use httpd alias to show a robots.txt file only if robots.txt does not exist.

Discussion in 'General Discussion' started by nat, Jun 25, 2007.

  1. nat

    nat Well-Known Member

    Joined:
    Jan 16, 2003
    Messages:
    204
    Likes Received:
    0
    Trophy Points:
    16
    Is it possible to use alias or somethign else in httpd to show a different file ONLY if the file does not exist?

    Example:

    1. I count more than 50 Slurp requests from many different IPs coming in just 5 seconds. From time to time it causes a high load.

    2. ONLY for people who don't have a robots.txt file I want to setup an alias to slow slurp down just a little bit.

    Alias /robots.txt /home/myserver/public_html/robots.txt

    robots.txt on myserver contains:

    User-agent: Slurp
    Crawl-delay: 2

    The above kills other peoples robots.txt file so that cannot be used.

    3. For people who already have a robots.txt file, what is in their file is what should be used.



    Right now I can only get it one way or the other. Does anyone know a way to get apache to show a file only if the person doesn't already have a file by the same name in their directory?
     
Loading...

Share This Page