The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Spider download protection

Discussion in 'General Discussion' started by Matsaki, May 5, 2011.

  1. Matsaki

    Matsaki Member

    Joined:
    Apr 23, 2011
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    Hi,
    I was looking and did not find a solution fir this. I want to i.e. protect my folder http://www.domain.com/images/ so that a spiders or any other can't browse to that catalog and download all the images.

    Also I would prefer if a spider or user could not browse through any catalogs.

    Is there a way through cPanel as the "Password Protect Directories" is not the same think as I understand?

    Thanks!
     
  2. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    194
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    You could enable Leech Protection from your cPanel, but I'm not sure your looking for that.
    You could place a plain index.html in the directory so that users viewing domain.com/images/ don't see the images, they'd see that index.html.
    You could use a robots.txt The Web Robots Pages
    You could edit the Apache config via WHM:
    WHM > Service Configuration > Apache Configuration > Global Configuration > Directory '/' Options

    If you password protect the directory, everyone visiting that directory would need the password. If you link to images in that directory for images on your website, users visiting your website will be asked for the password as well.
     
  3. Matsaki

    Matsaki Member

    Joined:
    Apr 23, 2011
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    I don't know what's it called and how it works. But in Linux servers there is a simple setting for this so that when I try to brose in to a /image directory I get the message

    And that setting makes this for all directories without a index page. As my site has a lot of party and album picture's I sometime have spiders who automatically downloads my images and upload them to their site. Also I don't like that people can browse through my directories.

    If I just knew the name of this setting I could google it and find out :confused:
     
  4. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    194
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    WHM > Service Configuration > Apache Configuration > Global Configuration > Directory '/' Options

    What options do you have set there, in this specific section?
     
  5. Matsaki

    Matsaki Member

    Joined:
    Apr 23, 2011
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    Thank you. I will ask the admin to have a look.
     
  6. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    If you only have cPanel access, then the option you are looking for per your description appears to be cPanel > Index Manager where you can select a directory such as public_html and then select "No Indexing" for that directory. This would then no longer allow any directory without an index page to be viewable in a browser.
     
  7. Matsaki

    Matsaki Member

    Joined:
    Apr 23, 2011
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    Ohh that sound good. But does that not prevent search engines from indexing HTML/PHP pages then too?
     
  8. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    No, it does not. It doesn't prevent them from spidering the images either, since this does not have to do with spidering content. It would prevent browsing content directly when no index page exists.

    Infopro already provided details on a robots.txt file to tell spiders what to not traverse.
     
  9. Matsaki

    Matsaki Member

    Joined:
    Apr 23, 2011
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    Many thanks. I will give it a try :)
     
  10. Matsaki

    Matsaki Member

    Joined:
    Apr 23, 2011
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    Aouch, I did a "No indexing" on the public_html directory, and now none of my domain works any longer. Even though I changed it back to "Default system settings" The error I get when I try to open/browse a domain is:

    What went wrong here??
     
  11. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    In File Manager, navigate to public_html and open up your .htaccess file to see the contents. Paste the contents here.
     
  12. Matsaki

    Matsaki Member

    Joined:
    Apr 23, 2011
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    Looks strange.


    IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*

    <Limit GET POST>
    order deny,allow
    deny from all
    allow from all
    </Limit>
    <Limit PUT DELETE>
    order deny,allow
    deny from all
    </Limit>
    AuthName scancyp.comOptions -Indexes
     
  13. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    It looks correct other than the last line. Please edit the last line to be the following (edit it in File Manager):

    Code:
    AuthName scancyp.com
    Options -Indexes
    Basically, the "Options -Indexes" didn't go to the next line for some reason when "No Indexing" was enabled.
     
  14. Matsaki

    Matsaki Member

    Joined:
    Apr 23, 2011
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    That almost gave me a heart attack. But now it's working exactly as I was looking for in above post. Don't know why the .htaccess was messed up. But the change you did helped and also in the cPanel it shows the correct "No indexing" now.

    I'm very grateful for the excellent and professional help here!
     
Loading...

Share This Page