Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Awstats robots reports

Discussion in 'General Discussion' started by Patrickintheforest, Aug 3, 2018.

Tags:
  1. Patrickintheforest

    Patrickintheforest Registered

    Joined:
    Aug 3, 2018
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Essex
    cPanel Access Level:
    Website Owner
    In my robots.txt file I have denied Yandexbot
    Under robots spider visitors, hits, I see for example
    Yandexbot 0/2
    Does this mean that the bot tried twice but was blocked.
    If so I know my Yandexbot deny has worked
    Patrick
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    44,344
    Likes Received:
    1,852
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello @Patrickintheforest,

    The access attempt itself from the bot/crawler is still logged, and that's what Awstats is showing you. Note the robots.txt file won't technically prevent a crawler/bot from accessing your website files. There's some useful information on this topic at:

    Block bots by specific urls with robots.txt in php apache environment

    If you have root access to the server, it's sometimes a better idea to block the bot in your firewall rules. There's a thread on this topic at:

    SOLVED - Blocking bad bots

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Patrickintheforest

    Patrickintheforest Registered

    Joined:
    Aug 3, 2018
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Essex
    cPanel Access Level:
    Website Owner
    Thank you cpanelmichael.
    My bandwidth has been drastically reduced since the deny.
     
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice