Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Outgoing Traffic

Discussion in 'General Discussion' started by mehnihma, Mar 18, 2013.

  1. mehnihma

    mehnihma Well-Known Member

    Joined:
    Dec 15, 2012
    Messages:
    57
    Likes Received:
    1
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    Hi

    I have big Bandwidth usage in outgoing traffic, i think it is http, it is around 10GB a day. I do now know what is creating that traffic. I do not have a to visitors to have that kind of traffic.
    How can I see what is the problem?

    I Awstats I have found :
    Unknown robot (identified by 'bot*') 398,018+456 91.85 GB 18 Mar 2013 - 03:36

    Maybe that is the problem. How to stop unknown bots?

    Thanks
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #1 mehnihma, Mar 18, 2013
    Last edited: Mar 18, 2013
  2. storminternet

    storminternet Well-Known Member

    Joined:
    Nov 2, 2011
    Messages:
    460
    Likes Received:
    0
    Trophy Points:
    66
    cPanel Access Level:
    Root Administrator
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. mehnihma

    mehnihma Well-Known Member

    Joined:
    Dec 15, 2012
    Messages:
    57
    Likes Received:
    1
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    So do I :)

    For now I have added in public_html root in .htacces:

    Code:
    RewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
    RewriteCond %{HTTP_USER_AGENT} spider [OR]
    RewriteCond %{HTTP_USER_AGENT} bot [OR]
    RewriteCond %{HTTP_USER_AGENT} crawl [OR]
    RewriteCond %{HTTP_USER_AGENT} ^[Jj]ava [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Wget
    RewriteRule .? - [F,L]
    But is this enough?
    If I have something in sub-folder do I have to create it also there? Is there any better solution?

    Thanks
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. arunsv84

    arunsv84 Well-Known Member

    Joined:
    Oct 20, 2008
    Messages:
    373
    Likes Received:
    1
    Trophy Points:
    68
    Location:
    127.0.0.1
    cPanel Access Level:
    Root Administrator
    This rule seems to be fine. Or would be netter to place the rules in your apache config file so it will be blocked for all domains. Make sure to distill apache config or cPanel will overwrite the changes made to config file.

    You will get the list of bad bots just by performing a Google search. Add that rules as well.

    Cheers!!!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. mehnihma

    mehnihma Well-Known Member

    Joined:
    Dec 15, 2012
    Messages:
    57
    Likes Received:
    1
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    Hi what do you mean by distill apache config

    Also these rules did not help

    Code:
    Unknown robot (identified by 'bot*')	74,451	19.26 GB	02 Apr 2013 - 14:12
    I cannot block it
    How can I find IP of that bot and block it because it is eating my bandwidth

    When I add to .htacces
    [CODERewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
    RewriteCond %{HTTP_USER_AGENT} spider [OR]
    RewriteCond %{HTTP_USER_AGENT} bot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^bot* [OR]
    RewriteCond %{HTTP_USER_AGENT} ^bot [OR]
    RewriteCond %{HTTP_USER_AGENT} bot* [OR]
    RewriteCond %{HTTP_USER_AGENT} crawl [OR]
    RewriteCond %{HTTP_USER_AGENT} [Jj]ava [OR]
    RewriteCond %{HTTP_USER_AGENT} Wget [OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(Ahrefs|Baidu|BlogScope|Butterfly|DCPbot|discoverybot|domain|Ezooms|ImageSearcherFree).*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(ips-agent|linkdex|MJ12|Netcraft|NextGenSearchBot|SISTRIX|Sogou|soso|TweetmemeBot|Unwind|Yandex).*$ [NC]
    RewriteRule ^/?.*$ "http\:\/\/127\.0\.0\.1" [R,L]][/CODE]


    I get:

    Code:
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'


    thanks
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #5 mehnihma, Apr 2, 2013
    Last edited: Apr 2, 2013
  6. storminternet

    storminternet Well-Known Member

    Joined:
    Nov 2, 2011
    Messages:
    460
    Likes Received:
    0
    Trophy Points:
    66
    cPanel Access Level:
    Root Administrator
    Block the IP 157.55.33.40 using csf firewall and see if you are still receiving bot alerts.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    16,309
    Likes Received:
    393
    Trophy Points:
    583
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    MSN Bot:
    /http://whatis.riskyinternet.com/what-is/ip/157.55.33.40/

    This might be useful:
    /http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice