The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Outgoing Traffic

Discussion in 'General Discussion' started by mehnihma, Mar 18, 2013.

  1. mehnihma

    mehnihma Well-Known Member

    Joined:
    Dec 15, 2012
    Messages:
    57
    Likes Received:
    1
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Hi

    I have big Bandwidth usage in outgoing traffic, i think it is http, it is around 10GB a day. I do now know what is creating that traffic. I do not have a to visitors to have that kind of traffic.
    How can I see what is the problem?

    I Awstats I have found :
    Unknown robot (identified by 'bot*') 398,018+456 91.85 GB 18 Mar 2013 - 03:36

    Maybe that is the problem. How to stop unknown bots?

    Thanks
     
    #1 mehnihma, Mar 18, 2013
    Last edited: Mar 18, 2013
  2. storminternet

    storminternet Well-Known Member

    Joined:
    Nov 2, 2011
    Messages:
    462
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
  3. mehnihma

    mehnihma Well-Known Member

    Joined:
    Dec 15, 2012
    Messages:
    57
    Likes Received:
    1
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    So do I :)

    For now I have added in public_html root in .htacces:

    Code:
    RewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
    RewriteCond %{HTTP_USER_AGENT} spider [OR]
    RewriteCond %{HTTP_USER_AGENT} bot [OR]
    RewriteCond %{HTTP_USER_AGENT} crawl [OR]
    RewriteCond %{HTTP_USER_AGENT} ^[Jj]ava [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Wget
    RewriteRule .? - [F,L]
    But is this enough?
    If I have something in sub-folder do I have to create it also there? Is there any better solution?

    Thanks
     
  4. arunsv84

    arunsv84 Well-Known Member

    Joined:
    Oct 20, 2008
    Messages:
    373
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    127.0.0.1
    cPanel Access Level:
    Root Administrator
    This rule seems to be fine. Or would be netter to place the rules in your apache config file so it will be blocked for all domains. Make sure to distill apache config or cPanel will overwrite the changes made to config file.

    You will get the list of bad bots just by performing a Google search. Add that rules as well.

    Cheers!!!
     
  5. mehnihma

    mehnihma Well-Known Member

    Joined:
    Dec 15, 2012
    Messages:
    57
    Likes Received:
    1
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Hi what do you mean by distill apache config

    Also these rules did not help

    Code:
    Unknown robot (identified by 'bot*')	74,451	19.26 GB	02 Apr 2013 - 14:12
    I cannot block it
    How can I find IP of that bot and block it because it is eating my bandwidth

    When I add to .htacces
    [CODERewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
    RewriteCond %{HTTP_USER_AGENT} spider [OR]
    RewriteCond %{HTTP_USER_AGENT} bot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^bot* [OR]
    RewriteCond %{HTTP_USER_AGENT} ^bot [OR]
    RewriteCond %{HTTP_USER_AGENT} bot* [OR]
    RewriteCond %{HTTP_USER_AGENT} crawl [OR]
    RewriteCond %{HTTP_USER_AGENT} [Jj]ava [OR]
    RewriteCond %{HTTP_USER_AGENT} Wget [OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(Ahrefs|Baidu|BlogScope|Butterfly|DCPbot|discoverybot|domain|Ezooms|ImageSearcherFree).*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*(ips-agent|linkdex|MJ12|Netcraft|NextGenSearchBot|SISTRIX|Sogou|soso|TweetmemeBot|Unwind|Yandex).*$ [NC]
    RewriteRule ^/?.*$ "http\:\/\/127\.0\.0\.1" [R,L]][/CODE]


    I get:

    Code:
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
    [Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'


    thanks
     
    #5 mehnihma, Apr 2, 2013
    Last edited: Apr 2, 2013
  6. storminternet

    storminternet Well-Known Member

    Joined:
    Nov 2, 2011
    Messages:
    462
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Block the IP 157.55.33.40 using csf firewall and see if you are still receiving bot alerts.
     
  7. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,455
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    MSN Bot:
    /http://whatis.riskyinternet.com/what-is/ip/157.55.33.40/

    This might be useful:
    /http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx
     
Loading...

Share This Page