Outgoing Traffic

[mehnihma]

Hi

I have big Bandwidth usage in outgoing traffic, i think it is http, it is around 10GB a day. I do now know what is creating that traffic. I do not have a to visitors to have that kind of traffic.
How can I see what is the problem?

I Awstats I have found :
Unknown robot (identified by 'bot*') 398,018+456 91.85 GB 18 Mar 2013 - 03:36

Maybe that is the problem. How to stop unknown bots?

Thanks

 

[storminternet]

I hope you will find your answer here at bandwidth reporting

 

[mehnihma]

So do I

For now I have added in public_html root in .htacces:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} spider [OR]
RewriteCond %{HTTP_USER_AGENT} bot [OR]
RewriteCond %{HTTP_USER_AGENT} crawl [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Jj]ava [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget
RewriteRule .? - [F,L]

But is this enough?
If I have something in sub-folder do I have to create it also there? Is there any better solution?

Thanks

 

[arunsv84]

This rule seems to be fine. Or would be netter to place the rules in your apache config file so it will be blocked for all domains. Make sure to distill apache config or cPanel will overwrite the changes made to config file.

You will get the list of bad bots just by performing a Google search. Add that rules as well.

Cheers!!!

 

[mehnihma]

Hi what do you mean by distill apache config

Also these rules did not help

Unknown robot (identified by 'bot*')	74,451	19.26 GB	02 Apr 2013 - 14:12

I cannot block it
How can I find IP of that bot and block it because it is eating my bandwidth

When I add to .htacces
[CODERewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} spider [OR]
RewriteCond %{HTTP_USER_AGENT} bot [OR]
RewriteCond %{HTTP_USER_AGENT} ^bot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^bot [OR]
RewriteCond %{HTTP_USER_AGENT} bot* [OR]
RewriteCond %{HTTP_USER_AGENT} crawl [OR]
RewriteCond %{HTTP_USER_AGENT} [Jj]ava [OR]
RewriteCond %{HTTP_USER_AGENT} Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(Ahrefs|Baidu|BlogScope|Butterfly|DCPbot|discoverybot|domain|Ezooms|ImageSearcherFree).*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(ips-agent|linkdex|MJ12|Netcraft|NextGenSearchBot|SISTRIX|Sogou|soso|TweetmemeBot|Unwind|Yandex).*$ [NC]
RewriteRule ^/?.*$ "http\:\/\/127\.0\.0\.1" [R,L]][/CODE]


I get:

[Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'

thanks

 

[storminternet]

Block the IP 157.55.33.40 using csf firewall and see if you are still receiving bot alerts.

 

[Infopro]

MSN Bot:
/http://whatis.riskyinternet.com/what-is/ip/157.55.33.40/

This might be useful:
/http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx