mehnihma

Well-Known Member
Dec 15, 2012
57
1
8
cPanel Access Level
Root Administrator
Hi

I have big Bandwidth usage in outgoing traffic, i think it is http, it is around 10GB a day. I do now know what is creating that traffic. I do not have a to visitors to have that kind of traffic.
How can I see what is the problem?

I Awstats I have found :
Unknown robot (identified by 'bot*') 398,018+456 91.85 GB 18 Mar 2013 - 03:36

Maybe that is the problem. How to stop unknown bots?

Thanks
 
Last edited:

mehnihma

Well-Known Member
Dec 15, 2012
57
1
8
cPanel Access Level
Root Administrator
So do I :)

For now I have added in public_html root in .htacces:

Code:
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} spider [OR]
RewriteCond %{HTTP_USER_AGENT} bot [OR]
RewriteCond %{HTTP_USER_AGENT} crawl [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Jj]ava [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget
RewriteRule .? - [F,L]
But is this enough?
If I have something in sub-folder do I have to create it also there? Is there any better solution?

Thanks
 

arunsv84

Well-Known Member
Oct 20, 2008
373
1
68
127.0.0.1
cPanel Access Level
Root Administrator
This rule seems to be fine. Or would be netter to place the rules in your apache config file so it will be blocked for all domains. Make sure to distill apache config or cPanel will overwrite the changes made to config file.

You will get the list of bad bots just by performing a Google search. Add that rules as well.

Cheers!!!
 

mehnihma

Well-Known Member
Dec 15, 2012
57
1
8
cPanel Access Level
Root Administrator
Hi what do you mean by distill apache config

Also these rules did not help

Code:
Unknown robot (identified by 'bot*')	74,451	19.26 GB	02 Apr 2013 - 14:12
I cannot block it
How can I find IP of that bot and block it because it is eating my bandwidth

When I add to .htacces
[CODERewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} spider [OR]
RewriteCond %{HTTP_USER_AGENT} bot [OR]
RewriteCond %{HTTP_USER_AGENT} ^bot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^bot [OR]
RewriteCond %{HTTP_USER_AGENT} bot* [OR]
RewriteCond %{HTTP_USER_AGENT} crawl [OR]
RewriteCond %{HTTP_USER_AGENT} [Jj]ava [OR]
RewriteCond %{HTTP_USER_AGENT} Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(Ahrefs|Baidu|BlogScope|Butterfly|DCPbot|discoverybot|domain|Ezooms|ImageSearcherFree).*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(ips-agent|linkdex|MJ12|Netcraft|NextGenSearchBot|SISTRIX|Sogou|soso|TweetmemeBot|Unwind|Yandex).*$ [NC]
RewriteRule ^/?.*$ "http\:\/\/127\.0\.0\.1" [R,L]][/CODE]


I get:

Code:
[Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:43 2013] [alert] [client 157.56.93.228] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.253] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.176] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.56.92.157] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'
[Wed Apr 03 03:01:42 2013] [alert] [client 157.55.33.40] /home/domain/public_html/.htaccess: RewriteCond: cannot compile regular expression '*bot'


thanks
 
Last edited: