SOLVED Blocking bad bots

nunoleite · Jun 7, 2018

Hi!

I have seen lots of bots accessing my websites on my VPS. For now i just block IPs temporarily using CSF, but i would like to have a better and global solution.

So, i'm thinking in 2 options...
first:
Apache Configuration -> Include Editor -> “Pre Main Include”

Code:

<Directory "/home">
   SetEnvIfNoCase User-Agent "MJ12bot" bad_bots
   SetEnvIfNoCase User-Agent "AhrefsBot" bad_bots
   SetEnvIfNoCase User-Agent "SemrushBot" bad_bots
   SetEnvIfNoCase User-Agent "Baiduspider" bad_bots
   ...
  <RequireAll>
     Require all granted
     Require not env bad_bots
  </RequireAll>
</Directory>

or

Code:

<Directory "/home">
  BrowserMatchNoCase "Baiduspider" bots
  BrowserMatchNoCase "HTTrack" bots
  BrowserMatchNoCase "Yandex" bots
  ...
  Order Allow,Deny
  Allow from ALL
  Deny from env=bots
</Directory>

second:
using ModSecurity rules

Code:

SecRule REQUEST_HEADERS:User-Agent "CareerBot" "deny,log,noauditlog,severity:2,msg:'Spiderbot blocked',status:403"

I don't know if this codes are 100% correct, as i found them on the internet and have not tested.

Can i have some advice about these two options, using apache or modsecurity and if these codes would work?

Thanks
Nuno Leite

fuzzylogic · Jun 8, 2018

Your Modsec rule would not work, it has no id (which is mandatory).
Below is a copy of OWASP CRS rule 913102 (a Paranoia Level 2 rule), edited so as to block all the bots you listed in your examples.
I am not recommending that all these bots should be blocked, just offering working syntax for the bots you decide you want to block.
The id has had the number 1 added at the end so as to never cause a duplicate id error.

Code:

SecRule REQUEST_HEADERS:User-Agent "@rx ^(?:MJ12bot|AhrefsBot|SemrushBot|Baiduspider|HTTrack|Yandex|CareerBot)$" \
 "msg:'Found User-Agent associated with web crawler/bot',\
  severity:'CRITICAL',\
  id:9131021,\
  rev:'1',\
  phase:request,\
  block,\
  t:none,\
  ver:'OWASP_CRS/3.0.0',\
  maturity:'9',\
  accuracy:'9',\
  capture,\
  logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',\
  tag:'application-multi',\
  tag:'language-multi',\
  tag:'platform-multi',\
  tag:'attack-reputation-crawler',\
  tag:'OWASP_CRS/AUTOMATION/CRAWLER',\
  tag:'WASCTC/WASC-21',\
  tag:'OWASP_TOP_10/A7',\
  tag:'PCI/6.5.10',\
  tag:'paranoia-level/2',\
  setvar:'tx.msg=%{rule.msg}',\
  setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},\
  setvar:tx.%{rule.id}-OWASP_CRS/AUTOMATION/CRAWLER-%{matched_var_name}=%{matched_var},\
  setvar:ip.reput_block_flag=1,\
  expirevar:ip.reput_block_flag=%{tx.reput_block_duration},\
  setvar:'ip.reput_block_reason=%{rule.msg}'"

nunoleite · Jun 8, 2018

Hi!

The list of bots i have in the examples are not necessarily the ones o need to block has there are only 3 or 4 that i see more and have big impact on the server load.

So with this code i can use just the ModSecurity Tools and add this custom rule and changing the bot list on the first line would block all the bots i need, right?

With this approach i can add and remove easily the bots i need to block in the whole server, right?

Thanks

cPanelMichael · Jun 8, 2018

nunoleite said:
So with this code i can use just the ModSecurity Tools and add this custom rule and changing the bot list on the first line would block all the bots i need, right?

With this approach i can add and remove easily the bots i need to block in the whole server, right?

Hello,

You can use the rule as an example, but note some of the entries are designed for use with the OWASP rule set:

OWASP ModSecurity CRS - cPanel Knowledge Base - cPanel Documentation

Thank you.

fuzzylogic · Jun 8, 2018

cPanelMichael is correct the rule I posted relies on other OWASP CRS rules to do the blocking.
It was wrong of me to assume you would have this in your environment.

Here is another rule example based on the faulty rule example you posted.
It will work as a standalone rule or alongside any rule-set I know of.

Code:

SecRule REQUEST_HEADERS:User-Agent "@rx ^(?:MJ12bot|AhrefsBot)$" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:none,block,status:403"

I tested this rule and it returned a 403 status for request with either of the following headers...
User-Agent: MJ12bot
User-Agent: AhrefsBot

nunoleite said:
So with this code i can use just the ModSecurity Tools and add this custom rule and changing the bot list on the first line would block all the bots i need, right?

That is correct.
Add bots so that the regex has this form
^(?:bot1|bot2|bot3)$

nunoleite said:
With this approach i can add and remove easily the bots i need to block in the whole server, right?

ModSecurity rules added through...
Home » Security Center » ModSecurity™ Tools » Rules List » Add Rule
are applied to all http requests to the cPanel server.

nunoleite · Jun 11, 2018

Thanks fuzzylogic.
That's what i was looking for... a simple rule that could block these bad bots.

I don't have OWASP rules installed because some time ago i tried that and it created lots of problems with some CMS i have in the server, and i didn't investigate better what rules to enable or disable to be compatible.

I will try this new SecRule, thanks.

What about the other option using apache configuration? Is it valid? Or using modsecurity is better?

Thanks

cPanelMichael · Jun 11, 2018

nunoleite said:
What about the other option using apache configuration? Is it valid? Or using modsecurity is better?

Mod_Security rules are a better option in my opinion. It will make it easier for you to exclude rules for specific accounts if necessary.

Thank you.

nunoleite · Jun 12, 2018

Hi!
I have published this rule:

Code:

SecRule REQUEST_HEADERS:User-Agent "@rx ^(?:AhrefsBot|MJ12bot|Yandex)$" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:none,block,status:403"

But i still see this visitors with this user agent:
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; MJ12Bot | Home | from Majestic)

This isn't being blocked.

fuzzylogic · Jun 13, 2018

If you want to match a fragment of the User-Agent you require a looser regular expression.

Code:

SecRule REQUEST_HEADERS:User-Agent "@rx (?:MJ12bot|AhrefsBot)" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:none,block,status:403"

Also it is possible you want to cover variations of uppercase and lowercase.
This can be achieved by having modsecurity transform all User-Agent values to lowercase then enter the bot names in the regex as all lowercase.

Code:

SecRule REQUEST_HEADERS:User-Agent "@rx (?:ahrefsbot|mj12bot|yandex)" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:lowercase,block,status:403"

nunoleite · Jun 13, 2018

Hi!
With this rule:

Code:

SecRule REQUEST_HEADERS:User-Agent "@rx (?:AhrefsBot|MJ12bot|Yandex)" "msg:'Spiderbot blocked',phase:1,log,id:777777,t:none,block,status:403"

I think MJ12bot are being blocked, but i still see: user-agent: Mozilla/5.0 (compatible; AhrefsBot/5.2; +ahrefs.com/robot/)

hum..... strange....

nunoleite · Jun 13, 2018

Hi!
I have made some changes...
I have added this 3 rules:

Code:

SecRule REQUEST_HEADERS:User-Agent "@rx (?:AhrefsBot)" "msg:'AhrefsBot Spiderbot blocked',phase:1,log,id:7777771,t:none,block,status:403"
SecRule REQUEST_HEADERS:User-Agent "@rx (?:MJ12bot)" "msg:'MJ12bot Spiderbot blocked',phase:1,log,id:7777772,t:none,block,status:403"
SecRule REQUEST_HEADERS:User-Agent "@rx (?:Yandex)" "msg:'Yandex Spiderbot blocked',phase:1,log,id:7777773,t:none,block,status:403"

With these 3 rules i know in the "hits list" what is going on with each bot and what is hitting the rule, because they are being logged separately.

If i understand right the bots still have access to sites, but they receive 0bytes and an 403 http error.
After some hits the CSF firewall blocks permanently the IP. Is this right? And is this the right behavior?

cPanelMichael · Jun 13, 2018

nunoleite said:
After some hits the CSF firewall blocks permanently the IP. Is this right? And is this the right behavior?

Hello @nunoleite,

Yes, this is the case as long as you have enabled the LF_MODSEC feature in CSF.

Thank you.

nunoleite · Jun 13, 2018

Hi!
Thanks.
Now i think this is working fine, as i can see lots of Hits in the rules and some being blocked.

But analyzing visitors it seems that AhrefsBot is still being served.

Is this possible?

WebJIVE · Jan 8, 2019

Good article on this subject and one we're going to look into.

linuxadmin.io/blocking-bad-useragents-modsecurity-fail2ban/

and another one that we're testing

geekytuts.net/cpanel/block-bad-bots-cpanel-globally-apache/

Thread starter	Similar threads	Forum	Replies	Date
	ModSecurity OWASP blocking GTMetrix	Security	1	Dec 12, 2022
A	SOLVED csf not blocking IPs	Security	2	Oct 12, 2022
V	Blocking Bad Bots in Apache configuration	Security	1	Feb 3, 2021
	Best practices for blocking bad bots?	Security	1	Aug 19, 2020
	Help with Globally blocking Bad Bots	Security	10	Mar 29, 2020

Search

Search

SOLVED Blocking bad bots

nunoleite

Well-Known Member

fuzzylogic

Well-Known Member

nunoleite

Well-Known Member

cPanelMichael

Administrator

fuzzylogic

Well-Known Member

nunoleite

Well-Known Member

cPanelMichael

Administrator

nunoleite

Well-Known Member

fuzzylogic

Well-Known Member

nunoleite

Well-Known Member

nunoleite

Well-Known Member

cPanelMichael

Administrator

nunoleite

Well-Known Member

WebJIVE

Well-Known Member