Mod Security Rules Incorrectly Blocking Googlebot

[Another Blogger]

Hello everyone

I'm using CSF Firewall in my blog and the problem is the Mod_Security rules incorrectly blocks Googlebot regularly. It is happening almost every week.

In past I have removed a Mod Security rule which was blocking Googlebot and now a new Mod Security rule 950005 is blocking Googlebot.

Whenever it blocks Googlebot, I manually remove Googlebot IP address from deny list using WHM.

Is there any way to permanently stop this? How can I set Mod Security to not block Googlebot?

My hosting staff suggested me to add following rule at the end of Mod Security configuration:

#Allow googlebots
SecRule REMOTE_HOST googlebot.com$ allow,pass

But I'm not sure whether I should add it or not. Is there any security risk in adding this rule or is it ok? Is there any other recommended way to allow Googlebot?

Any kind of suggestions will be highly appreciated.

 

[Indianets]

Add this to your config. The right way to identify Google bot is it's User-Agent.

SecRule REQUEST_HEADERS:User-Agent "Googlebot" phase:1,nolog,allow,ctl:ruleEngine=off

 

[Another Blogger]

^^ Thanks for your reply but if someone edits browser's user-agent using add-on or configurations and make it look-like Googlebot's useragent then? Will it cause a security risk?

 

[quietFinn]

I would say that the right way to correct that issue is to remove (or maybe edit) the rule blocking Googlebot.

Where are the rules from? Atomicorp rules don't seem to have rule #950005.

 

[Another Blogger]

^^ I'm using CSF firewall. Following is the email alert content:

Time: Fri Jan 4 13:47:06 2013
IP: 66.249.73.150 (US/United States/crawl-66-249-73-150.googlebot.com)
Failures: 5 (mod_security)
Interval: 300 seconds
Blocked: Permanent Block

Log entries:

[Fri Jan 04 13:44:02 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

[Fri Jan 04 13:44:03 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

[Fri Jan 04 13:45:03 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

[Fri Jan 04 13:46:04 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

[Fri Jan 04 13:47:04 2013] [error] [client 66.249.73.150] ModSecurity: Access denied with code 501 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:s. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "119"] [id "950005"] [msg "Remote File Access Attempt"] [data "boot.ini"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [hostname "xxx.xxx"] [uri "/"] [unique_id "xxx"]

 

[quietFinn]

I would suggest that you disable LF_MODSEC, or at least change it to temporary block.

 

[Another Blogger]

^^ Will it not disable Mod security?

 

[quietFinn]

^^ Will it not disable Mod security?

No, it will only disable CSF/LFD checking of mod_security logs.
You can leave it on (LF_MODSEC = 1), but make the block temporary (LF_MODSEC_PERM = 3600, 1 hour block).

 

[Another Blogger]

^^ Thanks. But will it solve the problem? I mean it'll block Googlebot IP temporary and after 1 hour when Googlebot will try to crawl again, firewall will again block it temporary. Is it right?

PS: Can you please clarify what's the difference between enabling and disabling CSF checking of mod_security logs? I mean if we disable the checking, what will happen in that situation? How will the malicious IP address blocked by the server if we disable CSF checking? I'm a little bit confused between Mod security and CSF.

 

[quietFinn]

^^ Thanks. But will it solve the problem? I mean it'll block Googlebot IP temporary and after 1 hour when Googlebot will try to crawl again, firewall will again block it temporary. Is it right?

PS: Can you please clarify what's the difference between enabling and disabling CSF checking of mod_security logs? I mean if we disable the checking, what will happen in that situation? How will the malicious IP address blocked by the server if we disable CSF checking? I'm a little bit confused between Mod security and CSF.

I don't know what really is the best solution for this issue.
Mod_security blocking Googlebot is a "false positive", and usually those are fixed by creating an exception. As has been suggested you could "whitelist" Googlebot by it's User-Agent, but that would open a hole for anyone using a fake Googlebot User-Agent.

If you want to know what mod_security is and how it works you should start reading for ex. here:
https://github.com/SpiderLabs/ModSe...stions-(FAQ)#wiki-What_exactly_is_ModSecurity

CSF/LFD reads, if configured so, mod_security log file, and when it sees that one IP has been blocked LF_MODSEC times in 5 minutes it blocks the IP, either permanently or temporarily.
So it works so that mod_security is blocking something, in this case Googlebot, CSF/LFD sees those blocks in mod_security log, and blocks the used IP in the firewall.
In this case, mod_security is not blocking Googlebot because it is Googlebut, but because Googlebot is doing something that seems suspicious. If you can't prevent mod_security from doing that, you should at least make sure that Googlebot is not blocked permanently.

In our servers we don't have LF_MODESC enabled, but if it was, it would not do permanent blocking.

 

[Another Blogger]

^^ Thanks again for your help. So if I disable LF_MODSEC, Mod_Security will still check IP addresses based upon its rules and if it finds something malicious, it'll block it. Right?

 

[quietFinn]

^^ Thanks again for your help. So if I disable LF_MODSEC, Mod_Security will still check IP addresses based upon its rules and if it finds something malicious, it'll block it. Right?

Yes that's right.

 

[Another Blogger]

^^ Thanks for all your help.

 

[Indianets]

Sorry for my delayed arrival.

If you are using CSF, you can add .googlebot.com to "/etc/csf/csf.rignore"

.googlebot.com

- Vijay

 

[crazyaboutlinux]

It's a nice idea