Is mod_security blocking search engines???

What do your logs show? According to your logs, are SEs getting 406s, and if so, what is the reason cited for the 406?

:D Bailey

 

The key here is "against "REQUEST_HEADERS:User-Agent" required". Spiders do have the User-Agent configured so they are not blocked by mod_security - unless a rule is specifically blocking this user-agent, that is

 

*nods* The error cited means nothing without an IP address.

What IP generated that error? When you run a whois on the IP, who does it belong to? Is it a search engine, or a regular user?

If it's a search engine, then you have just answered your own question.

:D Bailey

 

I've tested the default mod_security rules installed by WHM and I am able to browse to sites using lynx. This error could be occurring for a couple different reasons, including the site itself, the version and configuration of lynx that your using, etc. If you would like, please submit a ticket with this information along with the site in question and I'd be happy to dig a little deeper to find the exact cause for the error.

 

Is there any way to disable this rule? This is causing major problems for us because we host files that are being accessed by a (stand-alone, non-browser) program that apparently gets blocked by this rule... This seems to be the block of mod_security config that makes up the rule:

Code:

SecRule REQUEST_HEADERS:User-Agent "(?:\b(?:(?:indy librar|snoop)y|microsoft url control|lynx)\b|d(?:ownload demon|isco)|w(?:3mirror|get)|l(?:ibwww|wp)|p(?:avuk|erl)|cu(?:sto|rl)|big brother|autohttp|netants|eCatch)" \
        "chain,log,auditlog,msg:'Request Indicates an automated program explored the site',id:'990011',severity:'5'"
SecRule REQUEST_HEADERS:User-Agent "!^apache.*perl"

I have tried commenting out the lines, but no go, if I do, all the HTTP requests get blocked with 406...

For the time being I cleared the mod_security config file and now it's all working, but I'd rather keet the rest of the checks in place...

I cannot modify at all the program that is requesting these files to include the correct headers in its HTTP requests, nor can I require the maker to change their program to suit... so the only option is to remove this check.

Any suggestions?

EDIT: I should mention that in the access logs, the requests show up as:

Code:

201.211.96.234 - - [24/Jan/2008:07:43:47 -0800] "GET /server/etpub/etpub_client-20070801.pk3 HTTP/1.1" 406 496 "ET://72.5.249.131:27960" "ID_DOWNLOAD/2.0 libcurl/7.12.2"

so maybe there is a way to re-write the rule so that it allows requests with "ET://xxxx" as the referrer that would otherwise be blocked under this rule...