mod_rewrite is cancelling out mod_security on cpanel servers.. why?!?

[qwerty]

Ok this has been an issue that has been bugging me for a LONG time now. It's discussed here atomicorp.com • View topic - mod_security + mod_rewrite with strange behavior

Basically, if you're using modsec with a cpanel box and you have an account which has wordpress (with the default .htaccess incl. mod_rewrite rules in it) - mod_sec will not filter traffic for that account.

Why ? Because it appears that due to some weird way that cpanel loads apache modules .. mod_rewrite appears to take precedence and so cancels out mod_sec in some circumstances eg. default wordpress .htaccess mod_rewrite rules..

I've edited my httpd.conf and put the line Include "/usr/local/apache/conf/modsec2.conf" at the very top, hoping that this would fix it (by means of making modsec load FIRST before anything else) but it doesn't. So for some strange reason, if you have any accounts using mod_rewrite such as Wordpress, which is a FREQUENT hacking access point, you're in trouble even if you have modsec + good modsec rules installed.

Any help would be appreciated. I am certain that this is a cpanel specific issue (to do with apache module ordering/loading/config) because vanilla centos + apache + modsec does not have this issue at all..

 

[k-planethost]

You can also use your browser to test the rules by going to a URL similar to this:

http://your_host/foo.php?foo=http://www.example.com
cpanel now uses mod sec 2.6.2 try an easy apache update you say on your link that you use mod_security 2.5.13
the most of got rules doesnt work on mod_security 2.5.13
are you using system priority?
are you using (cmc) and have you disable any specific rules for that domain?

 

[qwerty]

I'm on modsec 2.6 already, the issue is the same.

I'm not using system priority. Why ? What does it do ?

Yes I am using CMC but don't see the relevance. We have hundreds of customers who have Wordpress installations.

 

[k-planethost]

System Priority | R-fx Networks
this is system priority
i am not sure if this project can help you to give priority to apache modules you can have a look

 

[qwerty]

oh that, yeah i seriously doubt it.

Since mod_rewrite is compiled statically into apache I am assuming only Cpanel could change the module loading by making modifications to EasyApache.

This is a serious issue with serious security implications that I believe very few people are aware of. For all intents and purposes, any account with Wordpress installed is automatically vulnerable and not protected by modsec at all due to the default rewrites in wordpress..

 

[qwerty]

And to complicate matters MUCH further ...

I just tested this theory on 5 servers. All very similar cpanel/ceonts5 setups.

4 out of 5 exhibited behaviour as described above ie. wordpress sites arent protected due to the mod_rewrite rules.

1 out of 5 servers, modsec is working as expected ie. it's blocking access even with wordpress rewrites .. BUT, and this is the confusing WTF part - modsec is only blocking if you're using Internet Explorer. If you use Chrome for example, it's still getting through without problems..

What. The. ****************.

 

[qwerty]

One thing I wish to clarify .. when I am able to load a page suc as hhttp://wordpress-site-hosted-by-me.com/?=http://foo.bar modsec DOES still make an entry in the audit_log saying that it has 'intercepted it' and given a 501 error BUT it hasn't actually blocked it as the page loads normally.

I think there's a serious bug in either easyapache or modsec. I believe it's EA ie. cpanel's apache compiles, as this issue does not occur on vanilla servers...

 

[qwerty]

Weird as it may sound think I've found a 'workaround' for whatever is causing this ...

My modsec is configured to show error 501 when triggered.

Now if I put a 501.shtml in the user account (with wordpress / rewrites) I get the error page instead of the url loading normally.

Weird huh ? Can't believe I actually stumbled accross this when it could've been a million other things.

So to me it seems as if the error page configuration is messing with modsec and when there's no 501.shtml available the urls that SHOULD be blocked don't get blocked because the 501.shtml is missing.

 

[k-planethost]

yes the webpage sometimes loads but on mod sec there is an entry 501.shtml as you say above.
with the link that i give you results
Not Found

The requested URL /foo.php was not found on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request
if you try 4-5 times your ip should be block.
depends the adjustment on csf.
thats a test only
try to fake attack on one of your wordpress accounts such as sql reject and see if the rules are working properly. i dont think that has to do anything with cpanel this issue thats mod sec issue

 

[cPanelTristan]

Hello,

If you believe there might be an issue in our implementation of mod_rewrite in conjunction with mod_security, then please submit a bug report at http://go.cpanel.net/bugs to inquire about the issue with a link to this forum thread. If you would be able to post the ticket number here after submitting one so we can track the progress, that would be wonderful.

Thanks!

 

[qwerty]

I believe it's something to do with mod_rewrite and error documents which is causing modsec to not block the loading of pages when a modsec rule is triggered.

As mentioned above, if I put a 501.shtml in the customer's account, 501 is displayed when modsec is triggered ie. it works correctly. Without 501.shtml, modsec logs the error in audit_log BUT you're still able to load the url that triggered the modsec rule.

I've submitted a bug report ID 2031141

 

[k-planethost]

check your configuration on mod sec have you made any changes
me i dont use asl lite but free delayed rools of got root and some extra
if i fake attack with this link joomla sites return this on mozilla,exploler etc
Forbidden

You don't have permission to access /foo.php on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

after a while server says goodbuy to attack...depends from csf with 403 error

if i try to fake attack wordpress sites pages appears to load saying object not found
BUT the entry has as follows on audit log and server again says goodbuy to attack
[Mon Nov 28 10:40:23 2011] [error] [client my ip] ModSecurity: Access denied with code 403 (phase 2). Match of "beginsWith http:/%{SERVER_NAME}/" against "MATCHED_VAR" required. [file "path to my rules/10_asl_rules.conf"] [line "481"] [id "340162"] [rev "249"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Remote File Injection attempt in ARGS (AE)"] [data "http:/"] [severity "CRITICAL"] [hostname "domain.gr"] [uri "/foo.php"] [unique_id "TtNI97AJJfsAABycHMsAAAAH"]
all the servers are centos cpanel mod sec works exept if i have disable the rule id for specific domain

 

[qwerty]

well this issue is still present in latest apache (installed via easyapache) + latest modsec 2.6.6 (compiled from source)

I only noticed it today again because I've started using Atomicorp rules and they specify a different error (403) for triggered events. So since I didn't have a global 403.shtml in my errordocuments.conf, pages would still load normally even when they should be getting blocked. And yes, just as before, every 'hit' is logged in modsec_audit.log BUT if you don't have the correct error doc (403.shtml in this case) the page still loads normally.

There's a serious bug somewhere in apache's default config (cpanel only, doesn't seem to affect vanilla servers) to do with these error docs.

 

[mtindor]

.
.
This definitely is confusing, but do we really know that it isn't effectively blocking? It's one thing to throw a test at it such as "http:/www.yourdomain.com/?=http://foo.bar" -- because those specific tests aren't going to modify anything on the server / exploit anything in actuality.

A real test would be to actually throw an exploit [that should be blocked by modsecurity] at a site that is truly vulnerable [if mod_security were disabled] and then see if the exploit happens.

I say that because:

• As others have reported, if I have some rewrite rules in an htaccess file [such as the standard Wordpress rewrite rules when using permalinks], it does appear to display the site page rather than an error if I don't have a 403.shtml in the document root.
• mod_security does log it as being blocked
  
  

  [Tue Jun 26 10:04:24 2012] [error] [client 1.2.3.4] ModSecurity: Access denied with code 403 (phase 2). Match of "rx ://%{SERVER_NAME}/" against "MATCHED_VARS:" required. [file "/usr/local/apache/conf/modsec_rules/10_asl_rules.conf"] [line "493"] [id "340162"] [rev "274"] [msg "Atomicorp.com WAF Rules: Remote File Injection attempt in ARGS (AE)"] [data "http://foo.bar"] [severity "CRITICAL"] [hostname "www.mydomain.com"] [uri "/"] [unique_id "[email protected]"]

• Apache logs a 403 as well, along with producing the regular page output (item B)
  
  

  1.2.3.4 - - [26/Jun/2012:12:37:45 -0400] "GET /?=http://foo.bar HTTP/1.1" 403 51877 "-" "Mozilla/5.0 (Windows NT 6.0; rv:13.0) Gecko/20100101 Firefox/13.0.1"
  1.2.3.4 - - [26/Jun/2012:12:37:46 -0400] "GET /wp-includes/css/admin-bar.css?ver=3.4 HTTP/1.1" 304 - "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0; rv:13.0) Gecko/20100101 Firefox/13.0.1"
  1.2.3.4 - - [26/Jun/2012:12:37:46 -0400] "GET /wp-content/plugins/contact-form-7/includes/css/styles.css?ver=3.2 HTTP/1.1" 304 - "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0; rv:13.0) Gecko/20100101 Firefox/13.0.1"
  1.2.3.4 - - [26/Jun/2012:12:37:46 -0400] "GET /wp-includes/js/jquery/jquery.js?ver=1.7.2 HTTP/1.1" 304 - "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0; rv:13.0) Gecko/20100101 Firefox/13.0.1"
  1.2.3.4 - - [26/Jun/2012:12:37:46 -0400] "GET /wp-includes/js/admin-bar.js?ver=3.4 HTTP/1.1" 304 - "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0; rv:13.0) Gecko/20100101 Firefox/13.0.1"
  1.2.3.4 - - [26/Jun/2012:12:37:46 -0400] "GET /wp-content/plugins/contact-form-7/includes/js/jquery.form.js?ver=3.09 HTTP/1.1" 304 - "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0; rv:13.0) Gecko/20100101 Firefox/13.0.1"
  1.2.3.4 - - [26/Jun/2012:12:37:46 -0400] "GET /wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=3.2 HTTP/1.1" 304 - "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0; rv:13.0) Gecko/20100101 Firefox/13.0.1"
  1.2.3.4 - - [26/Jun/2012:12:37:47 -0400] "GET /wp-includes/images/admin-bar-sprite.png?d=20111130 HTTP/1.1" 304 - "http://www.mydomain.com/wp-includes/css/admin-bar.css?ver=3.4" "Mozilla/5.0 (Windows NT 6.0; rv:13.0) Gecko/20100101 Firefox/13.0.1"

And when mod_security is disabled, Apache logs the usual 200 | 304 status:

1.2.3.4 - - [26/Jun/2012:12:49:18 -0400] "GET /?=http://foo.bar HTTP/1.1" 200 44350 "-" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
1.2.3.4 - - [26/Jun/2012:12:49:18 -0400] "GET /wp-content/themes/journalist/style.css HTTP/1.1" 200 8366 "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
1.2.3.4 - - [26/Jun/2012:12:49:18 -0400] "GET /wp-content/plugins/contact-form-7/includes/css/styles.css?ver=3.2 HTTP/1.1" 200 887 "http://www.mydomain.com?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
1.2.3.4 - - [26/Jun/2012:12:49:18 -0400] "GET /wp-content/plugins/wp-recaptcha/recaptcha.css HTTP/1.1" 200 1739 "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
1.2.3.4 - - [26/Jun/2012:12:49:18 -0400] "GET /wp-content/plugins/contact-form-7/includes/js/jquery.form.js?ver=3.09 HTTP/1.1" 200 14238 "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
1.2.3.4 - - [26/Jun/2012:12:49:18 -0400] "GET /wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=3.2 HTTP/1.1" 200 6630 "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
1.2.3.4 - - [26/Jun/2012:12:49:18 -0400] "GET /wp-includes/js/jquery/jquery.js?ver=1.7.2 HTTP/1.1" 200 94861 "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
1.2.3.4 - - [26/Jun/2012:12:49:19 -0400] "GET /wp-content/themes/journalist/images/top.gif HTTP/1.1" 200 169 "http://www.mydomain.com/?=http://foo.bar" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
1.2.3.4 - - [26/Jun/2012:12:49:20 -0400] "GET /wp-content/themes/journalist/favicon.ico HTTP/1.1" 200 1150 "-" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"

When mod_security is enabled and a 403.shtml (or whatever errorpage corresponds to the status code your mod_security is set to deliver upon blocking) is present, the only thing that shows up in the Apache log is:

1.2.3.4 - - [26/Jun/2012:12:57:31 -0400] "GET /?=http://foo.bar HTTP/1.1" 403 6 "-" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"

Notice, the above example is what Apache shows in the log if a ###.shtml errorpage exists. If one doesn't exist, it shows up like "item B" farther up above.

I think a real test of whether it's blocking the malicious / questionable content is, for example, to actually throw a real exploit at a known vulnerable version of Wordpress [where the Wordpress is using permalinks and has the .htaccess file in place].

Either way, this strangeness shouldn't occur. At the very least something is confused. And the worst case is that mod_security is actually bypassed.

Mike

 

[qwerty]

there's a forum thread on atomicorp.com about the same issue posted by one of their customers who uses cpanel and from memory I think they suspect it has something to do with the way cpanel loads apache module or the order it is done in. I'll try and do some further testing and see how I go.

As for finding out if modsec is really blocking or just logging, I think that should be easy to test. i'll just create a php script that does something like <?php echo $crap; ?> and then visit that page and pass ?crap=http://test.com as an argument to it. If it prints that on the page, modsec isn't blocking

 

[qwerty]

Ok I am having some trouble testing this but what is definately happening is that, if there is any kind of rewriting going on eg. default Wordpress .htacces contains rewrites, the rewrite will take precedence over modsec.

If I modify the server's errordocuments.conf and put this in there, the issue stops...

Alias /error /usr/local/apache

ErrorDocument 403 /error/403.shtml
ErrorDocument 501 /error/501.shtml

This is too much for me to get to the bottom of it but suffice to say, default cpanel apache configs COULD very well be vulnerable whenever mod_rewrites are in place (and many scripts have them by default) and the only reliable way I've found to stop this weirdness from happening is by modifying the errordocument.conf file as per above.

Subsequent easyapache uses won't overwrite errordocument.conf so it's a 'safe' long term solution but I'd still just LOVE to know wtf is going on without this workaround in place as it obviously doesn't happen on non-cpanel apache implementations.

 

[qwerty]

removed - issue resolved

 

[cPanelTristan]

This needs to be reported via our bug interface for the developer security team to look into. Please submit a ticket to http://go.cpanel.net/bugs for this behavior or use the bugs link at the top of the forum.

 

[qwerty]

never mind

 

[qwerty]

Ugh, yes I am an idiot. This domain has a whitelisted rule which somehow cancelled out most other rules.. *facepalm*

SOOOOOOOOOOO glad this is the case and apologies about the excitement above :D

With that said, the issue does still remain without the errordocument.conf mods described above BUT I can live with that as those are one-off changes that need to be made as EA doesn't change the errordocument.conf file.