modsec rule 942100 not being blocked, status 200

[bloatedstoat]

Hi there.

We're running the third party CSF rules and cpanel OWASP ModSecurity Core Rule Set V3.0 on our server.

Sifting the logs I've come across rafts of entries from the same IP address triggering the 942100 SQL Injection Attack Detected via libinjection rule.

The severity is CRITICAL yet the status is 200.

No entries appear in the firewall for the IP.

If I click on More to the right of the rule in ModSecurity Tools the drop down shows the attack vector but the Action, Description and Justification are all empty.

Does the status 200 mean that the attack was successful and database data was retrieved?

And, how can I get critical status events to be blocked at the firewall level.

Thank you.

 

[fuzzylogic]

No the 200 reported here does not indicate the attack was successful.
Quite the opposite, if rule 942100 was hit then the request would have been blocked.

As you are using CFS I will use it in the troubleshooting approach.
In the WHM Modsecurity Tools Hits list find the hit you have posted about.
Copy the Time portion of the Time/Date.
Go to WHM >> Plugins >> CSF Watch System Logs >> /usr/local/apache/logs/modsec_audit.log
Set Refresh to Pause and Lines to 10000
Do a browser search for the Time you copied.
You should find a detailed log of the problem request.
One of the items on the first line is a request unique_id looks like
WU3Wi1eVY4Y7NjR1cBKonAAAAMQ
Copy it.
Further down the modsec_audit.log for this hit in the H section is a list of messages from all the rules this request hit.
First will be the message from rule 942100 (its action is to block by the way even though it does not say so here)
Next message will be from rule 949110 (Its action is also to block, but importantly its message is "Access denied with code 403").
Reading this log file regularly and reading the rules themselves will teach you a lot about how modsecurity and its rule sets work.

Now to CFS blocking.
Now on the same CSF page select the /usr/local/apache/logs/error_log
Do a browser search for the unique_id you copied in the last step.
If the modsec_audit.log had 3 Messages for this hit then the error_log will have 3 consecutive entries for that unique_id
The entry for rule 949110 will have the Message "ModSecurity: Access denied with code 403"
This is the file that CSF parses to get ips to block.
CSF looks for that Message in that file.

To get a permanent block in CSF the CSF settings should be...
MODSEC_LOG = /usr/local/apache/logs/error_log
LF_INTERVAL = 86400 ( No. of seconds over which to count. Default is 3600)
LF_MODSEC = 5 (Count of string "Modsecurity: Access denied" per ip for block action)
LF_MODSEC_PERM = 1 (0 = No blocking, 1 = Block Permanently, x = Block for x seconds except 1 or 0)

So on my server if an ip does 5 requests in a day that are blocked by Modsecurity rules it goes onto the CSF permanent block list.

 

[bloatedstoat]

@fuzzylogic , thanks for that mate.
Your detailed response has been a great help.
Thanks again!

 

[cPanelMichael]

Hello,

I'm happy to see the previous post helped. I'm marking this thread as solved.

Thank you.

 

[bloatedstoat]

Sorry to say that this is still not working for me.
In fact there are no entries at all in the apache error log for "ModSecurity: Access denied with code 403"

In WHM modsecurity tools is this line:

2017-06-28 16:06:43 obfuscated.com    OFFENDING_IP CRITICAL 404 942100: SQL Injection Attack Detected via libinjection

Copy the Time portion of the Time/Date. (16:06:43)

vi /usr/local/apache/logs/modsec_audit.log

Search for the time 16:06:43; the entire block from the time down with that time is the following, the status is not Access denied with 403 it's a 404:

88494 --1cf96b2c-A--
88495 [28/Jun/2017:16:06:43 +1000] WVNHc4qzQyvAi-iZvEcPMwAAAVY OFFENDING_IP 40380 OUR_IP 80
88496 --1cf96b2c-B--
88497 POST /wp-cods.php HTTP/1.1
88498 Accept-Encoding: identity
88499 Content-Length: 298
88500 Accept-Language: en-US,en;q=0.8
88501 Connection: close
88502 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36
88503 Host: obfuscated.com
88504 Referer: obfuscated.com
88505 Content-Type: application/x-www-form-urlencoded
88506
88507 --1cf96b2c-C--
88508 sort=cHJpbnQgbWQ1KDg4ODg4OCk7&fuckyou4321=print%28md5%2833333%29%29%3B&c=print+md5%285555%29%3B&sam=cHJpbnQgbWQ1KDQ0NDQpOw%3D%3D&p=cd121eaf&coco=print%28md5%2811111%29%29%3B&array=cHJpbnQgbWQ1KDc3Nzc3KTs%3D&cmd=print+md5%286666%29%3B&yt=print%28md5%28999999%29%29%3B&sss=print%28md5%282222%29%29%3B
88509 --1cf96b2c-F--
88510 HTTP/1.1 404 Not Found
88511 Content-Length: 328
88512 Connection: close
88513 Content-Type: text/html; charset=iso-8859-1
88514
88515 --1cf96b2c-H--

88516 Message: Warning. detected SQLi using libinjection with fingerprint 'T(f(1' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: T(f(1 found within ARGS:fuckyou4321: print(md5(33333));"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag"language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]

88517 Message: Warning. detected SQLi using libinjection with fingerprint 'Tf(1)' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: Tf(1) found within ARGS:c: print md5(5555);"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]

88518 Message: Warning. detected SQLi using libinjection with fingerprint 'T(f(1' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: T(f(1 found within ARGS:coco: print(md5(11111));"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]

88519 Message: Warning. detected SQLi using libinjection with fingerprint 'Tf(1)' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: Tf(1) found within ARGS:cmd: print md5(6666);"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]

88520 Message: Warning. detected SQLi using libinjection with fingerprint 'T(f(1' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: T(f(1 found within ARGS:yt: print(md5(999999));"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]

88521 Message: Warning. detected SQLi using libinjection with fingerprint 'T(f(1' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: T(f(1 found within ARGS:sss: print(md5(2222));"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]

88522 Apache-Error: [file "mod_suphp.c"] [line 792] [level 3] File does not exist: %s
88523 Stopwatch: 1498630003112278 218982 (- - -)
88524 Stopwatch2: 1498630003112278 218982; combined=4757, p1=458, p2=4025, p3=64, p4=152, p5=58, sr=69, sw=0, l=0, gc=0
88525 Producer: ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/); OWASP_CRS/3.0.0.
88526 Server: Apache
88527 Engine-Mode: "ENABLED"
88528
88529 --1cf96b2c-Z--

Search /usr/local/apache/logs/error_log for the string identifier WVNHc4qzQyvAi-iZvEcPMwAAAVY taken from the modsec_audit.log

[Wed Jun 28 16:06:43.329048 2017] [:error] [pid 537361:tid 140012983076608] [client 205.186.148.140] ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'T(f(1' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: T(f(1 found within ARGS:fuckyou4321: print(md5(33333));"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "obfuscated.com"] [uri "/wp-cods.php"] [unique_id "WVNHc4qzQyvAi-iZvEcPMwAAAVY"]

9571 [Wed Jun 28 16:06:43.329148 2017] [:error] [pid 537361:tid 140012983076608] [client 205.186.148.140] ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'Tf(1)' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: Tf(1) found within ARGS:c: print md5(5555);"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "obfuscated.com"] [uri "/wp-cods.php"] [unique_id "WVNHc4qzQyvAi-iZvEcPMwAAAVY"]

9572 [Wed Jun 28 16:06:43.329261 2017] [:error] [pid 537361:tid 140012983076608] [client 205.186.148.140] ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'T(f(1' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: T(f(1 found within ARGS:coco: print(md5(11111));"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "obfuscated.com"] [uri "/wp-cods.php"] [unique_id "WVNHc4qzQyvAi-iZvEcPMwAAAVY"]

9573 [Wed Jun 28 16:06:43.329357 2017] [:error] [pid 537361:tid 140012983076608] [client 205.186.148.140] ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'Tf(1)' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: Tf(1) found within ARGS:cmd: print md5(6666);"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "obfuscated.com"] [uri "/wp-cods.php"] [unique_id "WVNHc4qzQyvAi-iZvEcPMwAAAVY"]

9574 [Wed Jun 28 16:06:43.329449 2017] [:error] [pid 537361:tid 140012983076608] [client 205.186.148.140] ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'T(f(1' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: T(f(1 found within ARGS:yt: print(md5(999999));"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "obfuscated.com"] [uri "/wp-cods.php"] [unique_id "WVNHc4qzQyvAi-iZvEcPMwAAAVY"]

9575 [Wed Jun 28 16:06:43.329527 2017] [:error] [pid 537361:tid 140012983076608] [client 205.186.148.140] ModSecurity: Warning. detected SQLi using libinjection with fingerprint 'T(f(1' [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "43"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: T(f(1 found within ARGS:sss: print(md5(2222));"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "obfuscated.com"] [uri "/wp-cods.php"] [unique_id "WVNHc4qzQyvAi-iZvEcPMwAAAVY"]

9576 [Wed Jun 28 16:06:43.330489 2017] [:error] [pid 537361:tid 140012983076608] [client 205.186.148.140:40380] File does not exist: /home/obfuscated/public_html/wp-cods.php, referer: obfuscated.com

As you can see there is not one mention at all of Access denied with 403.

CSF is clearly not picking up anything because the string it used in the regex is not present.

Whilst writing this up I also picked up that there is this error in the logs too:
88522 Apache-Error: [file "mod_suphp.c"] [line 792] [level 3] File does not exist: %s


So, ummm, where to from here?

Thank you.

 

[fuzzylogic]

The error...
88522 Apache-Error: [file "mod_suphp.c"] [line 792] [level 3] File does not exist: %s
is occurring right where I would expect the log line for rule 949110 to occur for this request.
So as you suggested this is the new troubleshooting vector.
I am not familiar with this error message.

Possibly...
request comes in and hits rule id 942100 once per dodgy ARG.
each hit of 942100 has a block action.
this causes redirect to 403 error page.
403 error page does not exist.
this causes the Apache-Error: [file "mod_suphp.c"] [line 792] [level 3] File does not exist: %s
this causes redirect to 404 error page.
mod_security stops logging before rule id 949110 is logged? (wild guess)

If this is what is happening then possibly adding a 403 error document may allow mod_security to continue logging.
A simple 403.shtml file in the web root of the account would do.

I have a 403.shtml file in the web root of all my domains so I don't get the...
Apache-Error: [file "mod_suphp.c"] [line 792] [level 3] File does not exist: %s for them.
I did however find instances of that error in my modsec_audit.log
They occurred when non-existent pages were requested from the raw ip address of my server.
For instance...
XX.XX.XX.XX/bogus.php
hit a non-blocking modsec rule 920350 due to host being an ip address
returned a generic cPanel 404 error page
generated the Apache-Error: [file "mod_suphp.c"] [line 792] [level 3] File does not exist: %s error

XX.XX.XX.XX/wp-config.php
hit a non-blocking modsec rule 920350 due to host being an ip address
hit a blocking rule 930130 due to the wp-config string
returned a generic cPanel 403 error page
It did not generate the "File does not exist" error.

I think this is a strong indicator that a adding a 403 error document might fix this problem.

 

[bloatedstoat]

I've opened a ticket for this.
There are not even any hits in the modsec database hits table.
Ticket ID 8664845.
I'll update once I have further information.

 

[bloatedstoat]

Partial progress here, one embarrassingly simple explanation for some hits going unflagged with 403 was that a rule had been white-listed in our configuration.

That said, 942100 and 942190 SQL injections are reported in the logs but not generating the required 403, so no "Inbound Anomaly Score Exceeded", as such we're not seeing blocks.

I've reported the hit to OWASP.

Is anyone else seeing anything similar with those two rules?

Thanks.