deciphering mod_security logs

[jeffschips]

Hello. I hope everyone is safe and healthy.

Is this the right place to ask a question about mod_security logs? I keep getting the same log entries from mod_security via csf's lfd output but don't really understand them.

Thanks.

 

[ServerHealers]

It would be easier to advise if you could attach the sample log or output you get, and the exact question/concern you've with it.

 

[jeffschips]

Thanks. Following is a sample output from mod_sec as compiled by csf for reporting. I'm particularly interested in detecting the actual links or code that references some kind of malicious activity because it's that code that needs to be submitted to the amazons, googles and microsoft abuse forms. I know, I know I can block the entire cloud through updated bulk lists but there are downsides to that namely it may also block legitimate links to my website on the same server which is being attacked, or so I've read.

Time: Fri Dec 16 08:51:39 2022 -0500
IP: 20.106.xx.xxx (US/United States/-)
Failures: 3 (mod_security)
Interval: 3600 seconds
Blocked: Temporary Block for 86400 seconds [LF_TRIGGER]

Log entries:

[Fri Dec 16 08:51:36.978979 2022] [:error] [pid 1533] [client 20.106.xx.xxx:60771] [client 20.106.xx.xxx] ModSecurity: Access denied with code 403 (phase 1). Matched phrase "/.env" at REQUEST_URI. [file "/etc/apache2/conf.d/modsec_vendor_configs/comodo_apache/02_Global_Generic.conf"] [line "117"] [id "210492"] [rev "3"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "45.xx.xxx.xxx"] [uri "/.env"] [unique_id "Y5x36Nr2w2PAPpsj5BClEgAAAAQ"]


[Fri Dec 16 08:51:37.067157 2022] [:error] [pid 1534] [client 20.106.xx.xxx:60799] [client 20.106.xx.xxx] ModSecurity: Access denied with code 403 (phase 2). Pattern match "^[\\\\d.:]+$" at REQUEST_HEADERS:Host. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "736"] [id "920350"] [msg "Host header is a numeric IP address"] [data "45.xx.xxx.xxx"] [severity "WARNING"] [ver "OWASP_CRS/3.3.4"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [hostname "45.xx.xxx.xxx"] [uri "/"] [unique_id "Y5x36XlAfDts4S2Bv_HczAAAAAY"]


[Fri Dec 16 08:51:37.127667 2022] [:error] [pid 2996] [client 20.106.xx.xxx:60810] [client 20.106.xx.xxx] ModSecurity: Access denied with code 403 (phase 1). Matched phrase "/.env" at REQUEST_URI. [file "/etc/apache2/conf.d/modsec_vendor_configs/comodo_apache/02_Global_Generic.conf"] [line "117"] [id "210492"] [rev "3"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "45.xx.xxx.xxx"][B] [uri "/.env"][/B] [unique_id "Y5x36QqyfSmS8IZWgytVmQAAAAc"]


When I see [uri "/.env"] I believe that would indicate a malicious user scanning for enivronmental variables, which are known to contain credentials.

 

[cPRex]

If you check these timestamps directly in the Apache log, does it give you more detail about the URL or file that caused the trigger? I'm not familiar with any CSF processing of ModSecurity rules or logs.

 

[jeffschips]

Here is an example of a mod_security triggered log found in the apache's error_logs:

[Mon Dec 19 21:35:23.087209 2022] [:error] [pid 14858] [client 85.62.xxx.xx:59737] [client 85.62.xxx.xx] ModSecurity: Access denied with code 403 (phase 1). Match of "rx ^(??:\\\\*|[^\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]+)\\\\/(?:\\\\*|[^\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]+))(?:\\\\s*+;\\\\s*+(??:charset\\\\s*+=\\\\s*+(?:\\"?(?:iso-8859-15?|windows-1252|utf-8)\\\\b\\"?))|(??:c(?:h(?:a(?:r(?:s(?:e[^t\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]|[^e\\"(),/:;<=>?![\\\\x5c ..." against "REQUEST_HEADERS:Accept" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1162"] [id "920600"] [msg "Illegal Accept header: charset parameter"] [data "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.4"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [hostname "api1pdts.movistar.es"] [uri "/"] [unique_id "Y6Efa9KiCof05rv4cTuR2gAAAAs"]

Here is another one:

[Mon Dec 19 21:45:10.407002 2022] [:error] [pid 14860] [client 51.104.xxx.xx:56676] [client 51.104.xxx.xx] ModSecurity: Access denied with code 403 (phase 2). Pattern match "^[\\\\d.:]+$" at REQUEST_HEADERS:Host. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "736"] [id "920350"] [msg "Host header is a numeric IP address"] [data "45.79.xxx.xx"] [severity "WARNING"] [ver "OWASP_CRS/3.3.4"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [hostname "45.79.xxx.xx"] [uri "/"] [unique_id "Y6EhtvJk_bMxXPQEVlY4BwAAAAY"]

In reporting these to microsoft and amazon they want the element that triggered the alert. To me that looks like the long string of slashes in the first example and in the second example, well, I don't know.

In order for Microsoft and Amazon to close these accounts they need evidence. That I'm at a loss to provide.

 

[ciao70]

Thanks. Following is a sample output from mod_sec as compiled by csf for reporting. I'm particularly interested in detecting the actual links or code that references some kind of malicious activity because it's that code that needs to be submitted to the amazons, googles and microsoft abuse forms. I know, I know I can block the entire cloud through updated bulk lists but there are downsides to that namely it may also block legitimate links to my website on the same server which is being attacked, or so I've read.

Time: Fri Dec 16 08:51:39 2022 -0500
IP: 20.106.xx.xxx (US/United States/-)
Failures: 3 (mod_security)
Interval: 3600 seconds
Blocked: Temporary Block for 86400 seconds [LF_TRIGGER]

Log entries:

[Fri Dec 16 08:51:36.978979 2022] [:error] [pid 1533] [client 20.106.xx.xxx:60771] [client 20.106.xx.xxx] ModSecurity: Access denied with code 403 (phase 1). Matched phrase "/.env" at REQUEST_URI. [file "/etc/apache2/conf.d/modsec_vendor_configs/comodo_apache/02_Global_Generic.conf"] [line "117"] [id "210492"] [rev "3"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "45.xx.xxx.xxx"] [uri "/.env"] [unique_id "Y5x36Nr2w2PAPpsj5BClEgAAAAQ"]


[Fri Dec 16 08:51:37.067157 2022] [:error] [pid 1534] [client 20.106.xx.xxx:60799] [client 20.106.xx.xxx] ModSecurity: Access denied with code 403 (phase 2). Pattern match "^[\\\\d.:]+$" at REQUEST_HEADERS:Host. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "736"] [id "920350"] [msg "Host header is a numeric IP address"] [data "45.xx.xxx.xxx"] [severity "WARNING"] [ver "OWASP_CRS/3.3.4"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [hostname "45.xx.xxx.xxx"] [uri "/"] [unique_id "Y5x36XlAfDts4S2Bv_HczAAAAAY"]


[Fri Dec 16 08:51:37.127667 2022] [:error] [pid 2996] [client 20.106.xx.xxx:60810] [client 20.106.xx.xxx] ModSecurity: Access denied with code 403 (phase 1). Matched phrase "/.env" at REQUEST_URI. [file "/etc/apache2/conf.d/modsec_vendor_configs/comodo_apache/02_Global_Generic.conf"] [line "117"] [id "210492"] [rev "3"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "45.xx.xxx.xxx"][B] [uri "/.env"][/B] [unique_id "Y5x36QqyfSmS8IZWgytVmQAAAAc"]


When I see [uri "/.env"] I believe that would indicate a malicious user scanning for enivronmental variables, which are known to contain credentials.

Hi,

The Modsecurity settings with CSF seem too restrictive to me.

I'd put

Failures: 5 (mod_security)
Interval: 300 seconds
Blocked: Temporary Block for 3600 seconds [LF_TRIGGER]

This way you avoid blocking even any legitimate traffic for some false positives.

In the event of an attack, Modsecurity already blocks it on its own, then on the 5th attempt in 5 minutes CSF intervenes and blocks it for 1 hour

I also point out that after the update of Modsecurity to 2.9.6 and OWASP to 3.3.4 some problems arose with some rules that did not appear before

Modsecurity 2.9.6 [Fix Security]

Hi, Do you use OWASP CRS? Yes, it is the ruleset causing the issues AFAIK, previously no problems at all.

forums.cpanel.net

In your log I saw this

[Mon Dec 19 21:35:23.087209 2022] [:error] [pid 14858] [client 85.62.xxx.xx:59737] [client 85.62.xxx.xx] [id "920600"]

This rule started to give some problems right after the update of Modsecurity and OWASP

It is probable that the IP 85.62.xxx.xx belongs to the admantx crawler ( Advertiser) which after OWASP update is blocked by the rule 920600

User-Agent: ias-ir/3.1 https://www.admantx.com/service-fetcher.html

Some things I wouldn't worry too much about, let Modsecurity and CSF do their job

 

[jeffschips]

Thank you for that consice explanation - very good! I'm a bit stumped on where in csf you are finding the "Failures: 5 (mod_security)"

Is it in this stanza?

# [*]Enable failure detection of repeated Apache mod_security rule triggers
LF_MODSEC = "5"
LF_MODSEC_PERM = "86400"

 

[ciao70]

Yes

One hour is enough for MODSEC_PERM.

At the fourth MODSEC intervention in 24 hours it will be permanently blocked

LF_MODSEC_PERM = 3600

 

[jeffschips]

Thanks again. you know I noticed that a lot of entries where building up in mod_sec logs and that there was also a repeating error in the apache/error_logs:

[Wed Dec 21 22:53:47.817244 2022] [:error] [pid 2788] [client 104.28.xx.xxx:11195] [client 104.28.xx.xxx] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/xxxxxxxxxxxxxx-global": Permission denied [hostname "xxxxxxx.xxxxxxxxxx.com"] [uri "/lists/ut.php"] [unique_id "Y6PUy1uEuUkwJm840V02XAAAAAs"]

Research showed that the "failed to access DBM file" as explained here: ModSecurity failed to access DBM file- common causes and fixes

meant there was a configuration error. I followed the recommendations here, in that posting:

Comparatively, switching to a combination of event and mod_suexec Multi-Processing Module is a safer solution. One can simply enable the Event MPM and disable the Worker MPM in the EA4 build profile. Once you start the next build, the EasyApache 4 update process will automatically take care of the installation.

Doing so immediately stopped the onslaught of hits against mod_security, so much so that I though mod_security was down. But it wasn't - it was functioning. So the question is why would following the suggestion immedately stop the onslaught of hits against mod_security whilst also stop any entries in the error_log?

Plus I'm wondering if removing mod_mpm_prefork, mod_ruid2 and mod_cgi and adding mod_cgid and mod_mpm_event somehow stopped reporting or otherwise compromised the system.

Stumped. . .

 

[ciao70]

Hi,

How to fix Failed to access DBM file

If you see the error in your Apache error log: [Wed Aug 19 15:35:40.205810 2020] [:error] [pid 2532] [client 91.199.212.132:50312] [client 0.0.0.0] ModSecurity: collections_remove_stale: Failed t...

support.cpanel.net