ModSecurity: collections_remove_stale: Failed to access DBM file

Operating System & Version
CloudLinux 3.10.0-962.3.2.lve1.5.32.el7.x86_64
cPanel & WHM Version
v.86.0.18

jndawson

Well-Known Member
Aug 27, 2014
274
28
28
Western US
cPanel Access Level
DataCenter Provider
We've looked at all the similar posts regarding this error, but all of the discussions apply to using mod_ruid2 or mpm_itk. We are using mpm_prefork & lsapi.

Customer complained about not being able to consistently access his cPanel portal without having to reload the page, and sometimes doesn't have access at all. Gets this error immediately upon logging in:
"customer.tld is not responding" and includes a "Recover webpage" button.

From the Apache log:
Code:
[Tue Apr 28 09:15:04.208002 2020] [:error] [pid 3490123] [client 12.34.56.78:49432] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/styled/current_style/sprites/icon_spritemap.png"] [unique_id "XqhWiI6b9kkqpENtqVbmfgAAAB8"], referer: https://cpanel.reliablefencecompany.com/
[Tue Apr 28 09:15:04.208053 2020] [:error] [pid 3490123] [client 12.34.56.78:49432] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/styled/current_style/sprites/icon_spritemap.png"] [unique_id "XqhWiI6b9kkqpENtqVbmfgAAAB8"], referer: https://cpanel.customer.tld/
[Tue Apr 28 09:22:02.948384 2020] [:error] [pid 3491089] [client 12.34.56.78:49514] client denied by server configuration: proxy:http://127.0.0.1:2082/cpsess7316180901/backend/passwordstrength.cgi, referer: https://cpanel.customer.tld/
[Tue Apr 28 09:22:02.950066 2020] [:error] [pid 3491089] [client 12.34.56.78:49514] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/403.shtml"] [unique_id "XqhYKs3PgVaZDxwoJSC-kgAAAAM"], referer: https://cpanel.customer.tld/
[Tue Apr 28 09:22:02.950098 2020] [:error] [pid 3491089] [client 12.34.56.78:49514] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/403.shtml"] [unique_id "XqhYKs3PgVaZDxwoJSC-kgAAAAM"], referer: https://cpanel.customer.tld/
From mod_sec log:
Code:
[28/Apr/2020:09:15:04 --0700] XqhWiI6b9kkqpENtqVbmfgAAAB8 12.34.56.78 49432 123.456.789.012 443
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/styled/current_style/sprites/icon_spritemap.png"] [unique_id "XqhWiI6b9kkqpENtqVbmfgAAAB8"]
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/styled/current_style/sprites/icon_spritemap.png"] [unique_id "XqhWiI6b9kkqpENtqVbmfgAAAB8"]
[28/Apr/2020:09:22:02 --0700] XqhYKs3PgVaZDxwoJSC-kgAAAAM 12.34.56.78 49514 123.456.789.012 443
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 67.170.161.151] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/403.shtml"] [unique_id "XqhYKs3PgVaZDxwoJSC-kgAAAAM"]
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/403.shtml"] [unique_id "XqhYKs3PgVaZDxwoJSC-kgAAAAM"]
/var/cpanel/secdatadir:
Code:
[ [email protected] ~># ls -l /var/cpanel/secdatadir/
total 0
-rw-r----- 1 root root 0 Feb 10 16:15 global.dir
-rw-r----- 1 root root 0 Feb 10 16:15 global.pag
-rw-r----- 1 root root 0 Feb 10 16:15 ip.dir
-rw-r----- 1 root root 0 Feb 10 16:15 ip.pag
-rw-r----- 1 root root 0 Feb 10 16:15 nobody-global.dir
-rw-r----- 1 root root 0 Feb 10 16:15 nobody-global.pag
-rw-r----- 1 root root 0 Feb 10 16:15 nobody-ip.dir
-rw-r----- 1 root root 0 Feb 10 16:15 nobody-ip.pag
The weird thing is this entry:
Code:
[Tue Apr 28 09:22:02.948384 2020] [:error] [pid 3491089] [client 12.34.56.78:49514] client denied by server configuration: proxy:http://127.0.0.1:2082/cpsess7316180901/backend/passwordstrength.cgi, referer: https://cpanel.customer.tld/
Is the whole thing related to password strength?
 
Last edited:

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
11,214
1,006
313
Houston
I'm actually curious if the whole password error is unrelated. My test server has everything in this directory set to be owned by nobodoy:

Code:
[[email protected] lauren]# cd /var/cpanel/secdatadir/
[[email protected] secdatadir]# ls -lah
total 16K
drwxrwx--T   2 root   nobody 4.0K Apr 28 16:00 .
drwx--x--x 129 root   root    12K Apr 28 16:42 ..
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 global.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 global.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 ip.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 ip.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-global.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-global.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-ip.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-ip.pag

and the secdatadir with the following permissions:
Code:
[[email protected] cpanel]# stat secdatadir/
  File: ‘secdatadir/’
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: fd01h/64769d    Inode: 655512      Links: 2
Access: (1770/drwxrwx--T)  Uid: (    0/    root)   Gid: (   99/  nobody)
Access: 2020-04-28 16:45:02.918217471 -0500
Modify: 2020-04-28 16:00:01.936524944 -0500
Change: 2020-04-28 16:00:01.936524944 -0500
 Birth: -
Now, I'm using the event mpm but I do believe the ownership of these files should not be associated with root.
 

jndawson

Well-Known Member
Aug 27, 2014
274
28
28
Western US
cPanel Access Level
DataCenter Provider
After poking around, it seems all of our CloudLinux7 boxes have root:root as owns in secdatadir, and our Centos6 boxes have nobody:nobody. Everybody's nobody:nobody now, but the the question remains: Why?
 

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
11,214
1,006
313
Houston
It may be in the way that CloudLInux jails VirtualHosts or it may be a specific security measure they have implemented, maybe ruid2 was installed at some point then removed, customized installation of modsecurity (as in what is added with imunify), etc. As indicated previously there's literally no way for me to know. It's a whole lot less complicated to identify the cause when it happens. You could even create an audit rule to log the change 6.5. Defining Audit Rules Red Hat Enterprise Linux 7 | Red Hat Customer Portal
 

jndawson

Well-Known Member
Aug 27, 2014
274
28
28
Western US
cPanel Access Level
DataCenter Provider
More info: Our Centos7 boxes have nobody:nobody owns.
Our CloudLinux boxes are practically brand new, out of the box installations. ruid2 was never used. It seems the CloudLinux installation script(s) are missing a step. Not surprising as we've run into several while installing the most recent CL boxes.