ModSecurity: collections_remove_stale: Failed to access DBM file

[jndawson]

We've looked at all the similar posts regarding this error, but all of the discussions apply to using mod_ruid2 or mpm_itk. We are using mpm_prefork & lsapi.

Customer complained about not being able to consistently access his cPanel portal without having to reload the page, and sometimes doesn't have access at all. Gets this error immediately upon logging in:
"customer.tld is not responding" and includes a "Recover webpage" button.

From the Apache log:

[Tue Apr 28 09:15:04.208002 2020] [:error] [pid 3490123] [client 12.34.56.78:49432] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/styled/current_style/sprites/icon_spritemap.png"] [unique_id "XqhWiI6b9kkqpENtqVbmfgAAAB8"], referer: https://cpanel.reliablefencecompany.com/
[Tue Apr 28 09:15:04.208053 2020] [:error] [pid 3490123] [client 12.34.56.78:49432] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/styled/current_style/sprites/icon_spritemap.png"] [unique_id "XqhWiI6b9kkqpENtqVbmfgAAAB8"], referer: https://cpanel.customer.tld/
[Tue Apr 28 09:22:02.948384 2020] [:error] [pid 3491089] [client 12.34.56.78:49514] client denied by server configuration: proxy:http://127.0.0.1:2082/cpsess7316180901/backend/passwordstrength.cgi, referer: https://cpanel.customer.tld/
[Tue Apr 28 09:22:02.950066 2020] [:error] [pid 3491089] [client 12.34.56.78:49514] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/403.shtml"] [unique_id "XqhYKs3PgVaZDxwoJSC-kgAAAAM"], referer: https://cpanel.customer.tld/
[Tue Apr 28 09:22:02.950098 2020] [:error] [pid 3491089] [client 12.34.56.78:49514] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/403.shtml"] [unique_id "XqhYKs3PgVaZDxwoJSC-kgAAAAM"], referer: https://cpanel.customer.tld/

From mod_sec log:

[28/Apr/2020:09:15:04 --0700] XqhWiI6b9kkqpENtqVbmfgAAAB8 12.34.56.78 49432 123.456.789.012 443
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/styled/current_style/sprites/icon_spritemap.png"] [unique_id "XqhWiI6b9kkqpENtqVbmfgAAAB8"]
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/styled/current_style/sprites/icon_spritemap.png"] [unique_id "XqhWiI6b9kkqpENtqVbmfgAAAB8"]
[28/Apr/2020:09:22:02 --0700] XqhYKs3PgVaZDxwoJSC-kgAAAAM 12.34.56.78 49514 123.456.789.012 443
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 67.170.161.151] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/403.shtml"] [unique_id "XqhYKs3PgVaZDxwoJSC-kgAAAAM"]
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 12.34.56.78] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "cpanel.customer.tld"] [uri "/___proxy_subdomain_cpanel/403.shtml"] [unique_id "XqhYKs3PgVaZDxwoJSC-kgAAAAM"]

/var/cpanel/secdatadir:

[ [email protected] ~># ls -l /var/cpanel/secdatadir/
total 0
-rw-r----- 1 root root 0 Feb 10 16:15 global.dir
-rw-r----- 1 root root 0 Feb 10 16:15 global.pag
-rw-r----- 1 root root 0 Feb 10 16:15 ip.dir
-rw-r----- 1 root root 0 Feb 10 16:15 ip.pag
-rw-r----- 1 root root 0 Feb 10 16:15 nobody-global.dir
-rw-r----- 1 root root 0 Feb 10 16:15 nobody-global.pag
-rw-r----- 1 root root 0 Feb 10 16:15 nobody-ip.dir
-rw-r----- 1 root root 0 Feb 10 16:15 nobody-ip.pag

The weird thing is this entry:

[Tue Apr 28 09:22:02.948384 2020] [:error] [pid 3491089] [client 12.34.56.78:49514] client denied by server configuration: proxy:http://127.0.0.1:2082/cpsess7316180901/backend/passwordstrength.cgi, referer: https://cpanel.customer.tld/

Is the whole thing related to password strength?

 

[cPanelLauren]

I'm actually curious if the whole password error is unrelated. My test server has everything in this directory set to be owned by nobodoy:

[[email protected] lauren]# cd /var/cpanel/secdatadir/
[[email protected] secdatadir]# ls -lah
total 16K
drwxrwx--T   2 root   nobody 4.0K Apr 28 16:00 .
drwx--x--x 129 root   root    12K Apr 28 16:42 ..
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 global.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 global.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 ip.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 ip.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-global.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-global.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-ip.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-ip.pag

and the secdatadir with the following permissions:

[[email protected] cpanel]# stat secdatadir/
  File: ‘secdatadir/’
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: fd01h/64769d    Inode: 655512      Links: 2
Access: (1770/drwxrwx--T)  Uid: (    0/    root)   Gid: (   99/  nobody)
Access: 2020-04-28 16:45:02.918217471 -0500
Modify: 2020-04-28 16:00:01.936524944 -0500
Change: 2020-04-28 16:00:01.936524944 -0500
 Birth: -

Now, I'm using the event mpm but I do believe the ownership of these files should not be associated with root.

 

[jndawson]

I changed owns to nobody:nobody and the log entries have stopped. Waiting on customer to see if they're still having issues.

 

[cPanelLauren]

Awesome, do also keep an eye for a bit on that directory to ensure it doesn't get changed back but I don't believe it should. The correct ownership for that should be nobody which enables apache to write to it.

 

[jndawson]

Customer is happy, no errors.

The question is why? Why are the owns wrong?

 

[cPanelLauren]

I have no way to know that, which is why I advised keeping an eye on the directory. If a cPanel process is changing it I'd assume after updating you'd see a change in ownership.

 

[jndawson]

After poking around, it seems all of our CloudLinux7 boxes have root:root as owns in secdatadir, and our Centos6 boxes have nobody:nobody. Everybody's nobody:nobody now, but the the question remains: Why?

 

[cPanelLauren]

It may be in the way that CloudLInux jails VirtualHosts or it may be a specific security measure they have implemented, maybe ruid2 was installed at some point then removed, customized installation of modsecurity (as in what is added with imunify), etc. As indicated previously there's literally no way for me to know. It's a whole lot less complicated to identify the cause when it happens. You could even create an audit rule to log the change 6.5. Defining Audit Rules Red Hat Enterprise Linux 7 | Red Hat Customer Portal

 

[jndawson]

More info: Our Centos7 boxes have nobody:nobody owns.
Our CloudLinux boxes are practically brand new, out of the box installations. ruid2 was never used. It seems the CloudLinux installation script(s) are missing a step. Not surprising as we've run into several while installing the most recent CL boxes.

 

[cPanelLauren]

It could be on purpose too, they run with different permissions on some items. If you purchased your CloudLinux licenses through us we'd be happy to take a look into the issue otherwise I'm sure they wouldn't mind taking a look if you opened a ticket with them here: CloudLinux - Main | New template

 

[WorkinOnIt]

My test server has everything in this directory set to be owned by nobodoy:

[[email protected] lauren]# cd /var/cpanel/secdatadir/
[[email protected] secdatadir]# ls -lah
total 16K
drwxrwx--T   2 root   nobody 4.0K Apr 28 16:00 .
drwx--x--x 129 root   root    12K Apr 28 16:42 ..
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 global.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 global.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 ip.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 ip.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-global.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-global.pag
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-ip.dir
-rw-r-----   1 nobody nobody    0 Apr 28 16:00 nobody-ip.pag

Now, I'm using the event mpm but I do believe the ownership of these files should not be associated with root.

This answer was helpful to me. I've just installed AlmaLinux and was getting the same issue.

I found the /var/cpanel/secdatadir/ files were "root root"

If you discover that, you can run this SSH command in terminal:

chown -R nobody:nobody secdatadir

 

[Jim Evans]

Same issue here on CentOS v7.9.200 and Ubuntu v20.04.4 hosts.
In my case the following were observed

CentOS

ls -l /var/cpanel/secdatadir/
total 0
-rw-r-----. 1 root root 0 Dec 23 2018 global.dir
-rw-r-----. 1 root root 0 Dec 23 2018 global.pag
-rw-r-----. 1 nobody nobody 0 Nov 5 06:00 ip.dir
-rw-r-----. 1 nobody nobody 0 Nov 5 06:00 ip.pag


Ubuntu

ls -l /var/cpanel/secdatadir/
total 0
-rw-r----- 1 root root 0 Aug 30 19:51 nobody-default_SESSION.dir
-rw-r----- 1 root root 0 Aug 30 19:51 nobody-default_SESSION.pag
-rw-r----- 1 root root 0 Aug 30 19:51 nobody-global.dir
-rw-r----- 1 root root 0 Aug 30 19:51 nobody-global.pag
-rw-r----- 1 root root 0 Aug 30 19:51 nobody-ip.dir
-rw-r----- 1 root root 0 Aug 30 19:51 nobody-ip.pag


If relevant one thing to note is that these hosts are also running CSF and ImunifyAV.