/var/cpanel/secdatadir/ip.pag has grown massive

DigitalEssence

Well-Known Member
May 21, 2014
49
5
8
cPanel Access Level
Root Administrator
Hi,

as per this other thread, my ip.pag file has grown huge (7.54GB)

I deleted the file and then restarted Apache but it instantly jumped up to 8GB.

What are my next steps? Apart from upgrading to EasyApache 4
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello,

We provide the following utility for EasyApache 4 only:

ModSecurity SDBM Utility - EasyApache 4 - cPanel Documentation

EasyApache 3 is nearing deprecation status, and thus new utilities like the one above are not planned for it. The following thread was recently posted to note this:

EasyApache 3: It's been a long road, but it will be time to say goodbye soon.

May I ask what in-particular is keeping you from upgrading to EasyApache 4? It's really the best solution to address this particular issue, aside from disabling Mod_Security.

Thank you.
 

4u123

Well-Known Member
PartnerNOC
Jan 2, 2006
939
22
168
We have this utility installed on all servers but we still get multiple warnings every day. This has been happening much more in the last couple of weeks.

If I look in my inbox right now I currently see notifications from every one of our servers as follows...

ModSecurity persistent IP database (/var/cpanel/secdatadir/ip.pag) size is 8.25GB

Recently I was seeing two or three of these every day, sometimes more - but today it's gone crazy, with notifications from every server. When you delete the file the same problem happens again within a few hours.

Can anyone advise how we can prevent this? What does this file actually do? Why is so much data written to it? I'm pretty sure there can't be 8GB's worth of IP entries being generated every day, surely that would amount to several million modsec rules being triggered, assuming it is storing the IP's that have been blocked via modsecurity? What use would this have anyway - considering the firewall can't block that many. Isn't there supposed to be a modsec database for all this stuff?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello @4u123,

Just to clarify, have you installed the ea-modsec-sdbm-util utility on these servers?

Code:
yum install ea-modsec-sdbm-util
Once it's installed, the maintenance portion of the nightly cPanel update process will automatically call the utility. If it's already installed on your servers, can you verify if you've made any modifications to prevent the daily cPanel update process from running (the upcp process should still run even if automatic updates are disabled)?

Thank you.
 
Last edited:

4u123

Well-Known Member
PartnerNOC
Jan 2, 2006
939
22
168
Yeah it's installed on all our servers. I don't remember this being installed by any of us but I just tried to install it and it's already there.

This feature request says that there is now an option to rotate this file...

Rotate -var-cpanel-secdatadir-ip.pag

...However, the file isn't listed in the "log rotation" option in WHM and the feature request conclusion doesn't provide any details of what I should be looking for in order to rotate it. Obviously I can do this manually, but there's supposed to be an option for it - unless that feature request is referencing this SDBM utility, but I wouldn't want to assume that.

I'm puzzled as to why this might suddenly be happening so frequently. I could put it down to huge increase in malicious bot activity but I don't know enough about what the file ip.pag is used for and the processes involved. Is this a cpanel thing? considering it's in /var/cpanel I presume so, rather than it being part of the usual ModSecurity config on any server.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
unless that feature request is referencing this SDBM utility, but I wouldn't want to assume that.
Hello,

It is in-fact referencing that utility, and it should be installed automatically on EA4 systems (I've updated my prior post to reflect that). Here's the quote from the cPanel 62 Release notes:

ModSecurity SDBM Utility
We added the ModSecurity SDBM Utility to cPanel & WHM version 62 systems that run EasyApache 4. This utility purges expired entries from the /var/cpanel/secdatadir/ip.pag cache file. The system runs this utility as a part of the nightly maintenance script, and we strongly recommend that you allow the maintenance script to run the utility.
Can you verify if manually running the "/scripts/shrink_modsec_ip_database -x" command on an affected system reduces the log file size? If so, and if the data quickly repopulates again to a large size, feel free to open a support ticket using the link in my signature so we can take a closer look to see what's happening.

Thank you.
 

4u123

Well-Known Member
PartnerNOC
Jan 2, 2006
939
22
168
It does work yes - and it doesn't suddenly (within 5 minutes) increase to a large size. My guess would be the files are getting "shrunk" correctly when the cron runs, but they are growing because a lot of data is being written to them.

Can you offer some possible reason for 8GB of information to be added to that file within a 24 hour period? Surely that's the real problem here?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello @4u123,

Do you happen to use the Comodo WAF Mod_Security ruleset? I know there was an issue in the past with their ruleset that lead to a problem with the ip.pag file growing to a large size. It was fixed in their 10/06/2017 update:

Rules Updates: Changelog - Free Modsecurity rules - Comodo Web Application Firewall | Page 9

Other than that, we do have an internal case open (CPANEL-16280) to note instances where the ModSecurity SDBM utility fails to shrink the ip.pag file during the cPanel update process (but works manually). We've not been able to reproduce the issue internally, so we are asking anyone facing that issue to open a support ticket so that we can take a closer look at an affected system.

Thank you.