Stop deleting domlogs every 24h by default (better domlog rotation options)

[quizknows]

I had been discussing a potential option for better rotation/archival of Apache access logs by default on cPanel systems with David Grega. I had been receiving replies from integrationATcpanel.net regarding this issue (case 56071), however, the couple replies I got were months apart and it has been months since the last reply. I understand that David Grega has passed, and wish to offer my sincere condolences to his friends, family, and co-workers.

This case is extremely important to myself and many other admins that I know, and I wish to bring it into public light. In short, deleting the domlogs every time stats run is a horrendous situation for security admins who need to investigate compromised websites. I work for a host with hundreds of techs, and cannot remember the last time a single day passed that missing domlogs did not hinder a security investigation.

I'm sure (or I hope) you guys can find the e-mail thread and get someone on this again. It seems silly to me to delete the domlogs every time stats run, especially with the 24h default frequency of stats processing. Currently the only option is just to set "Delete each domain’s access logs after stats run" to "off" in Tweak Settings. There seems to be no easy or sensible option to archive or rotate these logs, except to enable archiving of raw access logs in each users home directory via each individual cPanel account.

In case others wish to chime in on this situation, here were the last couple exchanges on this issue:

David-

I thank you for your reply, and appreciate you offering a flexible solution. Let me see if I can better explain why I'm making this request; I'm not after a flexible solution, I'm after making it no longer default to nuke the access logs every 24 hours.

Basically, here's the deal. I understand that cPanel mainly uses these logs for stats, however, as a security admin, they're absolutely critical for investigating any compromise of web applications. If someone doesn't realise they've been compromised before their stats run, and their server is set to the default settings in WHM, we lose the logs that night; consequently we're left completely in the dark as to how they got hacked. This literally happens to me several times a day, where I cannot investigate a compromised site or server due to missing domain access logs.

I really don't care how it's handled in the user interface, whether it's by days, size, etc. Myself and all the other security/abuse admins I know just want to have something, anything, more than [a maximum of] 24 hours of domlogs kept by default. Even a default of three days would be infinitely better than the current situation. Even making it default to archive them in the users home directory (as the user level cPanel option for raw access lgos does) would be infinitely better than the current situation.

Honestly, whatever is easiest for you guys is fine with me. Ideally, I'd like to see "Keep X number of days worth of domain access logs" in the stats and logs menu under Tweak Settings.

Thank you for your time and attention to this.

Sincerely,

Greetings,

I talked to our Product Manager about your feedback.

For the sake of helping us implement a system to accommodate your needs, ignore our current system and hypothetically let's start at a clean slate.

Which of the following would be easiest for you:
- Having an option to set an upper limit for the number of days logs are retained
- Ability to just configure a retention period with larger intervals (e.g. 2 week, 1 month, 3 months)

Sincerely,
David Grega
go.cPanel.net/dg

David-

Thank you for your reply. After consulting with management and letting some other techs express their feelings, we feel that it would be best to have the ability to set domain access log retention for any number of days between 1 and 60, based upon the specific needs of that server. I believe this is along the lines of your first option of being able to set an upper limit for the number of days logs are retained.

More importantly, we feel that the retention period should be set to at least seven days by default, with the ability to further configure it as needed. That way in case an incident occurs, the retention will already be there, rather than setting it up in the aftermath of the incident as we have been doing.

Please keep me posted on this. Again, I appreciate your time on this issue.

Best,