domlog rotation: The Definitive Answer?

[payne]

OK. After poking around my logs, seeing how big they are, here is what I decided my apache log/report policy should be:

1) rotate logs monthly, compressing the previous month's logs and deleting the earlier compressed logs... just leaving the current months logs (being added to) and the previous month's compressed log.

2) allow my customers to download the previous month's compressed log (giving them one month to do so before it is deleted).

3) I will personally ftp all their logs to a storage server and then, once a year actually mail them a cd with all of their previous year's logs burned on it (at least for the big clients). At the same time, I will make myself a cd and clean all the logs off my storage server, ready for the new year.

Also, I run urchin and it, in the best case scenario, produces reports that are about 10-12% of the size of the raw logs each month. Because of this, I can't keep them around forever. So my policy is to:

1) not count urchin reports against my account's disk quota
2) delete any urchin reports older than a year

I am pretty sure these policies will make even my most picky customers happy. Now the only question is, how do I get this to work in the cpanel context?

Specifically:
- how do I get the /home/account/tmp folder to not count against a customer's quota?
- which utility should I use to rotate raw logs and compress the previous months logs (and delete older)? WHM? logrotate? urchin?
- how can I make the previous month's logs easily available to clients?
- how does this all work with the cpanel options to archive previous months raw logs?
etc. etc.

Any answers to any of these questions would be appreciated by myself and, I'm guessing, to many others, too.

cPanel.net Support Ticket Number:

[payne]

ok... I guess there is no way to have the /tmp folder not count against a customer's quota (this is a response from a feature request ticket):

Hello, actually I'm afraid this is a bit more complicated than that, it would require re-writing a lot of the cpanel code as it relies quite a bit on the files in the /tmp directory being owned by the user otherwise it will freak a few other programs out. So at this time I can't see this as being feasable to be added into the system. It is a good idea but just a bit hard to implement easily at the moment.
--
Thanks
Clifford,

[GOT]

I know its not very scientific or technical, but I just give my users an extra 20 megs to compensate for the tmp and logs.

[payne]

On setting up the rest of it, it looks like I could set up WHM to not keep the log files at the end of the month (Tweak Settings) which would produce a new file each month... and set each individual cpanel to 1) archive log files and 2) remove previous month's archive (in the raw log manager).

Since this is the policy for the whole server, does anyone know how to have each account set with these options by default?

Then I would need to figure out how to easily ftp the previous month's logs for each client out to my storage computer.

[payne]

ok... the settings for archiving and removing old archives for log files are stored in: /home/{user}/.cpanel-logs

the contents are:

archive-logs=1
remove-old-archived-logs=1

So, I just need to copy this file into each home directory... and should be able to put this file in the skeleton directory to make it a default setting for all new accounts.

Then I can either disable this feature from the cpanels or edit the skins to not have this option editable (if I don't want my cpanel users to modify these settings).

Very cool. So we can have cpanel manage all the logs and, with these settings, have the first and second items of my desired policy taken care of (rotate the logs, let the users download the previous month, etc.)

Left to do:
a) figure out an easy way to download every user's raw logs from each month: an ftp script? or modify cpanellogd to make a copy into a backup directory after rotating.
b) figure out how to handle quotas... now raw archived logs will be stored in /home/{account}/logs and urchin logs, which are about 20% the size of the raw logs will be stored in /home/{account}/tmp/urchin/data
c) automate the deletion of urchin reports older than a year.

Definitely making some progress. Does anyone else care about this stuff?

[Website Rob]

I'm sure we are all interested, just to different degrees.

[GOT]

definately interested.

[payne]

ok... so far everything worked. I copied the .cpanel-logs file into all existing accounts and into the skel directory (which works for new accounts). At the end of last month, everything worked beautifully... each account now has a 'logs' directory with {domain}-Oct-2003.gz in it--which is the compressed log that used to be in domlogs. Now domlogs has rotated and is starting new for the new month. I decided to leave the cpanel log manager options available to users and count any space the zipped logs take against their account. That way they can keep as many as they want there by unchecking the 'delete previous months' option or they can keep no raw logs by unchecking the 'keep' option. Whatever they want.

Now that I have those logs to play with, I found out that getting all the logs into a single directory to easily ftp them out is as easy as:

cp /backup/logs/ /home/*/logs/*.gz

then *poof* all the logs are in the /backup/logs directory (which I made for this purpose. Now I just download all the logs from this directory (then delete it to save space) and I have a backup copy of all the logs and can do whatever I want with them... namely to gather them together for my big customers and burn them to a cd once a year.

I can add this to a script along with the rm -f /backup/logs/* (before it) and call it from a CRON job to run on the first of each month. So far, everything is automated except for having to ftp down the logs sometime during each month. I can just have a CRONed lftp script do it for me if I wanted...

I think the only thing left to figure out is how to not count Urchin reports against the account's quota (they will add up significantly over the year) and delete any Urchin reports older than a year (should be easy to do with a script).

If anyone has tried this, let me know! Apparently this might be able to be done by changing the ownership... but then they might not be able to see the reports... maybe a group?

[payne]

I guess there is no way to not have the urchin reports count against an account's quota. I changed the group and user ownership on the reports to 'cpanel', (and then ran fixquotas) but they are still being counted against the account. Anyone else know how to do this?

[payne]

ok... maybe I spoke too soon. If I change the ownership this way:

chown -R cpanel:cpanel /home/{account}/tmp/urchin/data/reports/*

then it seems to work. The urchin reports are not counted against the user's quota while the user can still view the reports from their cpanel.

I tried changing the ownership further up the tree (like in the /home/{account}/tmp directory), but the reports were then not viewable through the cpanel. Apparently the permissions on the directories are more restrictive. I suppose you *could* change the permissions on the directories, but the simple ownership change of the reports seems to do the trick. Now the last test is to see if the reports can still be generated (moved) in this directory with the new ownership. I guess I'll let you know tomorrow.

[dysk]

Hi-
Since cPanel sets up the quota's against the users and not the groups, you could even chown -R cpanel:username ~/blah/blah/; chmod -R 770 ~/blah/blah/
That ould make it so that users can't snoop on each other's logs, and it still wouldn't count against the person's quota.


Regards,
Erek Dyskant
Unix Consultant

[payne]

reports are still generated in the account's tmp folder, so all is good. However, the permissions are reset after they are generated... so, I guess the last piece of this whole puzzle is to set up a nightly (or every 6 hours or something) cron job to set the permissions. The better bet would be to trigger the permission change at the end of the log script... but that would be customizing the installation, which we have been trying to stay away from.