Log Rotation

[feanor]

portman: you are very very lucky to have come across such a cordial group of clients.
:p

In the request, don't forget an option to TRIM/TRUNCATE to a certain number of lines (or by date, specifically). backups could be tricky, considering how LARGE domlogs can get, depending on the server and who's who in the zoo on the box in question. Perhaps that could be tied into the backup facility?
(hopefully /domlogs is gzipped or more when that occurs)

just food for thought

 

[Elena]

Don't forget to have this completely turned off! :p

Like some have mentioned, not every client is going to have their logs cut on the same day if you do have a script already installed. Some clients may require weekly if their logs are huge, some may only require monthly.

Option to turn it off completely, if I can keep using the script I have in place now I'd be verry happy.

 

[portman]

Lucky

I am I have great clients for the most part. Let's not mention that great piece of software cpanel we all are lucky to have!

I guess I will write up the enahancement request.

 

[moronhead]

I have never seen Cpanel rotating logs on the 31st of a month. It does its job either on the 28, 29 or 30th. This isn't something new either, it's been working like that since at least since January 2001.

The previous domlogs always seem to get deleted once rotated, unless they get tucked in a secret location that we don't know of. The only exceptions are the parked, pointed and redirected domains, they accumulate the logs month after month. I guess that's because no stats are produced for them.

 

[portman]

Moronhead. I have never seen the logs rotate on my servers

Anyway here is the proposed change...I will edit for comments. But since this seems a great option I figure why just please me

Ehancement Request :: Logrotate

Purpose: To enhance the manageability of the logrotate utility. To provide customizable solutions to permit cpanel users to accommodate clients who require more advanced statistics evaluation.

Options:

Multiple Logs: Permit the customization of the number of logs to keep in historical incrementation. Example: A 4 log history will keep on disk the last 4 rotations of the log.

Multiple Increments: Permit the customization of time frame of rotation. Example: Weekly, Bi Monthly, Monthly rotation of the logs.

Multiple Log Cleanup: Permit the customization of rotation method. Example: Zero Out of Log (Current Solution) or if Multiple Logs is checked rotation to “old” file(s).

 

[electro]

Hey people...

I'm just a small end user of cpanel.. It's great apart from this one thing! This rotate thing just started on my server last month lost a load of logs for me and my clients, the current situation is not good, it's hard to write a crontab for the end of the month! (when to copy? 25? 26? 27? 28? 29? 30? 31? )Better at the begining... (good old 1!)

The above proposals by portman are good, however, a &turn off all log rotation& switch would be ideal for those of us that like to just write a small crontab to copy, gzip, backup and generaly manipulate our logs ourselves. This would allow it to be done on a daily basis for really busy/ closely looked after sites, to bimonthly or whatever for small sites that will be looked at rarely.

Please make this work for us, everytime I get something set up for these raw logs, something changes and I have to redo a bunch of crontabs, plus loose a week/monts worth of stats.

Apart from this, Cpanel rocks

 

[portman]

I'll add the turn off completely to the list. Good observations!

 

[roman]

Arghh

Looks like we lost all of August's logs at 2am... :-( Cpanel needs an option to move them somewhere rather than deleting the logs... Logs are very important to some of our customers... They are also detailed proof of usage.

 

[ecoutez]

Ugh, same here...

Any word on if this will be addressed? It doesn't seem that we can easily do this ourselves since the code is in cpanellogd. While modifying the code is possible, I suspect it'll just be overwritten when CPanel next updates.

The obvious workaround is to maintain multiple sets of logs, but that's such a huge waste. Ideally we should be able to specify...
- How many months worth of logs to keep
- If these should be zipped
- Where these should be stored
- etc

I'd love to see logs rotated more often than once a month. On my other servers, I run a log rotation nightly, and rotate logs over a certain size (a few megs in my case works well). I don't like having large active logs when I can avoid it.

- Jason

 

[unite the cows]

anyone know where the httpd logs are at all? In /var/log/httpd/ there is nothing. I am kind of curious as to why this is. My server just recently crashed and i have no clue why cause I cannot read or find some logs

 

[ecoutez]

Apache Server logs: /etc/httpd/logs
Apache Domain (VirtualHost) logs: /etc/httpd/domlogs/

- Jason

 

[unite the cows]

Failed loading /usr/local/Zend/lib/ZendOptimizer.so: /usr/local/Zend/lib/ZendOptimizer.so: cannot open shared object file: No Such File or Directory

What is causing this and what can I do to fix this? Sorry to get off topic. Thank you for showing me the correct place for the logs.

 

[Juanra]

There are several /scripts/installzendopt-php*** scripts to install the Zend Optimizer according to you PHP version.

 

[unite the cows]

what if I am using php4.2.2 i looks like 4.2.1 is the latest on /scripts/installzendopt-php**** is it ok to use the 421 ?

 

[ecoutez]

[quote:665b9648ba][i:665b9648ba]Originally posted by bdraco[/i:665b9648ba]

Logs are rotated on the last day of the month, they always have been AFAIK. My guess is that one of the latest patches fixed rotation on the servers that is was broken on.[/quote:665b9648ba]

I'm sure you know, but domlogs aren't rotated... they're just deleted. I can't be the only one getting support requests from clients asking for raw log files which were deleted on the last day of the month.

I'd rather have CPanel not touch the domlogs at all than to throw them away completely. At least then I could setup an entry in logrotate.d and rotate them monthly instead of deleting the data entirely.

Any chance of allowing us to specify a rotation directory? Something like /etc/httpd/domlogs/rotated/ would be great for me. Maybe as part of version 5? *begging*

Thanks,
Jason

 

[showdoc]

log deletion -

hey folks -

this is my first forray into web hosting and (like you all) just lost our first month's worth of logs... argh!!!! although i did grab one from the 23rd... just playing around with it - trying out some other analyzing software - lucky that i did...

but, really, at THE VERY LEAST move a copy of the log into a designated folder BEFORE deleting it...and let us delete it when we want to... my logs are only like 300k...

and, hehehehe, like electro mentioned - sooo, exactly WHEN are we supposed to copy the &last& log of the &full& month? and what time? and if we copy it too early, we STILL LOSE the last data of the month when it gets deleted again!

it's very critical for us to &SEE& where and specifics of ALL the visitors to our site are.... (and i'm sure there are MANY companies out there that NEED that same information)

i just went round and round today with my hosting co. and read them the riot act about the deletion of the logs... if i was developing software for web hosting/analysis, i know that keeping a RECORD of ALL web activity would be of PARAMOUNT IMPORTANCE to my clients... they NEED that information to survive!!!! if they want to delete it themselves after they receive it, that's fine... but geeeeeeeeezzzz!!! =)

morelater
the show doctor

 

[dwh]

Summarize and delete

There's nothing wrong with deleting raw data after a period of time. It's unlikely that a particular user needs a particular line of what someone did at 8:15 am 3 months ago. What everyone seems to be missing is the basics on how this has been handled traditionally.

If data is rolled up properly, the few that need raw data can be given certain options. If the files are bigger then the HD then you need backup on tape. No reason HD has to store this. Pay for it, should be easy for CPanel to give u any option you want.

As for the data rollup, what stats do you need? I think every unique referer is important. If mr.X visited your site 100 times, all you need is 1 line summarizing that info as such:

IP | # of visits
-----------------
Mr. X | 100
Mr. Z | 212
etc..

If url http://y.com/somepage.html refered 1000 people to your site, you don't need every instance of that one month later. You need one line:

referer | # of referals
---------------------------
http://Y.com/somepage | 1000
http://x.com | 212

etc..

I'm oversimplifying it, but you just need to figure out the important unique data you want to keep, define that key, do regular rollups and everyone will be happy.

The problem is, if you wait a full month to handle rollups, especially if you use perl or something like that, the server will croak on all that data. But if you do a daily rollup, then summarize that at the end of the month, you have manageable and useful data that doesn't overwhelm your server.

This is definitely a really big issue that CPanel needs to address.

 

[Juanra]

We must keep logs for as much as 12 months.

 

[thomas]

Don't see what the problem is, you chose if the logs should be deleted or not in &Tweak Settings&.

 

[dwh]

Thomas,
I was confused by your comment because I didn't have any of those options, but it turns out I had a very old version of cpanel on my server. Unfortunately I upgraded. My server loads went through the roof. My domain stopped logging altogether. I get emails every 6 hours that my stats failed. I'm very confused and upset and don't know what to do! The log for one domain is over 2 Gigabyte.