Log Rotation

[sketchified]

When does cpanel rotate the user's domlogs (I noticed it's monthly, but the stats were rotated today for some reason) and where is this configured since I don't see it anywhere dealing with logrotate? We would like to setup custom rotations for some sites, but I don't want cpanel messing up a single user's rotation. For example, our custom rotation would be setup weekly and we don't want the monthly cpanel rotation messing it up one day before our custom rotation happens.

[sketchified]

uuum..tap, tap..this thing on?

[feanor]

In my experience, CPanel has never instilled any rotation of apache domlogs, ...... I think this is entirely in your hands.

So yea, by default they grow indefinitely.
Lovely, eh?

[Elena]

I see someone has already started this thread for me.

I have a log rotation script installed on the server, tonight it ran because it is set to do this monthly on all the domlogs. When I went to download the logs for about 16 clients who each month ask for their logs at the start of the month... I opened up the log and what do I seeeee!!! The logs, instead of containing data from May27-June30 only have information from June 30th-July1st!

CPanel not rotating logs you say?? I think not! Cpanel has to be rotating the logs... how I imagine what has just happened. Cpanel rotated the logs, then my script rotated the logs. Thus the logs I have on backup only have 1 days worth of information.

My provider just mailed me stating that &after peeking through the /usr/local/cpanel/cpanellogd code, and It appears that the logs are getting cleared. I need to look into it a bit more, but it does appear that cpanellogd is deleting the logs after it is done processing the log for the stats.&

Just deleting them!?! Holy crap, that's worse than having the log continue to grow on indefinetely. What about last months log stats? Are those now just completely gone, never to be seen again? Clients who really rely on the rawlog and use webtrends (pretty pricey software) are not going to be happy about this. Why wouldn't cpanel think of keeping a backed log, probably even have it in the control panel to download. If its always had the huge huge log.. I dont see why splitting it up and offering for download would be any worse.. only more convenient for the clients.

I also noticed like a week ago that there is a new default ftp login in the cpanel. username_logs - however I checked that to see if maybe backups were being moved in there.. and they aren't. So, I guess someone has some explaining to do. I've already had confirmed with 3 other cpanel hosts that they either have log data only from June30-July1 or only July 1 - aiye!

Ok.. that's the end of my rant. This only gets me angry because cpanel should be updating us on what is new/added in. if this was just a fluke and the logs aren't supposed to be removed.. then I'll understand. But if this is the new way they are thinking about handling the logs.. they best be preparing to make some backups and not just zap it to hell.

[Elena]

Did anyone else have their logs cut (rawlog in cpanel only showing from June 30th or July 1stand nothing from before this time).. or just a few hosts had this happen?

Does anyone else even care... hellooo?

[feanor]

OK, are you talking about stats output in individual users' control panels? That's an entirely different subject than saying the domlogs are being trimmed or completely truncated themselves.

I am confused by your post.

For some reason, we've always implemented a mechanism like this manually- CPanel does not take care of things within /usr/local/apache/domlogs .........

Has this changed recently? Did it change 8 months ago and nobody noticed? Am I the only one that didn't notice?

Anyone else have any comments on this issue? I am combing through a handful of machines, some old, some new- what's happening on everyone else's boxes?

[Elena]

This happened recently because last month my domlog rotation script worked perfect! Checking on if the script messed up, no it didn't. It ran once and did what it was supposed to do.

I am not talking about individual users control panel stats (not webalizer or analog) I am talking about the Apache domain logs (rawlog you download from the control panel).

---edit for more explanation
The rotation script I use moves the logs from where I store them /backup/domlogs to /backup/rotated-domlogs, it then deletes the old log from /backup/domlogs so they can start over for the new month.

Each month I go and download a few of the files in /backup/rotated-domlogs so I can give them to my clients who require the logs to use with Webtrends. Luckily I opened one last night and noticed the HUGE error before sending them. I opened all of them and they were all cut, showing information from June30-July first, ONLY 1 day worth of stats!

So if my script moves the logs from /backup/domlogs to rotated-domlogs, the logs in domlogs before my script got to them had to have already been cut by something else... I suspected cpanel and got confirmation from other cpanel hosts that dont have a rotation script that all their domlogs were cut too.

So I come here to see if anyone else had this happen and someone already had.. so I added on to this thread what I had just discovered to. Are you still confused? I'm trying my best to explain the situation and understand it myself.

[ZachICU]

I am curious also what happened to the logs?

I think there should be a setting if cpanel is going to rotate logs. What If I really need logs from last month and cpanel just decided to delete them?

[portman]

We too see our domlogs trucated to the previous mentioned dates on all our servers. Any word on why here?

[Brad]

So are we.. Did a grep for an IP on a target domains log and had no result. So, looked to see how far back log went and it was new for the day, others are still intact.

Something seems to have changed though..

[Elena]

The part that really confuses me are that the logs that got moved to my rotated directory... the domain logs were the only ones with 1 days worth information, the subdomain logs on the other hand had an entires month worth of information (what the domain logs should have had).

Although all the logs in the regular (not a backup) directory start from yesterday or today. So they all got reset by my script I assume. Why the difference? Are the domlogs only cut by cpanel and now the subdomain logs are the ones that will go on indefinetely? I'm stumped.

Brad, were the ones that were still intact domain or subdomain logs?

My biggest concern, if cpanel is cutting the domlogs... why isn't it backing up the previous information somewhere to be made available for download? Why isn't it also cutting the subdomain logs?

[feanor]

Ah, this was changed recently:

-rwxr-xr-x 1 root root 27634 Jun 10 13:27 cpanellogd*
(/usr/local/cpanel/)

I am going to attempt to bring this to bdraco's attention- if you take a look at it, it's gotten a bit more sophisticated ..... it's his creation though so I'm assuming he could explain it best and save me a few hours of attempting to decipher it.

Perhaps there is some mechanism in the works for the backend to help fine tune and control this, to make sure we aren't actually losing data.

[bdraco]

Logs are rotated on the last day of the month, they always have been AFAIK. My guess is that one of the latest patches fixed rotation on the servers that is was broken on.

[Elena]

If logs are rotated on the last day of the month.. where are they rotated to? I mean, once the log is cut, where does the old information go? You can't just have this information deleted into oblivion! omg...

If as far as you know, and you should now since you program this stuff, why for years have logs never been rotated until now? What &patch& that I'm unaware of could have started it to rotate?

This just completely stinks.. why cut and delete logs at the end of the month? cut them, fine, but delete them?!?! How are clients supposed to download an entire month of data if its just deleted at the end of the month when the file would hold the most data? insanity.. LOL

I usually don't get pissed off, but when business clients who use this information to analyze their marketing get screwed out of an entire months data.. this just stinks.

[feanor]

I am sure there is more to this than meets the eye------

Nick, can there possibly be a mechanism to disable this via WHManager? And/or allow us to fine tune when things get trimmed, to how many lines, and when, etc? I think a lot of people have things they've written themseves to handle this, (among other things) but now if the cpanel operation is working, and it is the 1st of each month- people are definitely going to want flexibility.

Suggestions?