When does cpanel rotate the user's domlogs (I noticed it's monthly, but the stats were rotated today for some reason) and where is this configured since I don't see it anywhere dealing with logrotate? We would like to setup custom rotations for some sites, but I don't want cpanel messing up a single user's rotation. For example, our custom rotation would be setup weekly and we don't want the monthly cpanel rotation messing it up one day before our custom rotation happens.
I see someone has already started this thread for me. 
I have a log rotation script installed on the server, tonight it ran because it is set to do this monthly on all the domlogs. When I went to download the logs for about 16 clients who each month ask for their logs at the start of the month... I opened up the log and what do I seeeee!!! The logs, instead of containing data from May27-June30 only have information from June 30th-July1st!
CPanel not rotating logs you say?? I think not! Cpanel has to be rotating the logs... how I imagine what has just happened. Cpanel rotated the logs, then my script rotated the logs. Thus the logs I have on backup only have 1 days worth of information.
My provider just mailed me stating that &after peeking through the /usr/local/cpanel/cpanellogd code, and It appears that the logs are getting cleared. I need to look into it a bit more, but it does appear that cpanellogd is deleting the logs after it is done processing the log for the stats.&
Just deleting them!?! Holy crap, that's worse than having the log continue to grow on indefinetely. What about last months log stats? Are those now just completely gone, never to be seen again? Clients who really rely on the rawlog and use webtrends (pretty pricey software) are not going to be happy about this. Why wouldn't cpanel think of keeping a backed log, probably even have it in the control panel to download. If its always had the huge huge log.. I dont see why splitting it up and offering for download would be any worse.. only more convenient for the clients.
I also noticed like a week ago that there is a new default ftp login in the cpanel. username_logs - however I checked that to see if maybe backups were being moved in there.. and they aren't. So, I guess someone has some explaining to do. I've already had confirmed with 3 other cpanel hosts that they either have log data only from June30-July1 or only July 1 - aiye!
Ok.. that's the end of my rant. This only gets me angry because cpanel should be updating us on what is new/added in. if this was just a fluke and the logs aren't supposed to be removed.. then I'll understand. But if this is the new way they are thinking about handling the logs.. they best be preparing to make some backups and not just zap it to hell. :p
I have a log rotation script installed on the server, tonight it ran because it is set to do this monthly on all the domlogs. When I went to download the logs for about 16 clients who each month ask for their logs at the start of the month... I opened up the log and what do I seeeee!!! The logs, instead of containing data from May27-June30 only have information from June 30th-July1st!
CPanel not rotating logs you say?? I think not! Cpanel has to be rotating the logs... how I imagine what has just happened. Cpanel rotated the logs, then my script rotated the logs. Thus the logs I have on backup only have 1 days worth of information.
My provider just mailed me stating that &after peeking through the /usr/local/cpanel/cpanellogd code, and It appears that the logs are getting cleared. I need to look into it a bit more, but it does appear that cpanellogd is deleting the logs after it is done processing the log for the stats.&
Just deleting them!?! Holy crap, that's worse than having the log continue to grow on indefinetely. What about last months log stats? Are those now just completely gone, never to be seen again? Clients who really rely on the rawlog and use webtrends (pretty pricey software) are not going to be happy about this. Why wouldn't cpanel think of keeping a backed log, probably even have it in the control panel to download. If its always had the huge huge log.. I dont see why splitting it up and offering for download would be any worse.. only more convenient for the clients.
I also noticed like a week ago that there is a new default ftp login in the cpanel. username_logs - however I checked that to see if maybe backups were being moved in there.. and they aren't. So, I guess someone has some explaining to do. I've already had confirmed with 3 other cpanel hosts that they either have log data only from June30-July1 or only July 1 - aiye!
Ok.. that's the end of my rant. This only gets me angry because cpanel should be updating us on what is new/added in. if this was just a fluke and the logs aren't supposed to be removed.. then I'll understand. But if this is the new way they are thinking about handling the logs.. they best be preparing to make some backups and not just zap it to hell. :p
Did anyone else have their logs cut (rawlog in cpanel only showing from June 30th or July 1stand nothing from before this time).. or just a few hosts had this happen?
Does anyone else even care... hellooo?
Does anyone else even care... hellooo?
OK, are you talking about stats output in individual users' control panels? That's an entirely different subject than saying the domlogs are being trimmed or completely truncated themselves.
I am confused by your post.
For some reason, we've always implemented a mechanism like this manually- CPanel does not take care of things within /usr/local/apache/domlogs .........
Has this changed recently? Did it change 8 months ago and nobody noticed? Am I the only one that didn't notice?
Anyone else have any comments on this issue? I am combing through a handful of machines, some old, some new- what's happening on everyone else's boxes?
I am confused by your post.
For some reason, we've always implemented a mechanism like this manually- CPanel does not take care of things within /usr/local/apache/domlogs .........
Has this changed recently? Did it change 8 months ago and nobody noticed? Am I the only one that didn't notice?
Anyone else have any comments on this issue? I am combing through a handful of machines, some old, some new- what's happening on everyone else's boxes?
This happened recently because last month my domlog rotation script worked perfect! Checking on if the script messed up, no it didn't. It ran once and did what it was supposed to do.
I am not talking about individual users control panel stats (not webalizer or analog) I am talking about the Apache domain logs (rawlog you download from the control panel).
---edit for more explanation
The rotation script I use moves the logs from where I store them /backup/domlogs to /backup/rotated-domlogs, it then deletes the old log from /backup/domlogs so they can start over for the new month.
Each month I go and download a few of the files in /backup/rotated-domlogs so I can give them to my clients who require the logs to use with Webtrends. Luckily I opened one last night and noticed the HUGE error before sending them. I opened all of them and they were all cut, showing information from June30-July first, ONLY 1 day worth of stats!
So if my script moves the logs from /backup/domlogs to rotated-domlogs, the logs in domlogs before my script got to them had to have already been cut by something else... I suspected cpanel and got confirmation from other cpanel hosts that dont have a rotation script that all their domlogs were cut too.
So I come here to see if anyone else had this happen and someone already had.. so I added on to this thread what I had just discovered to. Are you still confused? I'm trying my best to explain the situation and understand it myself.
I am not talking about individual users control panel stats (not webalizer or analog) I am talking about the Apache domain logs (rawlog you download from the control panel).
---edit for more explanation
The rotation script I use moves the logs from where I store them /backup/domlogs to /backup/rotated-domlogs, it then deletes the old log from /backup/domlogs so they can start over for the new month.
Each month I go and download a few of the files in /backup/rotated-domlogs so I can give them to my clients who require the logs to use with Webtrends. Luckily I opened one last night and noticed the HUGE error before sending them. I opened all of them and they were all cut, showing information from June30-July first, ONLY 1 day worth of stats!
So if my script moves the logs from /backup/domlogs to rotated-domlogs, the logs in domlogs before my script got to them had to have already been cut by something else... I suspected cpanel and got confirmation from other cpanel hosts that dont have a rotation script that all their domlogs were cut too.
So I come here to see if anyone else had this happen and someone already had.. so I added on to this thread what I had just discovered to. Are you still confused? I'm trying my best to explain the situation and understand it myself.
The part that really confuses me are that the logs that got moved to my rotated directory... the domain logs were the only ones with 1 days worth information, the subdomain logs on the other hand had an entires month worth of information (what the domain logs should have had).
Although all the logs in the regular (not a backup) directory start from yesterday or today. So they all got reset by my script I assume. Why the difference? Are the domlogs only cut by cpanel and now the subdomain logs are the ones that will go on indefinetely? I'm stumped.
Brad, were the ones that were still intact domain or subdomain logs?
My biggest concern, if cpanel is cutting the domlogs... why isn't it backing up the previous information somewhere to be made available for download? Why isn't it also cutting the subdomain logs?
Although all the logs in the regular (not a backup) directory start from yesterday or today. So they all got reset by my script I assume. Why the difference? Are the domlogs only cut by cpanel and now the subdomain logs are the ones that will go on indefinetely? I'm stumped.
Brad, were the ones that were still intact domain or subdomain logs?
My biggest concern, if cpanel is cutting the domlogs... why isn't it backing up the previous information somewhere to be made available for download? Why isn't it also cutting the subdomain logs?
Ah, this was changed recently:
-rwxr-xr-x 1 root root 27634 Jun 10 13:27 cpanellogd*
(/usr/local/cpanel/)
I am going to attempt to bring this to bdraco's attention- if you take a look at it, it's gotten a bit more sophisticated ..... it's his creation though so I'm assuming he could explain it best and save me a few hours of attempting to decipher it.
Perhaps there is some mechanism in the works for the backend to help fine tune and control this, to make sure we aren't actually losing data.
-rwxr-xr-x 1 root root 27634 Jun 10 13:27 cpanellogd*
(/usr/local/cpanel/)
I am going to attempt to bring this to bdraco's attention- if you take a look at it, it's gotten a bit more sophisticated ..... it's his creation though so I'm assuming he could explain it best and save me a few hours of attempting to decipher it.
Perhaps there is some mechanism in the works for the backend to help fine tune and control this, to make sure we aren't actually losing data.
B
bdraco
Guest
Logs are rotated on the last day of the month, they always have been AFAIK. My guess is that one of the latest patches fixed rotation on the servers that is was broken on.
If logs are rotated on the last day of the month.. where are they rotated to? I mean, once the log is cut, where does the old information go? You can't just have this information deleted into oblivion! omg...
If as far as you know, and you should now since you program this stuff, why for years have logs never been rotated until now? What &patch& that I'm unaware of could have started it to rotate?
This just completely stinks.. why cut and delete logs at the end of the month? cut them, fine, but delete them?!?! How are clients supposed to download an entire month of data if its just deleted at the end of the month when the file would hold the most data? insanity.. LOL
I usually don't get pissed off, but when business clients who use this information to analyze their marketing get screwed out of an entire months data.. this just stinks.
If as far as you know, and you should now since you program this stuff, why for years have logs never been rotated until now? What &patch& that I'm unaware of could have started it to rotate?
This just completely stinks.. why cut and delete logs at the end of the month? cut them, fine, but delete them?!?! How are clients supposed to download an entire month of data if its just deleted at the end of the month when the file would hold the most data? insanity.. LOL
I usually don't get pissed off, but when business clients who use this information to analyze their marketing get screwed out of an entire months data.. this just stinks.
I am sure there is more to this than meets the eye------
Nick, can there possibly be a mechanism to disable this via WHManager? And/or allow us to fine tune when things get trimmed, to how many lines, and when, etc? I think a lot of people have things they've written themseves to handle this, (among other things) but now if the cpanel operation is working, and it is the 1st of each month- people are definitely going to want flexibility.
Suggestions?
Nick, can there possibly be a mechanism to disable this via WHManager? And/or allow us to fine tune when things get trimmed, to how many lines, and when, etc? I think a lot of people have things they've written themseves to handle this, (among other things) but now if the cpanel operation is working, and it is the 1st of each month- people are definitely going to want flexibility.
Suggestions?
I would also like to see this to be able to be config'd via whm, if its going to be put in place.
We send out logs to some of our clients on the first of the month, so this very much affects us.
As i said it would be nice to be able to config this via whm. Possibly even to exclude certain domains from the logrotation or be able to say when you want them to be deleted. Every month, two months, or three.
Just a thought..
C.
We send out logs to some of our clients on the first of the month, so this very much affects us.
As i said it would be nice to be able to config this via whm. Possibly even to exclude certain domains from the logrotation or be able to say when you want them to be deleted. Every month, two months, or three.
Just a thought..
C.
Yes, I agree there should be some sort of function within WHM that turns on/off log rotation or something similiar to avoid conflicts with custom rotation scripts. Or, tell us what to cut out of cpanellogd, if that's where this all happens.
P.S. Our logs were all rotated on the 27th of June, which is not the last day of the month.
P.S. Our logs were all rotated on the 27th of June, which is not the last day of the month.
I just found out from my provider that cpanel will &rotate a domlog on the last day of the month [b:01e56871bb]after it has crunched the stats for that domlog[/b:01e56871bb]. Therefore account A's domlog might be rotated at one time during the day while account B's domlog might be rotated several hours later, or several minutes later, depending on how busy the server is which directly determines how busy the cpanellogd daemon is (remember that the stats don't update if the server load is over 1.0 on a single cpu server).&
So, what do I find upsetting about this? All the logs are rotated at a different time.. so setting something up to rotate the logs before cpanel gets to them will be near impossible. My provider didn't tell me if cpanel will just be deleting the logs in the rotation process, but if it is! errr.. what is there that we can do?
So, what do I find upsetting about this? All the logs are rotated at a different time.. so setting something up to rotate the logs before cpanel gets to them will be near impossible. My provider didn't tell me if cpanel will just be deleting the logs in the rotation process, but if it is! errr.. what is there that we can do?
I can see
I can see how Nick does not find this a major issue (in the beginning) since it seems he assumes this is the method all along as it is intended to function. I think I would very much take the same position in that situation. Feel free to correct me if I am wrong. I have not seen cPanel work in this capacity previous as I can see others have not in a long time as well.
That said stats are significant for about 1% of our 3500 clients. They are small but vocal group. The method of calculating stats does not make working around this easy. If fact I'd say darn near impossible!
So, who is going to write up the enhancement request? I see a few options: Logrotate: Monthly, Weekly, Bi-Weekly; Backup Increment: 0 (none as it is)-N.
I can see how Nick does not find this a major issue (in the beginning) since it seems he assumes this is the method all along as it is intended to function. I think I would very much take the same position in that situation. Feel free to correct me if I am wrong. I have not seen cPanel work in this capacity previous as I can see others have not in a long time as well.
That said stats are significant for about 1% of our 3500 clients. They are small but vocal group. The method of calculating stats does not make working around this easy. If fact I'd say darn near impossible!
So, who is going to write up the enhancement request? I see a few options: Logrotate: Monthly, Weekly, Bi-Weekly; Backup Increment: 0 (none as it is)-N.
| Thread starter | Similar threads | Forum | Replies | Date |
|---|---|---|---|---|
|
Log rotation of User logs (/home/username/logs) | Server Management | 1 | |
| T | Enable log rotation? | Server Management | 4 | |
|
Removing cpanel log rotation through CLI or API | Server Management | 5 | |
| Z | Launch application before cPanel log rotation? | Server Management | 0 | |
| D | Log Rotation? | Server Management | 4 |