Raw Log Manager crippled in latest revs

[mikeawise]

Greetings,

I'm an end-user for a reseller hoster who uses cPanel, so forgive me if I don't use the correct lingo.


My clients' sites have been hosted with hosters using cPanel for several years and have never had prior problem with being able to get archived cumulative logs as long as I checked the box to do that within the Raw Log Manager CP.

About 2-3 months ago, I noticed that the two most recent sites I had created were only making available as an archive the last (floating) 36-48 hours worth of traffic. I created a trouble ticket for that with my former hoster, who in between waiting 3-5 days to respond, told me in a series of 1-2 sentence emails that they didn't know what the response was.

Because of their poor response time, I switched to a different reseller hoster. Right off the bat, I noticed the same 36-48 hour floating log problem with them as well. At that point, I started to suspect it may be a cPanel issue with newer releases which didn't impact sites which had already been created and configured to archive logs in previous versions of cPanel.

I made a trouble ticket with my new hoster who claims to have filed a bug report with cPanel on August 21. I today asked them what the status of my ticket was, they said they were waiting for a fix from cPanel. When I repeatedly asked them if they had even received a confirmation of the bug or even an acknowledgment for the ticket initiation, they refuse to answer the question with anything other than:

"...this is not how CPanel works. We submit the report, and we wait for the fix, and we get nothing in between there, you just have to wait."


I can't see that a paid licensee would have submitted a bug ticket four days ago and not have even received an ack, so I think they may not be on the level with me.

I download my clients' site logs and parse them through my commercial web analytics software and then charge my clients for monthly detailed reports. This issue has already cost me the entire August, and judging by my hoster's nonchalant handling of my ticket, was really hoping they would have a fix in time not to lose September stats as well.

So my question is is their any way for people who ae not cPanel licensees or have a support contract to interact with cPanel support or at least find out that they are aware of problems with the latest 11.x revs with the Raw Log Manager archiving a floating 36-48 hours worth of httpd log and that are working on a fix?


TIA,

--Mike

 

[Infopro]

You might want to get your own server. That way you can handle log rotation as you see fit.

 

[cPanelDavidG]

I would recommend if your host believes that it is a bug in the product that they submit a support ticket about this issue so our staff can login as root, analyze the server and determine what needs to be done to remedy the situation for you. This would be the path to quickest resolution.

The bugzilla system is open to the general public and does not differentiate between license holders vs. end-users. However, every new bugzilla entry is reviewed by our QA staff but there is rarely an acknowledgment of receipt (unlike our ticket system).

 

[mikeawise]

I would recommend if your host believes that it is a bug in the product that they submit a support ticket about this issue so our staff can login as root, analyze the server and determine what needs to be done to remedy the situation for you. This would be the path to quickest resolution...

They claim to have already done that a week ago (Aug. 21). When I asked them whether the bug report had even been acknowledged...much less assigned a ticket number and priority level; they repeatedly insisted that "cPanel doesn't work that way." "The licensee submits a report and then waits for a fix with no acknowledgment that the report was ever even read."


I can't see how that can be right for a commercial product. I can't be the only one who thinks having complete httpd logs for my hosted sites is important.



--Mike

 

[cPanelDavidG]

They claim to have already done that a week ago (Aug. 21). When I asked them whether the bug report had even been acknowledged...much less assigned a ticket number and priority level; they repeatedly insisted that "cPanel doesn't work that way." "The licensee submits a report and then waits for a fix with no acknowledgment that the report was ever even read."


I can't see how that can be right for a commercial product. I can't be the only one who thinks having complete httpd logs for my hosted sites is important.



--Mike

Can you clarify who you mean by "they" - they as in cPanel or they as in your web hosting provider (not cPanel)?

 

[mikeawise]

Can you clarify who you mean by "they" - they as in cPanel or they as in your web hosting provider (not cPanel)?

They (my hosting provider: hostnine.com) claims to have reported the issue to cPanel support on Aug. 21. My hoster went on to say that cPanel "doesn't work that way" when I asked my hoster if their bug report had even been acknowledged. They are claiming that one submits bug reports to cPanel support and then sits back and waits for an undefined period of time for a fix...with no acknowledgment or ticker number assigned during that waiting period.


Right now, I'm just trying to determine of whether cPanel is even aware that there is very likely a problem with the Raw Log Manager not properly archiving logs (when told to do so) and only archiving a floating 36-48 hours of traffic (which overwrites the previously archived floating 36-48 hours of traffic). It is nearly completely broken...and I experienced the exact same issue with my now former provider soon after they upgraded their cPanel installs.


--Mike

 

[cPanelDavidG]

They (my hosting provider: hostnine.com) claims to have reported the issue to cPanel support on Aug. 21. My hoster went on to say that cPanel "doesn't work that way" when I asked my hoster if their bug report had even been acknowledged. They are claiming that one submits bug reports to cPanel support and then sits back and waits for an undefined period of time for a fix...with no acknowledgment or ticker number assigned during that waiting period.


Right now, I'm just trying to determine of whether cPanel is even aware that there is very likely a problem with the Raw Log Manager not properly archiving logs (when told to do so) and only archiving a floating 36-48 hours of traffic (which overwrites the previously archived floating 36-48 hours of traffic). It is nearly completely broken...and I experienced the exact same issue with my now former provider soon after they upgraded their cPanel installs.


--Mike

Note that since publicly submitted bug reports are not tickets, they are not assigned ticket numbers. Instead, they are entered into our feature request system at http://bugzilla.cpanel.net

I have checked bugzilla (you can as well) and noticed that the 2 bugs that were recently reported through bugzilla regarding log rotation/truncation have been addressed by cPanel staff. These reports are noted as follows:

http://bugzilla.cpanel.net/show_bug.cgi?id=5657
http://bugzilla.cpanel.net/show_bug.cgi?id=5467

Note, you can add yourself as CC on any bugzilla bug report to be informed of updates regarding resolution.

In light of this post, I checked my own accounts on another hosting provider and noticed that my logs go back to the beginning of the month (I have it set to auto-delete logs at the end of every month). So this may be an issue that's isolated to a specific server configuration.

 

[mikeawise]

Note that since publicly submitted bug reports are not tickets, they are not assigned ticket numbers. Instead, they are entered into our feature request system at http://bugzilla.cpanel.net

I have checked bugzilla (you can as well) and noticed that the 2 bugs that were recently reported through bugzilla regarding log rotation/truncation have been addressed by cPanel staff. These reports are noted as follows:

http://bugzilla.cpanel.net/show_bug.cgi?id=5657
http://bugzilla.cpanel.net/show_bug.cgi?id=5467

Note, you can add yourself as CC on any bugzilla bug report to be informed of updates regarding resolution.

In light of this post, I checked my own accounts on another hosting provider and noticed that my logs go back to the beginning of the month (I have it set to auto-delete logs at the end of every month). So this may be an issue that's isolated to a specific server configuration.

I don't know that they made a public bug report (they won't say).


It's worth noting that when I first noticed this issue (with my former provider), it only seemed to be effecting sites created AFTER a recent cPanel rev had been rolled out. The archiving of httpd traffic for previously created sites was not impacted...even though those older sites were then using a newer cPanel admin interface. This would explain why I didn't notice any problems prior to setting up the newer sites and why you haven't noticed it as well.

When I moved to my new hoster about a month ago, they were already using newer builds of cPanel...and the problem exists with ALL the sites I have created with them.


I find this comment from one of the bugzilla links you provided rather odd:

-------------------------------------------------------------------------------------
------ Comment #1 From Kenneth Power 2007-07-17 12:02 [reply] -------

The current design of cPanel is to "rotate" the logs in cpanel/logs during a
cPanel update. There is discussion on changing this behavior to be more in line
with typical log rotation
-------------------------------------------------------------------------------------

This certainly not the case with older versions of cPanel, and I can't think of any reason to reengineer the product to do this. It stands to reason that if a person has checked a box to have logs archived at the end of every month, they want their full logs archived. What purpose does having only a 36-48 hour floating window of log files serve? How can people generate any meaningful web analytics reports with only a tiny snapshot of traffic available to them?


--Mike

 

[cPanelDavidG]

I don't know that they made a public bug report (they won't say).


It's worth noting that when I first noticed this issue (with my former provider), it only seemed to be effecting sites created AFTER a recent cPanel rev had been rolled out. The archiving of httpd traffic for previously created sites was not impacted...even though those older sites were then using a newer cPanel admin interface. This would explain why I didn't notice any problems prior to setting up the newer sites and why you haven't noticed it as well.

When I moved to my new hoster about a month ago, they were already using newer builds of cPanel...and the problem exists with ALL the sites I have created with them.


I find this comment from one of the bugzilla links you provided rather odd:

-------------------------------------------------------------------------------------
------ Comment #1 From Kenneth Power 2007-07-17 12:02 [reply] -------

The current design of cPanel is to "rotate" the logs in cpanel/logs during a
cPanel update. There is discussion on changing this behavior to be more in line
with typical log rotation
-------------------------------------------------------------------------------------

This certainly not the case with older versions of cPanel, and I can't think of any reason to reengineer the product to do this. It stands to reason that if a person has checked a box to have logs archived at the end of every month, they want their full logs archived. What purpose does having only a 36-48 hour floating window of log files serve? How can people generate any meaningful web analytics reports with only a tiny snapshot of traffic available to them?


--Mike

Note that cpanel/logs refers to logs stored in:

* /usr/local/cpanel/logs/

These are logs relating to cPanel/WHM itself, not Apache (website) logs.

Note that since statistics are typically generated once every 24 hours on cPanel/WHM servers, with the prior stats data cached in ~/tmp, even with this log rotation glitch on a server, web statistics should not be adversely affected.

 

[mikeawise]

Note that cpanel/logs refers to logs stored in:

* /usr/local/cpanel/logs/

These are logs relating to cPanel/WHM itself, not Apache (website) logs.

Gotcha, then that thread doesn't seem to apply to this issue.

Note that since statistics are typically generated once every 24 hours on cPanel/WHM servers, with the prior stats data cached in ~/tmp, even with this log rotation glitch on a server, web statistics should not be adversely affected.

And yet it is...with two different reseller hosters (and probably many more): icestorm.com and hostnine.com.

Interestingly enough, the AWStats component for both hosters does compute cumulative stats for the entire month...it's just the raw cumulative log files for that month (or any prior month) are not available for me to download and parse through the commercial web analytics programs I use...even though cumulative logs used to be available with older versions of cPanel. AWStats is great (especially for a free program), but it is not nearly as capable as programs like Summary, Sawmill, etc. which generate much more comprehensive reports....and in formats I can easily relay to my clients.


--Mike

 

[cPanelDavidG]

...

Interestingly enough, the AWStats component for both hosters does compute cumulative stats for the entire month...it's just the raw cumulative log files for that month (or any prior month) are not available for me to download and parse through the commercial web analytics programs I use...even though cumulative logs used to be available with older versions of cPanel. AWStats is great (especially for a free program), but it is not nearly as capable as programs like Summary, Sawmill, etc. which generate much more comprehensive reports....and in formats I can easily relay to my clients.


--Mike

It is possible that hosts have the following setting in Tweak settings checked "Delete each domain's access logs after stats run."

It would explain the "rotation effect" you are experiencing, but not necessarily why you are experienced this only on accounts created after the cPanel 11 rollout onto the servers.

 

[mikeawise]

It is possible that hosts have the following setting in Tweak settings checked "Delete each domain's access logs after stats run."

It would explain the "rotation effect" you are experiencing, but not necessarily why you are experienced this only on accounts created after the cPanel 11 rollout onto the servers.

That sounds like a plausible possibility. I will check with them very shortly on this. Perhaps that setting is on by default with newer builds...but doesn't revert sites created with previous builds. I would think that with such an option enabled that the ability to archive logs within Raw Log Manager would be grayed out.


Thanks for the tip though. Hopefully, my hoster can answer the question.


--Mike

 

[mikeawise]

After speaking with hostnine.com support and bringing up your comment about the Tweak settings access logs setting, I have determined that my hunch was correct: the newer builds of cPanel have this setting checked on by default.

Unfortunately, few of your licensees seem to be aware of this new default and probably have no reason to suspect that it is suddenly a default setting.


My hoster has unchecked the option, so the issue I'm having would seem to now be fixed. However, I have to believe that this is likely to become an issue for a lot of hosters and their customers as they start discovering their logs are disappearing when they have expressly told their cPanel admin interface to archive them?


Can you think of any reason why such an option should be enabled by default?


--Mike

 

[cPanelDavidG]

After speaking with hostnine.com support and bringing up your comment about the Tweak settings access logs setting, I have determined that my hunch was correct: the newer builds of cPanel have this setting checked on by default.

Unfortunately, few of your licensees seem to be aware of this new default and probably have no reason to suspect that it is suddenly a default setting.


My hoster has unchecked the option, so the issue I'm having would seem to now be fixed. However, I have to believe that this is likely to become an issue for a lot of hosters and their customers as they start discovering their logs are disappearing when they have expressly told their cPanel admin interface to archive them?


Can you think of any reason why such an option should be enabled by default?


--Mike

I can't speak for why it would be set as a default.

However a good reason to set this would be to reduce the amount of logs on the server as many people ignore access logs in favor of hosted stats. Enabling this will keep the stats accurate, keep log files small (dual purpose: less HD utilization with regards to logs - and since the log files are smaller, stats are generated faster and thus stats generation is less of a burden on the server). This can be useful for servers where logs take so long to process that they never complete processing by the time they are called again (usually 24 hours later).

 

[mikeawise]

I can't speak for why it would be set as a default.

However a good reason to set this would be to reduce the amount of logs on the server as many people ignore access logs in favor of hosted stats. Enabling this will keep the stats accurate, keep log files small (dual purpose: less HD utilization with regards to logs - and since the log files are smaller, stats are generated faster and thus stats generation is less of a burden on the server). This can be useful for servers where logs take so long to process that they never complete processing by the time they are called again (usually 24 hours later).

Understood, but shouldn't such decisions be left up to the end-user? After all, they are paying for the bandwidth, space, and CPU usage. No matter what, I would think such a significant change to log file retention should be prominently conveyed to cPanel licensees so that they can in turn disclose it to their customers.

Anyway, thanks for the info.

--Mike