RRDtool bandwidth graph gaps

[retali8]

I meant to also include a screenshot of the bandwidth graphs that we're seeing for this specific user.

[retali8]

We are seeing a very strange issue on several of our servers where a users bandwidth graph will have many gaps and have been unable to figure out what exactly is causing the issue. I've checked the file in /var/cpanel/bandwidth/{username}.hour and see the missing data for specific times of the day as you can see below..

2010.09.08T08-all=124029557
2010.09.08T08-http=124029557
2010.09.08T13-all=2367
2010.09.08T13-pop3=2367
2010.09.09T03-all=19130782
2010.09.09T03-http=19130782
2010.09.09T04-all=84447074
2010.09.09T04-http=84447074
2010.09.09T05-all=102852479
2010.09.09T05-http=102852479

Does anyone know why exactly this would be occurring? I've also checked /etc/stats.conf and no blackout hours are configured. I've enabled raw log archiving and they do have HTTP traffic around these times but for some reason it's not being recorded. I've also checked the /usr/local/cpanel/logs/stats_log and I'm not seeing any errors for this user but we are seeing it for other users on the same server and other servers as well.

[cPanel:Wade]

There are a couple of different mechanisms that can cause data not to appear in the bandwidth graphs. One of them is a long-standing issue and the other is more recent.

RRDtool Artifacts
The first is a little hard to understand. RRDtool takes an odd approach to storing data which results in unusual results. The data is actually stored as a rate of change of data, not as actual data points. This has caused odd graph artifacts in the past.

The artifact that sometimes caused "missing bandwidth" has to do with the rate-storage and the fact that the data is stored in "steps". For example, in the data for today, bandwidth is collected in 5 minute steps. So, in effect, all data in a particular 5 minute span is stored as a single number. Additionally, since RRDtool only stores a rate, you need two steps in order to generate a rate.

In the the traditional usage of RRDtool, if data comes in widely separated bursts (each of which fits in a single step) a rate can never be calculated because there are no two points close together. A rate requires two data points (to calculate a change), so a single data point does not get reported. Moreover, if bandwidth is being detected most of the time, this effect would not show up.

This issue was actually fixed several months ago by added "synthetic" data points with a value of 0 in the places where we did not get data. Although it seems obvious, this is not how RRDtool is normally used.

0-Length Remainder Files
The more recent issue has to do with a change to a module that cPanel depends on for several features: YAML::Syck. The YAML::Syck was changed a few months ago to throw an exception when a 0-length file is read with YAML::Syck. Before the change, the YAML::Syck returned empty data.

The Bandwidth processing regular reads and writes a YAML file that keeps up with data that we cannot yet write to RRDtool (because we are in the middle of one of the time steps). If the bandwidth processing doesn't have any left-over data, a 0-length remainder file is written. Before the YAML::Syck change, that was perfectly legitimate.

This issue has been fixed in the upcoming version of cPanel. Unfortunately, if you are using the 11.26 branch and YAML::Syck gets updated, there is a problem.

Conclusion
A new version of 11.26 should be released within a few days that has the updated code that solves this problem. If you cannot wait and definitely have the problem, I have attached a patch that should repair this problem (on 11.26.9).

This can be applied by executing
patch -p0 < 43327.patch
in the directory /usr/local/cpanel. (Use the real path to 43327.patch.)

Hope that helps.