Please Note: This thread is for discussion of the logging changes detailed in this post. Posts that are aggressive, or consist of only complaints will be removed
The following changes to log processing are forthcoming in cPanel 11.25.0.
1. How much data is retained when rotating log files
The cPanel log processor (cpanellogd), if so configured, will rotate a log file when it reaches a certain threshold. For the service log files in /usr/local/apache/logs and /usr/local/cpanel/logs the threshold is 300 MB. Due to a logic error, the log processor would retain the threshold amount of data in the log file.
Example
On the server red.example.com the /usr/local/apache/logs/access_log is configured to be rotated. This log file has 587 MB of data. When next cpanellogd evaluates the log for rotation it flags the Apache access_log for rotation. During the actual rotation 287 MB of data is moved to the archive ( /usr/local/apache/logs/archives ) while 300 MB of data is retained.
The default thresholds are:
The logic error erroneously ensured that the log file would need rotated the next time it is evaluated for rotation. On some systems this can be as often as 15 minutes from the last evaluation. The logic error is compounded by the necessity of a restart of the service whose log was just rotated. Again on some systems this can mean an Apache restart every 15 minutes.
To address this, the logic for data retention for log rotation is adjusted to the following:
This change is already in cPanel 11.25.0 builds 42552 and higher.
2. How Apache logs data
The current Apache configuration has Apache directly opening, and writing to, the log files for all the virtual hosts as well as itself. This requires Apache to be restarted whenever a file it opened needs rotated or deleted ( see the File Descriptors side bar in http://docs.cpanel.net/twiki/pub/AllDocumentation/ReleaseNotes/cPanellogd.pdf for more information ). Although graceful restarts are requested of Apache, as recent history shows certain Apache configurations do not handle the graceful restart gracefully[2].
To account for the Apache behavior, and to retain the data integrity of the domain and bytes logs, a new method for Apache to log information is being introduced. Rather than logging directly to files, Apache will log directly to a pipe. Both virtual host data, as well as the service level access data, will be written to the pipe. This feature is only available to servers using Apache 2.0 or 2.2.
Once Apache is configured to log to a pipe it is no longer necessary to restart Apache when rotating or deleting log files except when the Apache error_log needs to be rotated.
3. FAQ
3.1 How robust is this new log method?
In testing, Apache 2.x handled 90,000 requests per second across 3000 domains, on average. All data was placed in the appropriate log files.
Since Apache owns the pipe, if the reader, the program listening on the pipe, dies, Apache will restart it. In tests this event was indiscernible.
3.2 What is handled by this new logging method?
All logging that Apache does in the typical cPanel configuration is handled by the pipe method, except for the Apache error_log. This includes the Apache access_log, bytes log data and all other logs that were directly handled by Apache.
3.3 So this means no more Apache restarts every 15 minutes, right?
As mentioned above the Apache error_log is not handled via the pipe method, therefore if that log needs rotated an Apache restart will be required. To lengthen the time between Apache restarts for that log file one can do the following:
3.4 Does this change the location of the domain logs or their contents?
No. All logs remain in their current location. The contents of the log files also remains the same.
3.5 What about logs such as the suexec log?
This change only impacts log files that are directly handled by Apache. Certain other log files, such as the suexec log or the mod_suphp log, are not directly handled by Apache and therefore are not affected by this change.
3.6 When will this be available?
This is now available in cPanel 11.25.0.
3.7 How do I enable this feature?
You must have root access to enable this feature.
Note: a minor UI glitch made it into the builds. Once the piped logging is enabled and saved, the resulting UI will show the piped logging feature as unchecked. This will be fixed in the next build.
3.8 Will I need to rebuild httpd.conf when enabling this feature?
No. Upon enabling this feature /usr/local/apache/conf/httpd.conf will be regenerated and Apache will be restarted. For the curious you will note that the logging directives will be removed from all the virtual hosts. The only logging directives will be in the global section of httpd.conf
3.9 I use custom vhost templates, or what about my modifications to httpd.conf?
For the most part the standard rules with httpd.conf modification applies.
If you are using custom virtual host templates they will still be used. This means if they have logging directives they will still be honored. If you are not customizing the logging directives in your custom vhost template then we recommend you update your template to remove the logging directives if you want to take advantage of this feature. Of course, do that just before you enable the feature.
4. In Conclusion
We invite further comments and questions regarding these changes.
If you wish to partake of further testing of the new logging mechanism, please submit a support request with the server details. Mark the ticket: "ATTN QA - Apache Pipe Logging". Please note that we are specifically looking for real-world environments in which to measure the logging changes. This means use in a production environment.
The details regarding this will be added to the cpanellogd documentation. This documentation is at http://docs.cpanel.net/twiki/pub/AllDocumentation/ReleaseNotes/cPanellogd.pdf .
Please Note: This thread is for discussion of the logging changes detailed in this post. Posts that are aggressive, or consist of only complaints will be removed
---
[1] This logic is to account for other aspects of the product that rely upon data existing in the domain log file. This will be readdressed at a future date to make rotation of both types of log files consistent.
[2] From reports via the ticket system and here it appears there is something intrisically wrong with Apache internally. Those running 11.25.0 who experience Apache children left in a 'G' state for an indefinite length of time after a graceful restart, please open a support ticket. Please mark the ticket: ATTN QA - Apache Child G State.
The following changes to log processing are forthcoming in cPanel 11.25.0.
1. How much data is retained when rotating log files
The cPanel log processor (cpanellogd), if so configured, will rotate a log file when it reaches a certain threshold. For the service log files in /usr/local/apache/logs and /usr/local/cpanel/logs the threshold is 300 MB. Due to a logic error, the log processor would retain the threshold amount of data in the log file.
Example
On the server red.example.com the /usr/local/apache/logs/access_log is configured to be rotated. This log file has 587 MB of data. When next cpanellogd evaluates the log for rotation it flags the Apache access_log for rotation. During the actual rotation 287 MB of data is moved to the archive ( /usr/local/apache/logs/archives ) while 300 MB of data is retained.
The default thresholds are:
- For service logs: 300 MB
- For domain logs ( /usr/local/apache/domlogs ): 1024 MB
The logic error erroneously ensured that the log file would need rotated the next time it is evaluated for rotation. On some systems this can be as often as 15 minutes from the last evaluation. The logic error is compounded by the necessity of a restart of the service whose log was just rotated. Again on some systems this can mean an Apache restart every 15 minutes.
To address this, the logic for data retention for log rotation is adjusted to the following:
- For service logs the entirety of the log file is archived, the file thereby being reduced to zero-bytes
- For domain logs 500 MB of data is retained.[1]
This change is already in cPanel 11.25.0 builds 42552 and higher.
2. How Apache logs data
The current Apache configuration has Apache directly opening, and writing to, the log files for all the virtual hosts as well as itself. This requires Apache to be restarted whenever a file it opened needs rotated or deleted ( see the File Descriptors side bar in http://docs.cpanel.net/twiki/pub/AllDocumentation/ReleaseNotes/cPanellogd.pdf for more information ). Although graceful restarts are requested of Apache, as recent history shows certain Apache configurations do not handle the graceful restart gracefully[2].
To account for the Apache behavior, and to retain the data integrity of the domain and bytes logs, a new method for Apache to log information is being introduced. Rather than logging directly to files, Apache will log directly to a pipe. Both virtual host data, as well as the service level access data, will be written to the pipe. This feature is only available to servers using Apache 2.0 or 2.2.
Once Apache is configured to log to a pipe it is no longer necessary to restart Apache when rotating or deleting log files except when the Apache error_log needs to be rotated.
3. FAQ
3.1 How robust is this new log method?
In testing, Apache 2.x handled 90,000 requests per second across 3000 domains, on average. All data was placed in the appropriate log files.
Since Apache owns the pipe, if the reader, the program listening on the pipe, dies, Apache will restart it. In tests this event was indiscernible.
3.2 What is handled by this new logging method?
All logging that Apache does in the typical cPanel configuration is handled by the pipe method, except for the Apache error_log. This includes the Apache access_log, bytes log data and all other logs that were directly handled by Apache.
3.3 So this means no more Apache restarts every 15 minutes, right?
As mentioned above the Apache error_log is not handled via the pipe method, therefore if that log needs rotated an Apache restart will be required. To lengthen the time between Apache restarts for that log file one can do the following:
- Fix the problems that cause the Apache error_log to grow quickly. Often a broken PHP application will cause the Apache error log to swiftly grow in size. Determing the problem and resolving it will stem the growth of that file and thereby lengthen the time needed for the error_log to exceed the threshold.
- Increase the log rotation threshold. This is done in Tweak Settings by increasing the value of Threshold above which cpanellogd will rotate files.
3.4 Does this change the location of the domain logs or their contents?
No. All logs remain in their current location. The contents of the log files also remains the same.
3.5 What about logs such as the suexec log?
This change only impacts log files that are directly handled by Apache. Certain other log files, such as the suexec log or the mod_suphp log, are not directly handled by Apache and therefore are not affected by this change.
3.6 When will this be available?
This is now available in cPanel 11.25.0.
3.7 How do I enable this feature?
You must have root access to enable this feature.
- Login to WHM
- Click Service Configuration
- Click Apache Configuration
- Click Piped Log Configuration
- Check the box labeled Enable piped Apache logs
- Click the Save button
Note: a minor UI glitch made it into the builds. Once the piped logging is enabled and saved, the resulting UI will show the piped logging feature as unchecked. This will be fixed in the next build.
3.8 Will I need to rebuild httpd.conf when enabling this feature?
No. Upon enabling this feature /usr/local/apache/conf/httpd.conf will be regenerated and Apache will be restarted. For the curious you will note that the logging directives will be removed from all the virtual hosts. The only logging directives will be in the global section of httpd.conf
3.9 I use custom vhost templates, or what about my modifications to httpd.conf?
For the most part the standard rules with httpd.conf modification applies.
If you are using custom virtual host templates they will still be used. This means if they have logging directives they will still be honored. If you are not customizing the logging directives in your custom vhost template then we recommend you update your template to remove the logging directives if you want to take advantage of this feature. Of course, do that just before you enable the feature.
4. In Conclusion
We invite further comments and questions regarding these changes.
If you wish to partake of further testing of the new logging mechanism, please submit a support request with the server details. Mark the ticket: "ATTN QA - Apache Pipe Logging". Please note that we are specifically looking for real-world environments in which to measure the logging changes. This means use in a production environment.
The details regarding this will be added to the cpanellogd documentation. This documentation is at http://docs.cpanel.net/twiki/pub/AllDocumentation/ReleaseNotes/cPanellogd.pdf .
Please Note: This thread is for discussion of the logging changes detailed in this post. Posts that are aggressive, or consist of only complaints will be removed
---
[1] This logic is to account for other aspects of the product that rely upon data existing in the domain log file. This will be readdressed at a future date to make rotation of both types of log files consistent.
[2] From reports via the ticket system and here it appears there is something intrisically wrong with Apache internally. Those running 11.25.0 who experience Apache children left in a 'G' state for an indefinite length of time after a graceful restart, please open a support ticket. Please mark the ticket: ATTN QA - Apache Child G State.
Last edited:
