Page 1 of 3 123 LastLast
Results 1 to 15 of 42

Thread: cPanel 11.25 Log Processing

  1. #1
    cPanel Development cPanelKenneth's Avatar
    Join Date
    Apr 2006
    Posts
    4,279
    cPanel/WHM Access Level

    Root Administrator

    Default cPanel 11.25 Log Processing

    Please Note: This thread is for discussion of the logging changes detailed in this post. Posts that are aggressive, or consist of only complaints will be removed

    The following changes to log processing are forthcoming in cPanel 11.25.0.

    1. How much data is retained when rotating log files

    The cPanel log processor (cpanellogd), if so configured, will rotate a log file when it reaches a certain threshold. For the service log files in /usr/local/apache/logs and /usr/local/cpanel/logs the threshold is 300 MB. Due to a logic error, the log processor would retain the threshold amount of data in the log file.

    Example

    On the server red.example.com the /usr/local/apache/logs/access_log is configured to be rotated. This log file has 587 MB of data. When next cpanellogd evaluates the log for rotation it flags the Apache access_log for rotation. During the actual rotation 287 MB of data is moved to the archive ( /usr/local/apache/logs/archives ) while 300 MB of data is retained.

    The default thresholds are:
    • For service logs: 300 MB
    • For domain logs ( /usr/local/apache/domlogs ): 1024 MB


    The logic error erroneously ensured that the log file would need rotated the next time it is evaluated for rotation. On some systems this can be as often as 15 minutes from the last evaluation. The logic error is compounded by the necessity of a restart of the service whose log was just rotated. Again on some systems this can mean an Apache restart every 15 minutes.

    To address this, the logic for data retention for log rotation is adjusted to the following:
    • For service logs the entirety of the log file is archived, the file thereby being reduced to zero-bytes
    • For domain logs 500 MB of data is retained.[1]


    This change is already in cPanel 11.25.0 builds 42552 and higher.


    2. How Apache logs data

    The current Apache configuration has Apache directly opening, and writing to, the log files for all the virtual hosts as well as itself. This requires Apache to be restarted whenever a file it opened needs rotated or deleted ( see the File Descriptors side bar in http://docs.cpanel.net/twiki/pub/All...cPanellogd.pdf for more information ). Although graceful restarts are requested of Apache, as recent history shows certain Apache configurations do not handle the graceful restart gracefully[2].

    To account for the Apache behavior, and to retain the data integrity of the domain and bytes logs, a new method for Apache to log information is being introduced. Rather than logging directly to files, Apache will log directly to a pipe. Both virtual host data, as well as the service level access data, will be written to the pipe. This feature is only available to servers using Apache 2.0 or 2.2.

    Once Apache is configured to log to a pipe it is no longer necessary to restart Apache when rotating or deleting log files except when the Apache error_log needs to be rotated.


    3. FAQ

    3.1 How robust is this new log method?

    In testing, Apache 2.x handled 90,000 requests per second across 3000 domains, on average. All data was placed in the appropriate log files.

    Since Apache owns the pipe, if the reader, the program listening on the pipe, dies, Apache will restart it. In tests this event was indiscernible.


    3.2 What is handled by this new logging method?

    All logging that Apache does in the typical cPanel configuration is handled by the pipe method, except for the Apache error_log. This includes the Apache access_log, bytes log data and all other logs that were directly handled by Apache.


    3.3 So this means no more Apache restarts every 15 minutes, right?

    As mentioned above the Apache error_log is not handled via the pipe method, therefore if that log needs rotated an Apache restart will be required. To lengthen the time between Apache restarts for that log file one can do the following:

    • Fix the problems that cause the Apache error_log to grow quickly. Often a broken PHP application will cause the Apache error log to swiftly grow in size. Determing the problem and resolving it will stem the growth of that file and thereby lengthen the time needed for the error_log to exceed the threshold.
    • Increase the log rotation threshold. This is done in Tweak Settings by increasing the value of Threshold above which cpanellogd will rotate files.


    3.4 Does this change the location of the domain logs or their contents?

    No. All logs remain in their current location. The contents of the log files also remains the same.


    3.5 What about logs such as the suexec log?

    This change only impacts log files that are directly handled by Apache. Certain other log files, such as the suexec log or the mod_suphp log, are not directly handled by Apache and therefore are not affected by this change.


    3.6 When will this be available?

    This is now available in cPanel 11.25.0.

    3.7 How do I enable this feature?

    You must have root access to enable this feature.

    1. Login to WHM
    2. Click Service Configuration
    3. Click Apache Configuration
    4. Click Piped Log Configuration
    5. Check the box labeled Enable piped Apache logs
    6. Click the Save button


    Note: a minor UI glitch made it into the builds. Once the piped logging is enabled and saved, the resulting UI will show the piped logging feature as unchecked. This will be fixed in the next build.

    3.8 Will I need to rebuild httpd.conf when enabling this feature?

    No. Upon enabling this feature /usr/local/apache/conf/httpd.conf will be regenerated and Apache will be restarted. For the curious you will note that the logging directives will be removed from all the virtual hosts. The only logging directives will be in the global section of httpd.conf

    3.9 I use custom vhost templates, or what about my modifications to httpd.conf?

    For the most part the standard rules with httpd.conf modification applies.

    If you are using custom virtual host templates they will still be used. This means if they have logging directives they will still be honored. If you are not customizing the logging directives in your custom vhost template then we recommend you update your template to remove the logging directives if you want to take advantage of this feature. Of course, do that just before you enable the feature.

    4. In Conclusion


    We invite further comments and questions regarding these changes.

    If you wish to partake of further testing of the new logging mechanism, please submit a support request with the server details. Mark the ticket: "ATTN QA - Apache Pipe Logging". Please note that we are specifically looking for real-world environments in which to measure the logging changes. This means use in a production environment.


    The details regarding this will be added to the cpanellogd documentation. This documentation is at http://docs.cpanel.net/twiki/pub/All...cPanellogd.pdf .

    Please Note: This thread is for discussion of the logging changes detailed in this post. Posts that are aggressive, or consist of only complaints will be removed
    ---


    [1] This logic is to account for other aspects of the product that rely upon data existing in the domain log file. This will be readdressed at a future date to make rotation of both types of log files consistent.

    [2] From reports via the ticket system and here it appears there is something intrisically wrong with Apache internally. Those running 11.25.0 who experience Apache children left in a 'G' state for an indefinite length of time after a graceful restart, please open a support ticket. Please mark the ticket: ATTN QA - Apache Child G State.
    Last edited by cPanelKenneth; 02-03-2010 at 05:28 PM. Reason: httpd.conf customization questions; I cannot count
    Kenneth
    Development
    cPanel, Inc.

  2. #2
    Registered Member cPanel Partner NOC Badge
    Join Date
    Nov 2001
    Posts
    215
    cPanel/WHM Access Level

    DataCenter Provider

    Default

    To account for the Apache behavior, and to retain the data integrity of the domain and bytes logs, a new method for Apache to log information is being introduced. Rather than logging directly to files, Apache will log directly to a pipe. Both virtual host data, as well as the service level access data, will be written to the pipe. This feature is only available to servers using Apache 2.0 or 2.2.
    I assume from the above statement that httpd.conf will need to be rebuilt with new domain logging statements?

  3. #3
    cPanel Development cPanelKenneth's Avatar
    Join Date
    Apr 2006
    Posts
    4,279
    cPanel/WHM Access Level

    Root Administrator

    Default

    Quote Originally Posted by ffeingol View Post
    I assume from the above statement that httpd.conf will need to be rebuilt with new domain logging statements?
    Thank you for your question. I added that question, and a response, to the question section of the original post.
    Kenneth
    Development
    cPanel, Inc.

  4. #4
    Registered Member
    Join Date
    Mar 2006
    Location
    Turkey
    Posts
    151

    Default

    After the new system, will awstats and other statistics programs run smoothly?

    For example our access log for one domain is growing and becomes bigger than 1024 mb for 3 times a day and rotating.
    But we are only processing awstats once a day. So when awstats process starts, it only looks for the domain log file and it is not the right file (right but incomplete log file).

    What is the process for this?

  5. #5
    cPanel Development cPanelKenneth's Avatar
    Join Date
    Apr 2006
    Posts
    4,279
    cPanel/WHM Access Level

    Root Administrator

    Default

    Quote Originally Posted by trhosting.net View Post
    After the new system, will awstats and other statistics programs run smoothly?
    The log parsing programs ( e..g AWstats ) will work as well as they always have.

    Quote Originally Posted by trhosting.net View Post
    For example our access log for one domain is growing and becomes bigger than 1024 mb for 3 times a day and rotating.
    But we are only processing awstats once a day. So when awstats process starts, it only looks for the domain log file and it is not the right file (right but incomplete log file).

    What is the process for this?
    From your description you need to change your log processing configuration so the log is parsed before it is rotated.
    Kenneth
    Development
    cPanel, Inc.

  6. #6
    Registered Member
    Join Date
    Jan 2005
    Posts
    1,880

    Default

    Quote Originally Posted by cpanelkenneth View Post
    Rather than logging directly to files, Apache will log directly to a pipe. Both virtual host data, as well as the service level access data, will be written to the pipe. This feature is only available to servers using Apache 2.0 or 2.2.

    Once Apache is configured to log to a pipe it is no longer necessary to restart Apache when rotating or deleting log files except when the Apache error_log needs to be rotated.
    These log processing improvements seem to apply to Apache 2.x only.

    What should we be expecting if using Apache 1.3.x? Will the problem of Apache restarting every 15 minutes still be an issue?
    Jon Cram <jon@webignition.net>
    Web and interface design http://webignition.net/
    Hosting Reborn (free cpanel accounts) http://hostingreborn.com/

  7. #7
    Registered Member
    Join Date
    Aug 2009
    Posts
    15

    Default

    Where does Apache pipe the logs to?

  8. #8
    Registered Member cPanel Partner NOC Badge
    Join Date
    Nov 2001
    Posts
    215
    cPanel/WHM Access Level

    DataCenter Provider

    Default

    Is the new piped logging method currently available on any build? We'd like to begin our own in-house testing.

    Thanks

  9. #9
    cPanel Quality Assurance Analyst cPanelDon's Avatar
    Join Date
    Nov 2008
    Location
    Houston, Texas, U.S.A.
    Posts
    2,554
    cPanel/WHM Access Level

    DataCenter Provider

    Default

    Quote Originally Posted by webignition View Post
    These log processing improvements seem to apply to Apache 2.x only.

    What should we be expecting if using Apache 1.3.x? Will the problem of Apache restarting every 15 minutes still be an issue?
    The corrective change regarding cpanellogd is detailed in section 1 of the first post ("How much data is retained when rotating log files"); to reiterate and clarify, "This change is already in cPanel 11.25.0 builds 42552 and higher." To determine the latest available builds and version information please reference the following page: cPanel Downloads - Latest cPanel/WHM Builds

    For additional clarification please also reference FAQ section 3.3 of the first post ("So this means no more Apache restarts every 15 minutes, right?").
    Last edited by cPanelDon; 01-28-2010 at 06:38 PM. Reason: Clarification

  10. #10
    Registered Member cPanel Partner NOC Badge
    Join Date
    Nov 2001
    Posts
    215
    cPanel/WHM Access Level

    DataCenter Provider

    Default

    @cPanelDon,

    We just switched our testing VPS to run edge but it still appears to be running the file based logging. We went from Release to Edge. Do we need to run upcp with --force or manually rebuild httpd.conf? We are running Apache 2.2.14.

  11. #11
    cPanel Quality Assurance Analyst cPanelDon's Avatar
    Join Date
    Nov 2008
    Location
    Houston, Texas, U.S.A.
    Posts
    2,554
    cPanel/WHM Access Level

    DataCenter Provider

    Default

    Quote Originally Posted by ckoehler View Post
    Where does Apache pipe the logs to?
    When the piped logging method is enabled Apache will pipe logs to a common target process, a program started by Apache that is listening on the pipe; the receiving process will then automatically split the logs into their expected per-domain version. Please note that this does not change the location of nor the content of any Apache log files.

  12. #12
    cPanel Quality Assurance Analyst cPanelDon's Avatar
    Join Date
    Nov 2008
    Location
    Houston, Texas, U.S.A.
    Posts
    2,554
    cPanel/WHM Access Level

    DataCenter Provider

    Default

    Quote Originally Posted by ffeingol View Post
    Is the new piped logging method currently available on any build? We'd like to begin our own in-house testing.

    Thanks
    Quote Originally Posted by ffeingol View Post
    @cPanelDon,

    We just switched our testing VPS to run edge but it still appears to be running the file based logging. We went from Release to Edge. Do we need to run upcp with --force or manually rebuild httpd.conf? We are running Apache 2.2.14.
    The feature is in EDGE but is not automatically enabled; to participate in testing please refer to the following instruction (as per cPanelKenneth's post):
    If you wish to partake of further testing of the new logging mechanism, please submit a support request with the server details. Mark the ticket: "ATTN QA - Apache Pipe Logging". Please note that we are specifically looking for real-world environments in which to measure the logging changes. This means use in a production environment.

  13. #13
    Registered Member cPanel Partner NOC Badge
    Join Date
    Nov 2001
    Posts
    215
    cPanel/WHM Access Level

    DataCenter Provider

    Default

    @cPanelDon,

    Sorry, I totally missed the part about having to open a ticket.

    We run Tomcat on some of our cPanel servers. Will this new logging method work with Tomcat?

  14. #14
    cPanel Quality Assurance Analyst cPanelDon's Avatar
    Join Date
    Nov 2008
    Location
    Houston, Texas, U.S.A.
    Posts
    2,554
    cPanel/WHM Access Level

    DataCenter Provider

    Default

    Quote Originally Posted by ffeingol View Post
    We run Tomcat on some of our cPanel servers. Will this new logging method work with Tomcat?
    I'm not aware of related issues with Tomcat; to the best of my knowledge it should not affect the ability to write logs that are handled by Apache.

  15. #15
    cPanel Development cPanelKenneth's Avatar
    Join Date
    Apr 2006
    Posts
    4,279
    cPanel/WHM Access Level

    Root Administrator

    Default

    Quote Originally Posted by webignition View Post
    These log processing improvements seem to apply to Apache 2.x only.

    What should we be expecting if using Apache 1.3.x? Will the problem of Apache restarting every 15 minutes still be an issue?
    At this time we are only focusing on Apache 2.0 and 2.2. At a later time we may re-evaluate extending support to Apache 1.3.
    Kenneth
    Development
    cPanel, Inc.

Page 1 of 3 123 LastLast

Similar Threads

  1. Best way to implement additional log processing?
    By criznach in forum cPanel & WHM Discussions
    Replies: 2
    Last Post: 05-20-2014, 01:00 AM
  2. Re: cPanel 11.25 Log Processing Thread
    By classical in forum cPanel & WHM Discussions
    Replies: 6
    Last Post: 03-28-2011, 11:02 AM
  3. 100% uptime with cPanel while log rotation and processing
    By desiguru in forum cPanel & WHM Discussions
    Replies: 6
    Last Post: 04-07-2010, 05:24 PM
  4. Log processing...
    By JustinK in forum cPanel & WHM Discussions
    Replies: 0
    Last Post: 11-28-2002, 04:35 PM
bargain