1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

cPanel 11.25 Log Processing

Discussion in 'General Discussion' started by cPanelKenneth, Jan 25, 2010.

  1. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,319
    Likes Received:
    8
    Trophy Points:
    38
    Please Note: This thread is for discussion of the logging changes detailed in this post. Posts that are aggressive, or consist of only complaints will be removed

    The following changes to log processing are forthcoming in cPanel 11.25.0.

    1. How much data is retained when rotating log files

    The cPanel log processor (cpanellogd), if so configured, will rotate a log file when it reaches a certain threshold. For the service log files in /usr/local/apache/logs and /usr/local/cpanel/logs the threshold is 300 MB. Due to a logic error, the log processor would retain the threshold amount of data in the log file.

    Example

    On the server red.example.com the /usr/local/apache/logs/access_log is configured to be rotated. This log file has 587 MB of data. When next cpanellogd evaluates the log for rotation it flags the Apache access_log for rotation. During the actual rotation 287 MB of data is moved to the archive ( /usr/local/apache/logs/archives ) while 300 MB of data is retained.

    The default thresholds are:
    • For service logs: 300 MB
    • For domain logs ( /usr/local/apache/domlogs ): 1024 MB

    The logic error erroneously ensured that the log file would need rotated the next time it is evaluated for rotation. On some systems this can be as often as 15 minutes from the last evaluation. The logic error is compounded by the necessity of a restart of the service whose log was just rotated. Again on some systems this can mean an Apache restart every 15 minutes.

    To address this, the logic for data retention for log rotation is adjusted to the following:
    • For service logs the entirety of the log file is archived, the file thereby being reduced to zero-bytes
    • For domain logs 500 MB of data is retained.[1]

    This change is already in cPanel 11.25.0 builds 42552 and higher.


    2. How Apache logs data

    The current Apache configuration has Apache directly opening, and writing to, the log files for all the virtual hosts as well as itself. This requires Apache to be restarted whenever a file it opened needs rotated or deleted ( see the File Descriptors side bar in http://docs.cpanel.net/twiki/pub/AllDocumentation/ReleaseNotes/cPanellogd.pdf for more information ). Although graceful restarts are requested of Apache, as recent history shows certain Apache configurations do not handle the graceful restart gracefully[2].

    To account for the Apache behavior, and to retain the data integrity of the domain and bytes logs, a new method for Apache to log information is being introduced. Rather than logging directly to files, Apache will log directly to a pipe. Both virtual host data, as well as the service level access data, will be written to the pipe. This feature is only available to servers using Apache 2.0 or 2.2.

    Once Apache is configured to log to a pipe it is no longer necessary to restart Apache when rotating or deleting log files except when the Apache error_log needs to be rotated.


    3. FAQ

    3.1 How robust is this new log method?

    In testing, Apache 2.x handled 90,000 requests per second across 3000 domains, on average. All data was placed in the appropriate log files.

    Since Apache owns the pipe, if the reader, the program listening on the pipe, dies, Apache will restart it. In tests this event was indiscernible.


    3.2 What is handled by this new logging method?

    All logging that Apache does in the typical cPanel configuration is handled by the pipe method, except for the Apache error_log. This includes the Apache access_log, bytes log data and all other logs that were directly handled by Apache.


    3.3 So this means no more Apache restarts every 15 minutes, right?

    As mentioned above the Apache error_log is not handled via the pipe method, therefore if that log needs rotated an Apache restart will be required. To lengthen the time between Apache restarts for that log file one can do the following:

    • Fix the problems that cause the Apache error_log to grow quickly. Often a broken PHP application will cause the Apache error log to swiftly grow in size. Determing the problem and resolving it will stem the growth of that file and thereby lengthen the time needed for the error_log to exceed the threshold.
    • Increase the log rotation threshold. This is done in Tweak Settings by increasing the value of Threshold above which cpanellogd will rotate files.

    3.4 Does this change the location of the domain logs or their contents?

    No. All logs remain in their current location. The contents of the log files also remains the same.


    3.5 What about logs such as the suexec log?

    This change only impacts log files that are directly handled by Apache. Certain other log files, such as the suexec log or the mod_suphp log, are not directly handled by Apache and therefore are not affected by this change.


    3.6 When will this be available?

    This is now available in cPanel 11.25.0.

    3.7 How do I enable this feature?

    You must have root access to enable this feature.

    1. Login to WHM
    2. Click Service Configuration
    3. Click Apache Configuration
    4. Click Piped Log Configuration
    5. Check the box labeled Enable piped Apache logs
    6. Click the Save button

    Note: a minor UI glitch made it into the builds. Once the piped logging is enabled and saved, the resulting UI will show the piped logging feature as unchecked. This will be fixed in the next build.

    3.8 Will I need to rebuild httpd.conf when enabling this feature?

    No. Upon enabling this feature /usr/local/apache/conf/httpd.conf will be regenerated and Apache will be restarted. For the curious you will note that the logging directives will be removed from all the virtual hosts. The only logging directives will be in the global section of httpd.conf

    3.9 I use custom vhost templates, or what about my modifications to httpd.conf?

    For the most part the standard rules with httpd.conf modification applies.

    If you are using custom virtual host templates they will still be used. This means if they have logging directives they will still be honored. If you are not customizing the logging directives in your custom vhost template then we recommend you update your template to remove the logging directives if you want to take advantage of this feature. Of course, do that just before you enable the feature.

    4. In Conclusion


    We invite further comments and questions regarding these changes.

    If you wish to partake of further testing of the new logging mechanism, please submit a support request with the server details. Mark the ticket: "ATTN QA - Apache Pipe Logging". Please note that we are specifically looking for real-world environments in which to measure the logging changes. This means use in a production environment.


    The details regarding this will be added to the cpanellogd documentation. This documentation is at http://docs.cpanel.net/twiki/pub/AllDocumentation/ReleaseNotes/cPanellogd.pdf .

    Please Note: This thread is for discussion of the logging changes detailed in this post. Posts that are aggressive, or consist of only complaints will be removed
    ---


    [1] This logic is to account for other aspects of the product that rely upon data existing in the domain log file. This will be readdressed at a future date to make rotation of both types of log files consistent.

    [2] From reports via the ticket system and here it appears there is something intrisically wrong with Apache internally. Those running 11.25.0 who experience Apache children left in a 'G' state for an indefinite length of time after a graceful restart, please open a support ticket. Please mark the ticket: ATTN QA - Apache Child G State.
     
    #1 cPanelKenneth, Jan 25, 2010
    Last edited: Feb 3, 2010
  2. ffeingol

    ffeingol Member

    Joined:
    Nov 9, 2001
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    I assume from the above statement that httpd.conf will need to be rebuilt with new domain logging statements?
     
  3. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,319
    Likes Received:
    8
    Trophy Points:
    38
    Thank you for your question. I added that question, and a response, to the question section of the original post.
     
  4. trhosting.net

    trhosting.net Member

    Joined:
    Mar 7, 2006
    Messages:
    151
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Turkey
    After the new system, will awstats and other statistics programs run smoothly?

    For example our access log for one domain is growing and becomes bigger than 1024 mb for 3 times a day and rotating.
    But we are only processing awstats once a day. So when awstats process starts, it only looks for the domain log file and it is not the right file (right but incomplete log file).

    What is the process for this?
     
  5. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,319
    Likes Received:
    8
    Trophy Points:
    38
    The log parsing programs ( e..g AWstats ) will work as well as they always have.

    From your description you need to change your log processing configuration so the log is parsed before it is rotated.
     
  6. webignition

    webignition Active Member

    Joined:
    Jan 22, 2005
    Messages:
    1,880
    Likes Received:
    0
    Trophy Points:
    36
    These log processing improvements seem to apply to Apache 2.x only.

    What should we be expecting if using Apache 1.3.x? Will the problem of Apache restarting every 15 minutes still be an issue?
     
  7. ckoehler

    ckoehler New Member

    Joined:
    Aug 7, 2009
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    Where does Apache pipe the logs to?
     
  8. ffeingol

    ffeingol Member

    Joined:
    Nov 9, 2001
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    Is the new piped logging method currently available on any build? We'd like to begin our own in-house testing.

    Thanks
     
  9. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,558
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    The corrective change regarding cpanellogd is detailed in section 1 of the first post ("How much data is retained when rotating log files"); to reiterate and clarify, "This change is already in cPanel 11.25.0 builds 42552 and higher." To determine the latest available builds and version information please reference the following page: cPanel Downloads - Latest cPanel/WHM Builds

    For additional clarification please also reference FAQ section 3.3 of the first post ("So this means no more Apache restarts every 15 minutes, right?").
     
    #9 cPanelDon, Jan 28, 2010
    Last edited: Jan 28, 2010
  10. ffeingol

    ffeingol Member

    Joined:
    Nov 9, 2001
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    @cPanelDon,

    We just switched our testing VPS to run edge but it still appears to be running the file based logging. We went from Release to Edge. Do we need to run upcp with --force or manually rebuild httpd.conf? We are running Apache 2.2.14.
     
  11. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,558
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    When the piped logging method is enabled Apache will pipe logs to a common target process, a program started by Apache that is listening on the pipe; the receiving process will then automatically split the logs into their expected per-domain version. Please note that this does not change the location of nor the content of any Apache log files.
     
  12. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,558
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    The feature is in EDGE but is not automatically enabled; to participate in testing please refer to the following instruction (as per cPanelKenneth's post):
    If you wish to partake of further testing of the new logging mechanism, please submit a support request with the server details. Mark the ticket: "ATTN QA - Apache Pipe Logging". Please note that we are specifically looking for real-world environments in which to measure the logging changes. This means use in a production environment.
     
  13. ffeingol

    ffeingol Member

    Joined:
    Nov 9, 2001
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    @cPanelDon,

    Sorry, I totally missed the part about having to open a ticket.

    We run Tomcat on some of our cPanel servers. Will this new logging method work with Tomcat?
     
  14. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,558
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    I'm not aware of related issues with Tomcat; to the best of my knowledge it should not affect the ability to write logs that are handled by Apache.
     
  15. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,319
    Likes Received:
    8
    Trophy Points:
    38
    At this time we are only focusing on Apache 2.0 and 2.2. At a later time we may re-evaluate extending support to Apache 1.3.
     
  16. ffeingol

    ffeingol Member

    Joined:
    Nov 9, 2001
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    Is there an ETA when the new logging will be available in Edge? We'd like to test in our lab.
     
  17. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    12,134
    Likes Received:
    37
    Trophy Points:
    48
    Location:
    Pennsylvania



    I'm in. :)
     
  18. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,558
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    The last available ETA was for inclusion this week as mentioned in cPannelKenneth's initial post. To help further clarify, please note the feature is available in the latest EDGE builds but it is not automatically enabled; however, it may be manually enabled when participating in formal Quality Assurance testing via a support request as outlined earlier.
     
  19. motivez

    motivez New Member

    Joined:
    Nov 9, 2006
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    I am having this issue, I think, but as a result of the graceful restart, my apache is segfaulting.. thus requiring a manual restart.

    Any suggestions here?
     
  20. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,319
    Likes Received:
    8
    Trophy Points:
    38
    Please open a support request for the issue you are reporting.

    Thank you.
     

Share This Page