The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

log domlogs too big

Discussion in 'General Discussion' started by webstyler, Mar 1, 2010.

  1. webstyler

    webstyler Well-Known Member

    Joined:
    Nov 20, 2003
    Messages:
    432
    Likes Received:
    0
    Trophy Points:
    16
  2. madaboutlinux

    madaboutlinux Well-Known Member

    Joined:
    Jan 24, 2005
    Messages:
    1,052
    Likes Received:
    2
    Trophy Points:
    38
    Location:
    Earth
    Once the stats are processed, the domlog files are emptied. Are the stats for your domains are updating properly? If not, the domlog file for the domains will keep on growing and will continue to cause problems since stats won't be upgraded.
     
  3. webstyler

    webstyler Well-Known Member

    Joined:
    Nov 20, 2003
    Messages:
    432
    Likes Received:
    0
    Trophy Points:
    16
    Hello

    Stats are normally processed

    Under Tweak setting :

    "Delete each domain's access logs after stats run" isn't checked
    "Keep log files at the end of the month" IS checked

    Because on this server we need to archive for long time.

    ..
     
  4. webstyler

    webstyler Well-Known Member

    Joined:
    Nov 20, 2003
    Messages:
    432
    Likes Received:
    0
    Trophy Points:
    16
    I can set logrotate with this new httpd rules ..

    but my big log have the domain name (as mydomain.com) so *log not work properly..

    I prefer to not set /* because there are too many files.. may be useful set ALL with except .log

    How to fix this ?

    ----

    /usr/local/apache/domlogs/*log {
    missingok
    rotate 12
    notifempty
    weekly
    compress
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    }


    /usr/local/apache/logs/*log {
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    }
     
  5. webstyler

    webstyler Well-Known Member

    Joined:
    Nov 20, 2003
    Messages:
    432
    Likes Received:
    0
    Trophy Points:
    16
    think this may be right solution


    /usr/local/apache/domlogs/* {
    missingok
    rotate 12
    notifempty
    size=10000k
    compress
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    }


    /usr/local/apache/logs/*log {
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    }
     
  6. webstyler

    webstyler Well-Known Member

    Joined:
    Nov 20, 2003
    Messages:
    432
    Likes Received:
    0
    Trophy Points:
    16
    I would like also compress and rotate all logs on domain dir inside domlogs

    so think this is the most complete version..

    any suggest be appreciated

    --

    /usr/local/apache/domlogs/*/* {
    missingok
    rotate 12
    notifempty
    size=10000k
    compress
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    }

    /usr/local/apache/domlogs/* {
    missingok
    rotate 12
    notifempty
    size=10000k
    compress
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    }

    /usr/local/apache/logs/*log {
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
    }
     
  7. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Archive raw domain access logs via cPanel

    If needing to minimize disk space usage I recommend allowing the domain access logs to be deleted after stats run, and to not have the domain access logs kept at the end of the month.

    To archive logs for a longer period of time, I recommend enabling the archive option via the user's cPanel at the following menu path:
    cPanel: Main >> Logs >> Raw Access Logs >> Configure Logs

    For clarification, the above area has the following two options available:
    "Archive Logs in your home directory at the end of each stats run"
    "Remove the previous month's archived logs from your home directory at the end of each month"

    A custom script could be created (by the server administrator) and used if needing to apply one or both setting in bulk to multiple users or all users. The specific file to create and or modify is located within the user's home directory path as seen in the following example (where "username" is the same as the cPanel account username):
    Code:
    /home/username/.cpanel-logs
    When setup, the contents of the above file may have just one option defined or both, usually only when the option is enabled; if an option is disabled it is not required to have it defined as they are not set (enabled) by default.
    Code:
    archive-logs=1
    remove-old-archived-logs=1
    When the archive logs option is enabled via cPanel, as detailed above, the logs will be saved into a "logs" sub-directory within the user's home directory path, as seen in the following example:
    Code:
    /home/username/logs/
    The logs archived in the above path are also compressed to minimize disk usage; there will be a separate gzip-compressed file for each month.
     
    #7 cPanelDon, Mar 10, 2010
    Last edited: Mar 10, 2010
  8. kman

    kman Member

    Joined:
    Aug 18, 2003
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    Hi Don,

    I've read your answer above, which is very informative, plus http://forums.cpanel.net/f5/cpanel-11-25-log-processing-145417.html, as well as the PDF document.

    Our WHM is 11.25. I've some questions regarding log management of some of our high traffic sites though. These sites generate domlogs as fast as 500MB in a week, and hence it could get past 1GB hard limit (I read of this somewhere in the doc) of cpanellogd in another week or so. I'm not sure if this 1GB hard limit works or not because I've seen 1G+ domlogs before.

    One of our requirements is that we need to keep 120 days worth of each domain access_log. I'm quite certain that there's no way to achieve this through WHM, but we probably need to use /home/username/.cpanel-logs as you described.

    Steps:

    1. setting /home/username/.cpanel-logs >> archive-logs=1
    2. Keep log file at end of month = No ('Coz already obtained in 1 & to save space. This is what I'm not sure of.)
    3. Delete each domain's access logs after stats run = Whatever (Should have no effect due to 1.)

    Then we'll have our daily cron job to copy user logs over from their logs dir to our central log archive.

    Questions:
    1. Would these settings work?
    2. Could the customer change our setting in /home/username/.cpanel-logs via their cPanel, if .cpanel-logs is owned by root?
    3. The log archive in user logs dir will be 1 file/month (beginning - ending of each month regardless of settings in WHM)? I think I see some archive files that only begin on April 29 and end April 30. Yes, user's been here over a month.
    4. Why did I see 1G+ domlogs when it's stated that 1GB is a hard limit of domlogs size for cpanellogd? I've seen these before switching to using the pipelog.

    Thanks.
     
    #8 kman, May 6, 2010
    Last edited: May 6, 2010
  9. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    I believe the provided settings should work in your described scenario, allowing log data to be retained beyond 30 days by having archived copies stored within each user's home directory.

    Different than a file-system-imposed hard file size limitation, there are thresholds for the sizes of certain log files that may trigger an action such as log rotation depending on the server configuration; the cpanellogd information is detailed verbosely in the Release Notes document for cPanel Log Processing and Configuration (PDF). The Detail I describe further below is contributed in large part directly by this PDF document and specifically in section 3.2 under Rotation & Retention between pages 5 and 6 and in section 4.2 under Domain Log Rotation on page 9.

    I believe the 1GB reference may be that of a hard-coded threshold for rotation per the aforementioned release notes documentation, wherein it further clarifies the default behavior that I believe indicates why individual domain logs exceeded 1GB in size: Domain access Apache logs do not rotate by default. However, the logic and configuration that governs the retention of this data is separate of cPanel and Apache log files. The most obvious benefit of this approach is domain users can access and manage these raw logs. The threshold for rotation and file trimming on domain logs is set at 1GB. [...] Domain logs are hard coded to rotate at 1GB, as well as at the end of each month.

    If it is desired to enable rotation of domain access Apache logs (i.e., "domain logs") then it is necessary to enable the option "Keep monthly log files" (i.e., "Keep log files at the end of the month") that is available in WHM via the following menu path: WHM: Main >> Server Configuration >> Tweak Settings >> Stats and Logs
     
    #9 cPanelDon, May 12, 2010
    Last edited: May 12, 2010
Loading...

Share This Page