The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

domlogs over 2GB, how to rotate and not break stats

Discussion in 'General Discussion' started by vissa, Mar 22, 2005.

  1. vissa

    vissa Well-Known Member

    Joined:
    May 12, 2003
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    My domlogs are very large (over 2GB) about midway through the month. Cpanel normally will zip up the logs at the end of the month and put them in the user's folders automatically and wipe the domlog. However, because these logs are so large, it stops at 2GB and just reports errors in apache's error log (which really slows down the server).

    I need to be able to rotate/compress the log every week or every day, but still not break the functionality of cPanel. I use awstats and Urchin for my clients and I'm not sure how things will interact with cPanel's own logrotate system or the stats programs looking for specific filenames. Any suggestions, ideas or help?

    Thanks!
    -V

    PS: I really wish Cpanel's own logrunner would let you specify a timeframe other than the built in monthly rotation.
     
    #1 vissa, Mar 22, 2005
    Last edited: Mar 22, 2005
  2. dalem

    dalem Well-Known Member
    PartnerNOC

    Joined:
    Oct 24, 2003
    Messages:
    2,577
    Likes Received:
    40
    Trophy Points:
    48
    Location:
    SLC
    cPanel Access Level:
    DataCenter Provider
    it does and its been there for a long time

    tweek settings
    Stats and Logs
    Delete each domain's access logs after stats run

    set the days between runs or hours
    Number of days between processing log files and bandwidth usage (default 1, decimal values are ok):
     
  3. vissa

    vissa Well-Known Member

    Joined:
    May 12, 2003
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    AS far as I understood, this will delete the raw logs. I need histories of all the logs archived in the user's account. Will this option do that? Will it affect the stats? When does it run so I can time my own install of Urchin to run at the correct moment?

    Thanks for your reply
    -V
     
    #3 vissa, Mar 22, 2005
    Last edited: Mar 22, 2005
  4. vissa

    vissa Well-Known Member

    Joined:
    May 12, 2003
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    Also, I've always had the value set to one day, but it still processes the domlogs each month... Now some users set things in their own cpanel manually and it is archiving the logs in the user folders once a month (not often enough). Here are my settings.


    Delete each domain's access logs after stats run: OFF

    Keep log files at the end of the month (default is off as you can run out of disk space quickly) : OFF

    Number of days between processing log files and bandwidth usage (default 1, decimal values are ok): 1

    Stats Log Level (default is 1, larger numbers indicate more debug information in /usr/local/cpanel/logs/stats_log) [0...10]: 1

    The load average above the number of cpus at which logs file processing should be suspended (default 0): 0
     
  5. dalem

    dalem Well-Known Member
    PartnerNOC

    Joined:
    Oct 24, 2003
    Messages:
    2,577
    Likes Received:
    40
    Trophy Points:
    48
    Location:
    SLC
    cPanel Access Level:
    DataCenter Provider
    I cant comment on urchin I do no use it (maybe some who uses it can chime in)

    if you need your raw logs you can elect to archive them in the users's cpanel
     
  6. vissa

    vissa Well-Known Member

    Joined:
    May 12, 2003
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    Yes, but that only archives them once per month.. And if they are wiped every day, how will they get archived to the user's directory?

    -V
     
  7. dalem

    dalem Well-Known Member
    PartnerNOC

    Joined:
    Oct 24, 2003
    Messages:
    2,577
    Likes Received:
    40
    Trophy Points:
    48
    Location:
    SLC
    cPanel Access Level:
    DataCenter Provider
    no it achives them after every stats run

    you have not selected to delete the log files after the stats run if you had the value on the page in cpanel would change
     
    #7 dalem, Mar 22, 2005
    Last edited: Mar 22, 2005
  8. Devil Inside

    Devil Inside Well-Known Member

    Joined:
    Apr 4, 2003
    Messages:
    277
    Likes Received:
    0
    Trophy Points:
    16
    Where do you select to archive them?

    And can you archive only specific users?
     
  9. forlinuxsupport

    forlinuxsupport Well-Known Member
    PartnerNOC

    Joined:
    Dec 22, 2004
    Messages:
    386
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    hey all

    Im just wondering..
    Can i create a simple entry for the file (in domlogs) to rotate in logrotate.d dir to rotate the dom logs once a week and then copy them to the users folder. (all in the logrorate file)
    Anyone have a script or help on how to do this properly :eek:

    My main concern is will this interfere with cpanels rotate at the end of the month.
    How and when does cpanel to the final rotation at the end of the month, or just it just nuke the file and start fresh each month ?

    Cheers
    Andy
     
  10. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Using logrotate would definitely break the stats and the bandwidth usage in cPanel.
     
  11. forlinuxsupport

    forlinuxsupport Well-Known Member
    PartnerNOC

    Joined:
    Dec 22, 2004
    Messages:
    386
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Hi Chripy...

    just the person I was hoping to hear from.

    So what is the correct workaround then ?

    Surely cpanel must cater for when logs go over the 2 gig mark.

    I just want the easiest way to have the stats generated correctly each night, and keep a copy of the logs as well (so i cant use the delete logs after stats generation option).

    Any idea how the stats and bandwidth etc is generated ?? And also when ??

    Thanks for the help :)

    Cheers
    Andy
     
  12. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    I've not seen a solution. cPanel only offers the options of clearing down the logs after a stats run and increasing the frequency of stats runs with the monthly archives. I'd suggest logging an enhancement request in bugzilla for cPanel to cater for >=2GB domlog apache files and post the link to it here for people to vote on.
     
  13. forlinuxsupport

    forlinuxsupport Well-Known Member
    PartnerNOC

    Joined:
    Dec 22, 2004
    Messages:
    386
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Logrotate + stats = pleeez vote

    Hi

    I have created a buzilla enhancement, please vote for it.

    http://bugzilla.cpanel.net/show_bug.cgi?id=3153

    this seems to be a major issue, and I have read countless threads on it.

    Please feel free to add you own suggestions for how the logrorate and stats should work.

    Cheers
    Andy
     
  14. Faldran

    Faldran Well-Known Member

    Joined:
    May 28, 2002
    Messages:
    136
    Likes Received:
    0
    Trophy Points:
    16
    I have done a logrotate on ones that are larger than 2 GB over a period of month, set to rotate them weekly, without it breaking anything ( currently do not have one setup to use an example though )

    As a prerotate, I had it run "/scripts/runweblogs username" and it worked fine.
    and as postrotate to restart apache

    Always have it run well before cpanel does thier stats update, so they do not over-lap each other.

    Not setup one in a good while, but that always worked good for me, in the past.


    What I would love to see is the ability in cpanel, to set certian domains so you can rotate the logs weekly for ones over XXXX amount, and keep X rotations. ( instead of deleting them, and including the monthly logs, cause I always hate it when a script kiddie tries to get in on the last day of the month, and not have anything to go and look at on or after the 1st )
     
  15. abubin

    abubin Well-Known Member

    Joined:
    Dec 7, 2004
    Messages:
    393
    Likes Received:
    1
    Trophy Points:
    18
    here is my simple approach to tackling this problem which i am very satisfied with.

    For the standard stats program that comes with cpanel, like awstats, webalizer and analogs, I let cpanel handle it and had setting to clear the logs after each stats run. I don't need to stats after that as all the info already went into the stats programs. This stops it from ever taking too much space.

    As for urchin, I had it setup manually under a different folder name that cpanel would not find out. For eg, urchin-own. That way, cpanel will not handle my urchin. I then add another log entry in domains that i want urchin to log in httpd.conf file. These logs will be created in another log folder and I had logrotate running that will rotate these logs weekly or daily for high hits sites.

    This effectively solve my logs problems which is cpanel upgrade proof.
     
  16. forlinuxsupport

    forlinuxsupport Well-Known Member
    PartnerNOC

    Joined:
    Dec 22, 2004
    Messages:
    386
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Hi

    Great ideas guys. pleez vote for the bug i listed in previous post.
    If we can get cpanel to put it in that would be AWESOME !!!! :D :D

    cheers
    Andy
     
  17. bobbybobbertson

    bobbybobbertson Well-Known Member

    Joined:
    May 30, 2003
    Messages:
    54
    Likes Received:
    0
    Trophy Points:
    6
    quick temporary fix

    For those that have websites that are .gif or .jpg (or any extension) heavy, I set up environment variables to log those transactions to different log files.

    This in effect reduces the amount of lines that go in the main website log, reducing the rate at which the main log grows.

    Of course ".images" file I set up below will not be seen by the stats programs. You'd have to configure the stats programs to find them, and I don't know how to do that. I haven't done it because I don't really want to track image files. You'll also have to manually delete the ".images" file, and restart apache, when the ".images" file grows too large.

    So basically... I watch my logs and when they start to get too big, I throw the following in the virtualsite directive of the httpd.conf log. Then at the end of the month, after the main log is cleared, I comment it out and put the original CustomLog directive back. -- this isn't a great fix and is a pain, but its a work around that doesn't affect the stats programs (besides that the images won't be tracked by stats).


    SetEnvIfNoCase Request_URI "\.gif$" object_is_image=gif
    SetEnvIfNoCase Request_URI "\.jpg$" object_is_image=jpg
    CustomLog domlogs/mywebsite.com.images combined env=object_is_image
    CustomLog domlogs/mywebsite.com combined env=!object_is_image

    =============
    Or if you didn't even want to worry about the ".images" file, you could do this and just dump your "images" data to /dev/null:

    SetEnvIfNoCase Request_URI "\.gif$" object_is_image=gif
    SetEnvIfNoCase Request_URI "\.jpg$" object_is_image=jpg
    CustomLog /dev/null combined env=object_is_image
    CustomLog domlogs/mywebsite.com combined env=!object_is_image
     
    #17 bobbybobbertson, Mar 29, 2006
    Last edited: Mar 29, 2006
  18. bobbybobbertson

    bobbybobbertson Well-Known Member

    Joined:
    May 30, 2003
    Messages:
    54
    Likes Received:
    0
    Trophy Points:
    6
    another thought

    I think you can have multiple CustomLog outputs for each website. So for those that want to clear logs, but keep a copy for themselves, I'd think you could put the following in your htttpd.conf virtualsite directive:

    CustomLog domlogs/mywebsite.com combined
    CustomLog domlogs/mywebsite.com.mybackup combined

    I would think that would put every transaction in both files. Cpanel will take care of the normal one, and you'd have to take care of the ".mybackup" file. So you could set cpanel to delete the regular file in WHM ("Delete each domain's access logs after stats run"), and if cpanel doesn't use the httpd.conf file (which you added the second line) to find the files to delete, your second CustomLog file should still be around.
     
  19. PWSowner

    PWSowner Well-Known Member

    Joined:
    Nov 10, 2001
    Messages:
    2,948
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    ON, Canada
    You are correct about that. I do all sorts of customizing to my logs.
     
  20. vissa

    vissa Well-Known Member

    Joined:
    May 12, 2003
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    I am shocked that 1 year later Cpanel has not fixed the problem of log rotation with sites that generate more than 2GB log files per month. Simply specifying rotation more frequently would solve this problem (once per week, every X days, etc).

    http://bugzilla.cpanel.net/show_bug.cgi?id=3153

    18 Votes in Bugzilla (The MOST of all unresolved bug reports) and 20 comments. They have this listed as an "enhancement," but it is really a bug because once logs go over 2GB, apache server will stop functioning properly. Upgrading to Apache 2 is not an acceptable answer as I've seen mentioned elsewhere.

    -vissa
     
Loading...

Share This Page