How does cpanellogd work?

[DuncanMcLord]

Hi,

just a quick question, since i couldn't find any info about this, but how does cpanellogd work?
I mean, does it work at runtime, so if a user wants to access his stats it parses those?
Or does it wait for low-load times and then parse one log after the other?

I am asking, because i would like to know if i could define the time when awstats get updated!

Thanks,
Duncan

[rich2]

*bump* this needs an answer, i'd like to know as well.

[golloween]

Does anybody know when exactly cpanellogd runs AWStats and Webalizer to process logs?

[pingo]

Does cpanellogd run AWStats and Webalizer? - I wondered why process cpanellogd user root used more than 24 of the cpu resources for long period. While this is happening the server load is very high. So it is the stats and probably because there are more accounts now on the server. - Does anyone know if something can be done about this - I would also like to know if the time stats are processed could be changed?

John

[pingo]

I wonder why cpanel.net are not providing a faq for simple questions being asked again and again but not being answered here. - or does someone else care?

John

[MarlboroMan]

cpanellogd handles log processing and license updating (the main loop runs stats every other time it loops, and updates the license file every 25 loops)

Every time through the main loop, it checks if the following files are over 5MB and rotates them:
/usr/local/apache/logs/suexec_log,
/usr/local/apache/access_log,
/usr/local/apache/logs/error_log,
/etc/httpd/logs/error_log

If the logs are over 5MB and rotated - it restarts Apache with a SIGUSR1

During the logrun loops, it first does something with FTP & transfer logs (haven't looked too deeply into this), then goes through /var/cpanel/users and runs logs and updates bandwidth usage for every user, catching subdomains and the like as each user is ran.

After each loop through main, the daemon sleeps for 5 minutes.


That's pretty much it for the daemon part of cpanellogd.


It can also be called on a per-user basis - simply run /usr/local/cpanel/cpanellogd <username> to have it just run the logs for a specific user and exit.


::edit:: Slight correction
In case you were wondering, cpanellogd doesn't actually get to run stats. it just collects information needed to run stats on all domains, and passes it to /usr/local/cpanel/bin/logrunner - which is an executable file, not a perl file, so I can't tell you what goes on in there. I'd imagine it just checks if stats have been run on the domains it is passed in the last 24 hours, and if not, spawns a child to run stats for that domain.

All the load-checking and etc. have moved out of cpanellogd - I have to assume that it's in logrunner now.

::/edit::

[golloween]

MarlboroMan, thanks for the detailed answer! What's the difference between the "main loop" and the "logrun loop"?

[pingo]

Yes, thanks alot for that explanation

John

[MarlboroMan]

cpanellogd has an internal counter - it increases by 1 each time the main loop runs. Each time through the main loop, if the counter is even, it runs logs. If it's odd, it doesn't run logs. If its divisible by 25, it updates your license file.

[golloween]

Ok, it sleeps for 5 minutes after each iteration of the main loop and runs logs on every 2nd pass. If this is correct, the time between running logs should be 10 minutes.

But how these 10 minutes are related to the setting in WHM under the Tweak Settings menu: "Number of days between processing log files etc..."? Or this setting is handled by the Logrunner that simply skips loops if the time passed since the last logrun is lesser than the value of that setting?

[MarlboroMan]

I would assume so - logrunner gets passed information about running logs for all domains every 10 minutes - but only runs logs after system time is 24 hours past that of the lastrun file in the stats directory.

[Curious Too]

Originally posted by MarlboroMan
cpanellogd handles log processing and license updating (the main loop runs stats every other time it loops, and updates the license file every 25 loops)

Every time through the main loop, it checks if the following files are over 5MB and rotates them:
/usr/local/apache/logs/suexec_log,
/usr/local/apache/access_log,
/usr/local/apache/logs/error_log,
/etc/httpd/logs/error_log

If the logs are over 5MB and rotated - it restarts Apache with a SIGUSR1

Not on my machines. I have to manually delete the access_log, error_log, suexec_log and ssl_engine_log. I checked one machine tonight and saw:

%cpu %mem Time command
28.9 1.9 48:38 cpanellogd - updating logs

When it "finished" I checked my logs. The error_log was nearly 15M, the suexec_log was 29M.

[MarlboroMan]

Sorry - I read it wrong - if the filesize is over 500MB, it trims is to 5MB

[Curious Too]

Originally posted by MarlboroMan
Sorry - I read it wrong - if the filesize is over 500MB, it trims is to 5MB

What are the effects on log processing if you kill cpanellogd and restart it later?

[MarlboroMan]

Logs don't get processed, bandwidth totals aren't updated, etc.

I don't remember if CPanel runs a user's webalizer, etc stats if they request them through their CPanel and they haven't been run in the last 24 hours. It used to be that way, but I woulnd't put money on it anymore.