How does cpanellogd work?

[DuncanMcLord]

Hi,

just a quick question, since i couldn't find any info about this, but how does cpanellogd work?
I mean, does it work at runtime, so if a user wants to access his stats it parses those?
Or does it wait for low-load times and then parse one log after the other?

I am asking, because i would like to know if i could define the time when awstats get updated!

Thanks,
Duncan

 

[rich2]

*bump* this needs an answer, i'd like to know as well.

 

[golloween]

Does anybody know when exactly cpanellogd runs AWStats and Webalizer to process logs?

 

[pingo]

Does cpanellogd run AWStats and Webalizer? - I wondered why process cpanellogd user root used more than 24 of the cpu resources for long period. While this is happening the server load is very high. So it is the stats and probably because there are more accounts now on the server. - Does anyone know if something can be done about this - I would also like to know if the time stats are processed could be changed?

John

 

[pingo]

I wonder why cpanel.net are not providing a faq for simple questions being asked again and again but not being answered here. - or does someone else care?

John

 

[MarlboroMan]

cpanellogd handles log processing and license updating (the main loop runs stats every other time it loops, and updates the license file every 25 loops)

Every time through the main loop, it checks if the following files are over 5MB and rotates them:
/usr/local/apache/logs/suexec_log,
/usr/local/apache/access_log,
/usr/local/apache/logs/error_log,
/etc/httpd/logs/error_log

If the logs are over 5MB and rotated - it restarts Apache with a SIGUSR1

During the logrun loops, it first does something with FTP & transfer logs (haven't looked too deeply into this), then goes through /var/cpanel/users and runs logs and updates bandwidth usage for every user, catching subdomains and the like as each user is ran.

After each loop through main, the daemon sleeps for 5 minutes.


That's pretty much it for the daemon part of cpanellogd.


It can also be called on a per-user basis - simply run /usr/local/cpanel/cpanellogd <username> to have it just run the logs for a specific user and exit.


::edit:: Slight correction
In case you were wondering, cpanellogd doesn't actually get to run stats. it just collects information needed to run stats on all domains, and passes it to /usr/local/cpanel/bin/logrunner - which is an executable file, not a perl file, so I can't tell you what goes on in there. I'd imagine it just checks if stats have been run on the domains it is passed in the last 24 hours, and if not, spawns a child to run stats for that domain.

All the load-checking and etc. have moved out of cpanellogd - I have to assume that it's in logrunner now.

::/edit::

 

[golloween]

MarlboroMan, thanks for the detailed answer! What's the difference between the "main loop" and the "logrun loop"?

 

[pingo]

Yes, thanks alot for that explanation

John

 

[MarlboroMan]

cpanellogd has an internal counter - it increases by 1 each time the main loop runs. Each time through the main loop, if the counter is even, it runs logs. If it's odd, it doesn't run logs. If its divisible by 25, it updates your license file.

 

[golloween]

Ok, it sleeps for 5 minutes after each iteration of the main loop and runs logs on every 2nd pass. If this is correct, the time between running logs should be 10 minutes.

But how these 10 minutes are related to the setting in WHM under the Tweak Settings menu: "Number of days between processing log files etc..."? Or this setting is handled by the Logrunner that simply skips loops if the time passed since the last logrun is lesser than the value of that setting?

 

[MarlboroMan]

I would assume so - logrunner gets passed information about running logs for all domains every 10 minutes - but only runs logs after system time is 24 hours past that of the lastrun file in the stats directory.

 

[Curious Too]

Originally posted by MarlboroMan
cpanellogd handles log processing and license updating (the main loop runs stats every other time it loops, and updates the license file every 25 loops)

Every time through the main loop, it checks if the following files are over 5MB and rotates them:
/usr/local/apache/logs/suexec_log,
/usr/local/apache/access_log,
/usr/local/apache/logs/error_log,
/etc/httpd/logs/error_log

If the logs are over 5MB and rotated - it restarts Apache with a SIGUSR1

Not on my machines. I have to manually delete the access_log, error_log, suexec_log and ssl_engine_log. I checked one machine tonight and saw:

%cpu %mem Time command
28.9 1.9 48:38 cpanellogd - updating logs

When it "finished" I checked my logs. The error_log was nearly 15M, the suexec_log was 29M.

 

[MarlboroMan]

Sorry - I read it wrong - if the filesize is over 500MB, it trims is to 5MB

 

[Curious Too]

Originally posted by MarlboroMan
Sorry - I read it wrong - if the filesize is over 500MB, it trims is to 5MB

What are the effects on log processing if you kill cpanellogd and restart it later?

 

[MarlboroMan]

Logs don't get processed, bandwidth totals aren't updated, etc.

I don't remember if CPanel runs a user's webalizer, etc stats if they request them through their CPanel and they haven't been run in the last 24 hours. It used to be that way, but I woulnd't put money on it anymore.

 

[Curious Too]

I've killed and restarted cpanellogd and the stats, logs etc do get updated. I haven't checked to see if there is any missing data but the stats do get updated.

cpanellogd is a resource hog, not to mention the stats_log is growing so large so quickly now that it has to be deleted regualarly, and the only way to get the correct disk useage after deleting the stats_log is to restart cpanellogd.

 

[H2Hosting.com]

I think that something is wrong with cpanellogd after last updates as it takes too much resources of the server. We have dual P3-1130Mhz and 1000 1.5GbRAM boxes and cpanellogd use almost 20% of CPU!

6580 root 20 19 117M 114M 114M R N 18.6 11.3 490:04 cpanellogd - updating logs

26580 root 20 19 117M 114M 102M R N 25.3 11.3 490:49 cpanellogd - updating logs

26580 root 20 19 117M 114M 25836 R N 22.3 11.3 490:51 cpanellogd - updating logs

Cpanel programmers must fix it!!!

 

[FWC]

I've been seeing this since Cpanel updated the night of March 10th, too. It looks like it may be taking a long time to run rrdtool across all the domains.

 

[Brad]

Ours has been having the same problem too.. Hight resource useage.

 

[Sash]

Ditto.....

Mike