tailwatchd perpetually failing

[airynd]

Hello,

We're seeing a fair number of customers reporting tailwatchd failing constantly. This began about 24 hours ago, and I'm wondering if a recent update may have broken something. This is in the /usr/local/cpanel/logs/tailwatchd_log:

[START] 14636 1229052314
[STOP Ok] 14636 1229052817
[Thu Dec 11 22:33:37 2008] Current process '14636' stopped
[START] 15009 1229052817
[STOP Ok] 15009 1229053321
[Thu Dec 11 22:42:01 2008] Current process '15009' stopped
[START] 15386 1229053321
[STOP Ok] 15386 1229053824
[Thu Dec 11 22:50:24 2008] Current process '15386' stopped
[START] 15773 1229053825
[root@server.xxxxxxx.net] logs >> tail -f tailwatchd_log
[START] 14636 1229052314
[STOP Ok] 14636 1229052817
[Thu Dec 11 22:33:37 2008] Current process '14636' stopped
[START] 15009 1229052817
[STOP Ok] 15009 1229053321
[Thu Dec 11 22:42:01 2008] Current process '15009' stopped
[START] 15386 1229053321
[STOP Ok] 15386 1229053824
[Thu Dec 11 22:50:24 2008] Current process '15386' stopped
[START] 15773 1229053825

Naturally, an email is generated each time, and our customers are becoming... vexed.

Restarting cPanel will fix this for awhile, but an hour or two or five later, the same shenanigans occur.

Has anyone else seen this lately?

[Infopro]

What version of cPanel are you on? There were some updates recently to CURRENT.

Add crontab entry to automatically ensure that tailwatchd is always running
Added cron job to monitor Tailwatchd

http://changelog.cpanel.net/

[cpanelkenneth]

Hello,

We're seeing a fair number of customers reporting tailwatchd failing constantly. This began about 24 hours ago, and I'm wondering if a recent update may have broken something. This is in the /usr/local/cpanel/logs/tailwatchd_log:

[START] 14636 1229052314
[STOP Ok] 14636 1229052817
[Thu Dec 11 22:33:37 2008] Current process '14636' stopped
[START] 15009 1229052817
[STOP Ok] 15009 1229053321
[Thu Dec 11 22:42:01 2008] Current process '15009' stopped
[START] 15386 1229053321
[STOP Ok] 15386 1229053824
[Thu Dec 11 22:50:24 2008] Current process '15386' stopped
[START] 15773 1229053825
[root@server.dragonflihost.net] logs >> tail -f tailwatchd_log
[START] 14636 1229052314
[STOP Ok] 14636 1229052817
[Thu Dec 11 22:33:37 2008] Current process '14636' stopped
[START] 15009 1229052817
[STOP Ok] 15009 1229053321
[Thu Dec 11 22:42:01 2008] Current process '15009' stopped
[START] 15386 1229053321
[STOP Ok] 15386 1229053824
[Thu Dec 11 22:50:24 2008] Current process '15386' stopped
[START] 15773 1229053825

Naturally, an email is generated each time, and our customers are becoming... vexed.

Restarting cPanel will fix this for awhile, but an hour or two or five later, the same shenanigans occur.

Has anyone else seen this lately?

If I'm reading those time stamps properly, it looks like the restart is happening every 8 minutes, which roughly corresponds to when chkservd runs. What does the chkservd log contain?

[airynd]

Thanks for the responses. The customer is on cPanel version:

WHM 11.23.2 cPanel 11.23.6-R27698
CENTOS Enterprise 5.2 i686 on standard - WHM X v3.1.0

Here are a few entries from the chkservd.log file:

[Fri Dec 12 17:59:16 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => via EMAIL [level => 1]
Restarting tailwatchd....
[Fri Dec 12 18:07:40 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => [email] via EMAIL [level => 1]
Restarting tailwatchd....
[Fri Dec 12 18:16:04 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => [email]via EMAIL [level => 1]
Restarting tailwatchd....
[Fri Dec 12 18:24:27 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => [email]] via EMAIL [level => 1]
Restarting tailwatchd....

So, nothing terribly useful there... at least not to me. :-)

Thanks for any additional help!

[Messiah]

One of the functions performed by tailwatchd is eximstats, which has been recently deprecated. Tailwatchd attempts and is unable to restart eximstats when tailwatchd starts, so tailwatchd dies. Try logging into WHM, selecting Service Manager, then un-checking eximstats from the tailwatchd category. Click save, then you should stop getting these emails.

I imagine this will be fixed in an upcoming release.

[cpdan]

Interesting to note that its the old style chksrvd (IE the dameon) and not the new (IE the driver)

Service Check Started
[2008-12-09 21:32:09 UTC] Service check ....cpsrvd [[check command:N/A][tcp connect:+]]...cpdavd [[check command:+][tcp connect:N/A]]...Done
Service Check Finished

[cpdan]

Tailwatchd attempts and is unable to restart eximstats when tailwatchd starts, so tailwatchd dies.

It could be the eximstats driver dying oddly but it doesn't try to restart eximstats. The things the eximstats daemon did are now handled in tailwatchd daemon by the eximstats driver.

Hope that clears things up a little In the mean time I'd love to see a system that is exhibitiung this issue so we can address it ASAP. If anyone has one please open a ticket with the subject 'ATTN: Dan' and refenece this forum thread.

Thanks!

[cpdan]

I killed mysql and restarted and all was well (eximstats logs the queries it would have been doing to a file so that you can run them later once mysql is around)

Is there an error on the screen or in the logs about eximstats driver?

[cpdan]

WHM 11.23.2 cPanel 11.23.6-R27698
CENTOS Enterprise 5.2 i686 on standard - WHM X v3.1.0

Here are a few entries from the chkservd.log file:

...

[Fri Dec 12 18:16:04 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => [email]via EMAIL [level => 1]
Restarting tailwatchd....
[Fri Dec 12 18:24:27 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => [email]] via EMAIL [level => 1]
Restarting tailwatchd....

Thanks for any additional help!

Would you be able to submit a ticket so I can investigate?

https://tickets.cpanel.net/submit/

Make the subject start with 'ATTN: Dan' and referecen this thread, thanks!

[airynd]

Hi there -

Thanks for the great information, guys - that should help. Dan, I'll go ahead and open up a ticket with all the relevant details. I'll also uncheck eximstats in WHM. :-)

[airynd]

Ticket sent. :-)

[cpdan]

Thanks for the ticket, this is a similar issue that I discovered last night:

Either the Eximstas driver can not connect to mysql, or it can connect but the eximstats table is corrupt.

In that circumstance any queries are logged for later execution.

Also in that circumstance some database handle cleanup methods are called which happen to trigger a bug in either DBI or the mysql driver that results in a memory leak.

Since this happens very often on busy servers the memory leak eventually seg faults tailwatchd.

I'll post an update soon regarding the fix.

In the meantime, if you experience this just check your /usr/local/cpanel/logs/tailwatchd_log and if you see errors about not being able to connect to mysql or about some exim tables being corrupt then resolve that problem and the memory leak bug in DBI is not triggered.

[cpdan]

Hello,

Sorry for the delay. The workaround should be out for all tree's that use use tailwatchd.

Please let me know if you have any other issues with this, thanks!