The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

tailwatchd perpetually failing

Discussion in 'General Discussion' started by airynd, Dec 11, 2008.

  1. airynd

    airynd Member

    Joined:
    Jun 23, 2008
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Hello,

    We're seeing a fair number of customers reporting tailwatchd failing constantly. This began about 24 hours ago, and I'm wondering if a recent update may have broken something. This is in the /usr/local/cpanel/logs/tailwatchd_log:

    [START] 14636 1229052314
    [STOP Ok] 14636 1229052817
    [Thu Dec 11 22:33:37 2008] Current process '14636' stopped
    [START] 15009 1229052817
    [STOP Ok] 15009 1229053321
    [Thu Dec 11 22:42:01 2008] Current process '15009' stopped
    [START] 15386 1229053321
    [STOP Ok] 15386 1229053824
    [Thu Dec 11 22:50:24 2008] Current process '15386' stopped
    [START] 15773 1229053825
    [root@server.xxxxxxx.net] logs >> tail -f tailwatchd_log
    [START] 14636 1229052314
    [STOP Ok] 14636 1229052817
    [Thu Dec 11 22:33:37 2008] Current process '14636' stopped
    [START] 15009 1229052817
    [STOP Ok] 15009 1229053321
    [Thu Dec 11 22:42:01 2008] Current process '15009' stopped
    [START] 15386 1229053321
    [STOP Ok] 15386 1229053824
    [Thu Dec 11 22:50:24 2008] Current process '15386' stopped
    [START] 15773 1229053825

    Naturally, an email is generated each time, and our customers are becoming... vexed.

    Restarting cPanel will fix this for awhile, but an hour or two or five later, the same shenanigans occur.

    Has anyone else seen this lately?
     
    #1 airynd, Dec 11, 2008
    Last edited: Dec 17, 2008
  2. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,482
    Likes Received:
    203
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    What version of cPanel are you on? There were some updates recently to CURRENT.

    http://changelog.cpanel.net/
     
  3. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,461
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    If I'm reading those time stamps properly, it looks like the restart is happening every 8 minutes, which roughly corresponds to when chkservd runs. What does the chkservd log contain?
     
  4. airynd

    airynd Member

    Joined:
    Jun 23, 2008
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Thanks for the responses. The customer is on cPanel version:

    WHM 11.23.2 cPanel 11.23.6-R27698
    CENTOS Enterprise 5.2 i686 on standard - WHM X v3.1.0

    Here are a few entries from the chkservd.log file:

    [Fri Dec 12 17:59:16 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => via EMAIL [level => 1]
    Restarting tailwatchd....
    [Fri Dec 12 18:07:40 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => via EMAIL [level => 1]
    Restarting tailwatchd....
    [Fri Dec 12 18:16:04 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => [email]via EMAIL [level => 1]
    Restarting tailwatchd....
    [Fri Dec 12 18:24:27 2008] Service check ....clamd [+]...cpdavd [+]...cpsrvd [+]...exim [+]...ftpd [+]...httpd [+]...imap [+]...mysql [+]...named [+]...pop [+]...rsyslogd [+]...spamd [+]...syslogd [+]...tailwatchd [-Notification => [email]] via EMAIL [level => 1]
    Restarting tailwatchd....

    So, nothing terribly useful there... at least not to me. :)

    Thanks for any additional help!
     
  5. Messiah

    Messiah Member
    PartnerNOC

    Joined:
    Jul 5, 2006
    Messages:
    22
    Likes Received:
    0
    Trophy Points:
    1
    One of the functions performed by tailwatchd is eximstats, which has been recently deprecated. Tailwatchd attempts and is unable to restart eximstats when tailwatchd starts, so tailwatchd dies. Try logging into WHM, selecting Service Manager, then un-checking eximstats from the tailwatchd category. Click save, then you should stop getting these emails.

    I imagine this will be fixed in an upcoming release.
     
  6. cPDan

    cPDan cPanel Staff
    Staff Member

    Joined:
    Mar 9, 2004
    Messages:
    711
    Likes Received:
    4
    Trophy Points:
    18
    Interesting to note that its the old style chksrvd (IE the dameon) and not the new (IE the driver)

    Service Check Started
    [2008-12-09 21:32:09 UTC] Service check ....cpsrvd [[check command:N/A][tcp connect:+]]...cpdavd [[check command:+][tcp connect:N/A]]...Done
    Service Check Finished
     
  7. cPDan

    cPDan cPanel Staff
    Staff Member

    Joined:
    Mar 9, 2004
    Messages:
    711
    Likes Received:
    4
    Trophy Points:
    18
    It could be the eximstats driver dying oddly but it doesn't try to restart eximstats. The things the eximstats daemon did are now handled in tailwatchd daemon by the eximstats driver.

    Hope that clears things up a little :) In the mean time I'd love to see a system that is exhibitiung this issue so we can address it ASAP. If anyone has one please open a ticket with the subject 'ATTN: Dan' and refenece this forum thread.

    Thanks!
     
  8. cPDan

    cPDan cPanel Staff
    Staff Member

    Joined:
    Mar 9, 2004
    Messages:
    711
    Likes Received:
    4
    Trophy Points:
    18
    I killed mysql and restarted and all was well (eximstats logs the queries it would have been doing to a file so that you can run them later once mysql is around)

    Is there an error on the screen or in the logs about eximstats driver?
     
  9. cPDan

    cPDan cPanel Staff
    Staff Member

    Joined:
    Mar 9, 2004
    Messages:
    711
    Likes Received:
    4
    Trophy Points:
    18
     
  10. airynd

    airynd Member

    Joined:
    Jun 23, 2008
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Hi there -

    Thanks for the great information, guys - that should help. Dan, I'll go ahead and open up a ticket with all the relevant details. I'll also uncheck eximstats in WHM. :)
     
  11. airynd

    airynd Member

    Joined:
    Jun 23, 2008
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Ticket sent. :)
     
  12. cPDan

    cPDan cPanel Staff
    Staff Member

    Joined:
    Mar 9, 2004
    Messages:
    711
    Likes Received:
    4
    Trophy Points:
    18
    Thanks for the ticket, this is a similar issue that I discovered last night:

    Either the Eximstas driver can not connect to mysql, or it can connect but the eximstats table is corrupt.

    In that circumstance any queries are logged for later execution.

    Also in that circumstance some database handle cleanup methods are called which happen to trigger a bug in either DBI or the mysql driver that results in a memory leak.

    Since this happens very often on busy servers the memory leak eventually seg faults tailwatchd.

    I'll post an update soon regarding the fix.

    In the meantime, if you experience this just check your /usr/local/cpanel/logs/tailwatchd_log and if you see errors about not being able to connect to mysql or about some exim tables being corrupt then resolve that problem and the memory leak bug in DBI is not triggered.
     
  13. cPDan

    cPDan cPanel Staff
    Staff Member

    Joined:
    Mar 9, 2004
    Messages:
    711
    Likes Received:
    4
    Trophy Points:
    18
    Hello,

    Sorry for the delay. The workaround should be out for all tree's that use use tailwatchd.

    Please let me know if you have any other issues with this, thanks!
     
Loading...

Share This Page