The Community Forums

Interact with an entire community of cPanel & WHM users.
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

tailwatchd - chkservd on server status: hang

Discussion in 'General Discussion' started by crazyaboutlinux, Feb 11, 2012.

  1. crazyaboutlinux

    crazyaboutlinux Well-Known Member

    Joined:
    Nov 3, 2007
    Messages:
    938
    Likes Received:
    0
    Trophy Points:
    16
    hi guys,

    i am keep getting emails regarding "tailwatchd - chkservd on xxxxxx.xxxxxxx.xxxx status: hang"

    The chkservd sub-process with pid 7078 was running for 301 seconds. The sub-process was terminated as it exceeded the time between checks of 300 seconds. Please check /var/log/chkservd.log and /usr/local/cpanel/logs/tailwatchd_log to discover the cause.

    and i have checked the logs of /usr/local/cpanel/logs/tailwatchd_log

    Please find below output

    Code:
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] /var/log/maillog opened with inode 325624
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] /var/log/exim_mainlog opened with inode 325474
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Restored /var/log/maillog (size:153895090) to 153885825 (requested 153885825)
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Restored /var/log/exim_mainlog (size:72297657) to 72293870 (requested 72293870)
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Caught up /var/log/maillog to 153895090
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Reading back thirty lines of /var/log/maillog starting at 153878706
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Restoring /var/log/maillog to catch up position 153895090
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Restored /var/log/maillog to position 153895090
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Caught up /var/log/exim_mainlog to 72297657
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Reading back thirty lines of /var/log/exim_mainlog starting at 72281273
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Restoring /var/log/exim_mainlog to catch up position 72297657
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch] [INFO] Restored /var/log/exim_mainlog to position 72297657
    [2012-02-11 13:58:16 +0530] [Cpanel::TailWatch::ChkServd] [INFO] Opening /var/log/chkservd.log in append mode
    [2012-02-11 13:59:58 +0530] [Cpanel::TailWatch] [INFO] Opened /usr/local/cpanel/logs/tailwatchd_log in append mode
    [2012-02-11 13:59:58 +0530] [Cpanel::TailWatch] [INFO] inotify support not available (Linux::Inotify2 missing or non-functional)
    
    any ideas are welcome!
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    27,671
    Likes Received:
    419
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    Hello :)

    This notification is sent out when "tailwatchd::chkservd" hangs. I have seen a few instances where this has happened as a result of the "update_sa_rules" command hanging due to a SpamAssassin RPM installed on the system. Please check to see if you have a SpamAssassin RPM installed on your system:

    Code:
    # rpm -qa|grep -i spam
    If you do, I advise removing it, and reinstalling SpamAssassin via:

    Code:
    # /scripts/installspam
    To note, there is an internal case open to ensure cPanel is always running the instance of SpamAssassin that is installed by Perl.

    Thank you.
     
  3. guvenuzman

    guvenuzman Registered

    Joined:
    Sep 25, 2008
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    Hello cPanelMichael,

    I applied what they say, but I still have this problem is not solved.
    The following e-mails continue to arrive;

    "The chkservd sub-process with pid 32278 was running for 309 seconds. The sub-process was terminated as it exceeded the time between checks of 300 seconds. Please check /var/log/chkservd.log and /usr/local/cpanel/logs/tailwatchd_log to discover the cause."

    What can I do to solve this problem?

    Thanks
     
  4. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    18
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Are you running CloudLinux with CageFS enabled? If so, then please try disabling CageFS. We had an internal case 54061 about CageFS causing issues with CloudLinux for chkservd and tailwatchd. It is supposed to be disabled in checks as of cPanel 11.30.6.0 and higher.
     
  5. sharmila

    sharmila Registered

    Joined:
    Sep 11, 2007
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    Cloud linux is not installed on the server. The spam rpm is also installed but getting below notifications:-

    The chkservd sub-process with pid 18078 was running for 301 seconds. The sub-process was terminated as it exceeded the time between checks of 300 seconds. Please check /var/log/chkservd.log and /usr/local/cpanel/logs/tailwatchd_log to discover the cause.

    Please let me know the exact cause as this is occurring from past 2 days.


    Thank You
     
  6. rohit09

    rohit09 Well-Known Member

    Joined:
    Dec 10, 2010
    Messages:
    59
    Likes Received:
    0
    Trophy Points:
    6
    ****tailwatchd - chkservd on (Hostname) status: hang - How do I solve it?

    The chkservd sub-process with pid 13833 was running for 302 seconds. ****The sub-process was terminated as it exceeded the time between checks of 300 seconds. ****Please check /var/log/chkservd.log and /usr/local/cpanel/logs/tailwatchd_log to discover the cause.

    - I receive 100s of spam each hrs. Please tell me how do I stop it?

    What is the issue. I have normal OpenVZ with cPanel

    Thank You.
     
  7. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    18
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Hello,

    Unfortunately, it isn't possible to let you know the exact cause without login credentials to review your system, since the error message doesn't indicate what is causing it and the logs do not always show what is the cause (based on some prior tickets we've had on the issue).

    We did have two recent tickets where it was due to the update_sa_rules taking longer than the permitted 300 seconds that Michael mentioned, and in the past we've had tickets where it was due to CageFS. We have internal cases on each, and the CageFS issue is fixed already. The update_sa_rules case (57323) is slated to be available once your tier reaches 11.32

    If you wish to know what exactly is causing the issue, then please submit a ticket using WHM > Support Center > Contact cPanel or using the link in my signature. We would be happy to help track the resolution of the issue if you provide the ticket number here upon submitting one.

    Thanks!
     
  8. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    27,671
    Likes Received:
    419
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
  9. accretor

    accretor Member

    Joined:
    Feb 3, 2012
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    chkservd and tailwatchd

    I get this email notification at 1343 hrs local time:

    Code:
    The chkservd sub-process with pid 22115 was running for 301 seconds.  The sub-process was terminated as it exceeded the time between checks of 300 seconds.  Please check /var/log/chkservd.log and /usr/local/cpanel/logs/tailwatchd_log to discover the cause.
    A review of the logs gives no indication. Can anyone suggest a possible additional course of investigation? Time-relevant except of the logs follow.


    /var/log/chkservd.log from 1338 hrs onward:

    Code:
    [2012-02-21 13:38:49 -0800] Service check ....syslogd [[check command:+][tcp connect:N/A]]...sshd [[check command:+][tcp connect:N/A]]...spamd [Service Check Started
    Loading services .....cpanellogd....cpdavd....cpsrvd....exim....ftpd....httpd....imap....mysql....named....queueprocd....spamd....sshd....syslogd..Done
    [2012-02-21 13:44:12 -0800] Service check ....syslogd [[check command:+][tcp connect:N/A]]...sshd [[check command:+][tcp connect:N/A]]...spamd [[check command:+][tcp connect:N/A]]..
    .queueprocd [[check command:+][tcp connect:N/A]]...named [[check command:+][tcp connect:N/A]]...mysql [[check command:+][tcp connect:N/A]]...melange [[check command:N/A][tcp connect
    :N/A]]...imap [[socket_service_auth:1][check command:+][tcp connect:+]]...httpd [[check command:N/A][tcp connect:+]]...ftpd [[socket_service_auth:1][check command:+][tcp connect:+]]
    ...exim [[check command:+][tcp connect:+]]...entropychat [[check command:N/A][tcp connect:N/A]]...cpsrvd [[http_service_auth:1][check command:N/A][tcp connect:+]]...cpdavd [[http_se
    rvice_auth:1][check command:+][tcp connect:+]]...cpanellogd [[check command:+][tcp connect:N/A]]...Done
    Service Check Finished
    Service Check Started
    Loading services .....cpanellogd....cpdavd....cpsrvd....exim....ftpd....httpd....imap....mysql....named....queueprocd....spamd....sshd....syslogd..Done
    [2012-02-21 13:47:57 -0800] Service check ....syslogd [[check command:+][tcp connect:N/A]]...sshd [[check command:+][tcp connect:N/A]]...spamd [[check command:+][tcp connect:N/A]]...queueprocd [[check command:+][tcp connect:N/A]]...named [[check command:+][tcp connect:N/A]]...mysql [[check command:+][tcp connect:N/A]]...melange [[check command:N/A][tcp connect:N/A]]...imap [[socket_service_auth:1][check command:+][tcp connect:+]]...httpd [[check command:N/A][tcp connect:+]]...ftpd [[socket_service_auth:1][check command:+][tcp connect:+]]...exim [[check command:+][tcp connect:+]]...entropychat [[check command:N/A][tcp connect:N/A]]...cpsrvd [[http_service_auth:1][check command:N/A][tcp connect:+]]...cpdavd [[http_service_auth:1][check command:+][tcp connect:+]]...cpanellogd [[check command:+][tcp connect:N/A]]...Done
    Service Check Finished
    Service Check Started
    Loading services .....cpanellogd....cpdavd....cpsrvd....exim....ftpd....httpd....imap....mysql....named....queueprocd....spamd....sshd....syslogd..Done
    [2012-02-21 13:52:58 -0800] Service check ....syslogd [[check command:+][tcp connect:N/A]]...sshd [[check command:+][tcp connect:N/A]]...spamd [[check command:+][tcp connect:N/A]]...queueprocd [[check command:+][tcp connect:N/A]]...named [[check command:+][tcp connect:N/A]]...mysql [[check command:+][tcp connect:N/A]]...melange [[check command:N/A][tcp connect:N/A]]...imap [[socket_service_auth:1][check command:+][tcp connect:+]]...httpd [[check command:N/A][tcp connect:+]]...ftpd [[socket_service_auth:1][check command:+][tcp connect:+]]...exim [[check command:+][tcp connect:+]]...entropychat [[check command:N/A][tcp connect:N/A]]...cpsrvd [[http_service_auth:1][check command:N/A][tcp connect:+]]...cpdavd [[http_service_auth:1][check command:+][tcp connect:+]]...cpanellogd [[check command:+][tcp connect:N/A]]...Done
    Service Check Finished
    Service Check Started
    Loading services .....cpanellogd....cpdavd....cpsrvd....exim....ftpd....httpd....imap....mysql....named....queueprocd....spamd....sshd....syslogd..Done
    [2012-02-21 13:57:59 -0800] Service check ....syslogd [[check command:+][tcp connect:N/A]]...sshd [[check command:+][tcp connect:N/A]]...spamd [[check command:+][tcp connect:N/A]]...queueprocd [[check command:+][tcp connect:N/A]]...named [[check command:+][tcp connect:N/A]]...mysql [[check command:+][tcp connect:N/A]]...melange [[check command:N/A][tcp connect:N/A]]...imap [[socket_service_auth:1][check command:+][tcp connect:+]]...httpd [[check command:N/A][tcp connect:+]]...ftpd [[socket_service_auth:1][check command:+][tcp connect:+]]...exim [[check command:+][tcp connect:+]]...entropychat [[check command:N/A][tcp connect:N/A]]...cpsrvd [[http_service_auth:1][check command:N/A][tcp connect:+]]...cpdavd [[http_service_auth:1][check command:+][tcp connect:+]]...cpanellogd [[check command:+][tcp connect:N/A]]...Done
    Service Check Finished
    Service Check Started
    Loading services .....cpanellogd....cpdavd....cpsrvd....exim....ftpd....httpd....imap....mysql....named....queueprocd....spamd....sshd....syslogd..Done
    [2012-02-21 14:03:00 -0800] Service check ....syslogd [[check command:+][tcp connect:N/A]]...sshd [[check command:+][tcp connect:N/A]]...spamd [[check command:+][tcp connect:N/A]]...queueprocd [[check command:+][tcp connect:N/A]]...named [[check command:+][tcp connect:N/A]]...mysql [[check command:+][tcp connect:N/A]]...melange [[check command:N/A][tcp connect:N/A]]...imap [[socket_service_auth:1][check command:+][tcp connect:+]]...httpd [[check command:N/A][tcp connect:+]]...ftpd [[socket_service_auth:1][check command:+][tcp connect:+]]...exim [[check command:+][tcp connect:+]]...entropychat [[check command:N/A][tcp connect:N/A]]...cpsrvd [[http_service_auth:1][check command:N/A][tcp connect:+]]...cpdavd [[http_service_auth:1][check command:+][tcp connect:+]]...cpanellogd [[check command:+][tcp connect:N/A]]...Done
    Service Check Finished
    Service Check Started
    Loading services .....cpanellogd....cpdavd....cpsrvd....exim....ftpd....httpd....imap....mysql....named....queueprocd....spamd....sshd....syslogd..Done
    [2012-02-21 14:08:01 -0800] Service check ....syslogd [[check command:+][tcp connect:N/A]]...sshd [[check command:+][tcp connect:N/A]]...spamd [[check command:+][tcp connect:N/A]]...queueprocd [[check command:+][tcp connect:N/A]]...named [[check command:+][tcp connect:N/A]]...mysql [[check command:+][tcp connect:N/A]]...melange [[check command:N/A][tcp connect:N/A]]...imap [[socket_service_auth:1][check command:+][tcp connect:+]]...httpd [[check command:N/A][tcp connect:+]]...ftpd [[socket_service_auth:1][check command:+][tcp connect:+]]...exim [[check command:+][tcp connect:+]]...entropychat [[check command:N/A][tcp connect:N/A]]...cpsrvd [[http_service_auth:1][check command:N/A][tcp connect:+]]...cpdavd [[http_service_auth:1][check command:+][tcp connect:+]]...cpanellogd [[check command:+][tcp connect:N/A]]...Done
    Service Check Finished
    Service Check Started
    Loading services .....cpanellogd....cpdavd....cpsrvd....exim....ftpd....httpd....imap....mysql....named....queueprocd....spamd....sshd....syslogd..Done
    [2012-02-21 14:13:02 -0800] Service check ....syslogd [[check command:+][tcp connect:N/A]]...sshd [[check command:+][tcp connect:N/A]]...spamd [[check command:+][tcp connect:N/A]]...queueprocd [[check command:+][tcp connect:N/A]]...named [[check command:+][tcp connect:N/A]]...mysql [[check command:+][tcp connect:N/A]]...melange [[check command:N/A][tcp connect:N/A]]...imap [[socket_service_auth:1][check command:+][tcp connect:+]]...httpd [[check command:N/A][tcp connect:+]]...ftpd [[socket_service_auth:1][check command:+][tcp connect:+]]...exim [[check command:+][tcp connect:+]]...entropychat [[check command:N/A][tcp connect:N/A]]...cpsrvd [[http_service_auth:1][check command:N/A][tcp connect:+]]...cpdavd [[http_service_auth:1][check command:+][tcp connect:+]]...cpanellogd [[check command:+][tcp connect:N/A]]...Done
    Service Check Finished
    /usr/local/cpanel/logs/tailwatchd_log has no records dated after "[2012-02-19 04:02:39 -0800]". The last line is "Notification => __email removed__ via EMAIL [level => 1]"
     
  10. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    18
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    The best way to track this down if the suggestions provided in this thread do not assist would be to submit a ticket using the link in my signature or WHM > Support Center > Contact cPanel area. Thanks!
     
  11. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    This case is resolved in version 11.32.1.3 and later which is now propagating.
     
  12. PbG

    PbG Well-Known Member

    Joined:
    Mar 11, 2003
    Messages:
    238
    Likes Received:
    0
    Trophy Points:
    16
    David Please advise which tree this version is on? Eg: Stable, Release, etc

     
  13. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    27,671
    Likes Received:
    419
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    cPanel 11.32.1.4 is available on the "Current" tier. The tiers are merely an indication of update frequency, so it's better to focus on the actual version number. You can review the "Latest cPanel & WHM Builds (All Architectures)" at:

    layer2.cpanel.net

    Thank you.
     
  14. jonez334

    jonez334 Registered

    Joined:
    Dec 22, 2011
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    I'm also receiving this The chkservd sub-process with pid 27689 was running for 301 seconds. The sub-process was terminated as it exceeded the time between checks of 300 seconds. Please check /var/log/chkservd.log and /usr/local/cpanel/logs/tailwatchd_log to discover the cause. I need someone to explain thoroughly how to stop this and if its a big issue. I'm sorry for not being so tech savvy.
     
  15. gkgcpanel

    gkgcpanel Well-Known Member

    Joined:
    Jun 6, 2007
    Messages:
    217
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    DataCenter Provider
    So I have a question about this line in the email that is received:

    Code:
    The chkservd sub-process with pid 22115 was running for 301 seconds.  The sub-process was terminated ...
    
    If cPanel already knows the pid number (as evidenced in the email), why not also display what process that pid belongs to?

    That would really help in determining if it's exim, httpd, spamd, mysql, etc... By the time this email arrives, that pid no longer exists because it was terminated. So there's no way to know which process was actually terminated (and hopefully restarted). But if the pid is known prior to (or at the time) the HANG is determined, then you should be able to include the exact process name that is causing the problem right?
     
  16. sukanime

    sukanime Registered

    Joined:
    Apr 14, 2013
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    AFAIK, That the pid of chkservd its self.

    i have this kind of error too, but before it killed, i got email "High 5 minute load average alert - 8.05"
    then 4 minutes after that email, i got email "HANG: chkservd on server xxx"
    with message:
    Code:
    The chkservd sub-process with pid 558 ran for 301 seconds. This sub-process was
    terminated when it exceeded the time allowed between checks, which is 300 seconds.
    To determine why, you can check /var/log/chkservd.log and
    /usr/local/cpanel/logs/tailwatchd_log.
    
    So i compare this information with ps.txt form email "High 5 minute load average alert - 8.05"
    And i got this:
    Code:
    root       558  0.0  0.0  30304  2284 ?        D    07:35   0:00  \_ tailwatchd -
    chkservd
    
    CMIIW.
     
  17. palav0s

    palav0s Well-Known Member

    Joined:
    Jul 15, 2011
    Messages:
    55
    Likes Received:
    0
    Trophy Points:
    6
    Hi,

    What is the safest way to remove it?

    Yum requires a bunch of important packages to be removed for dependencies:
    Code:
    cpanel-perl-514-Task-Cpanel-Core
    cronie                                       
    cronie-noanacron
    crontabs
    exim
    sysstat
     
  18. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    27,671
    Likes Received:
    419
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    Hello :)

    Please keep in mind that you have replied to an older thread. cPanel has now moved to an RPM-based distribution system. That being said, you should be able to remove the older SpamAssassin RPM with a command such as:

    Code:
    rpm -e --nodeps package-name
    Thank you.
     
  19. palav0s

    palav0s Well-Known Member

    Joined:
    Jul 15, 2011
    Messages:
    55
    Likes Received:
    0
    Trophy Points:
    6
    Thank you for your reply, I thought it was related to the same issue (on this release).
    By the way I dont's have
    Code:
    /scripts/installspam
    on version 11.38.2.1

    Next time I will start with a new thread thanks for mentioning.
     
    #19 palav0s, Aug 15, 2013
    Last edited: Aug 15, 2013
  20. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    27,671
    Likes Received:
    419
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    Yes, "/scripts/installspam" is no longer used, as it's implemented with the RPM system.

    Thank you.
     
Loading...

Share This Page