Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

tailwatchd The service “tailwatchd” appears to be down

Discussion in 'CloudLinux' started by ally1, Nov 29, 2017.

  1. ally1

    ally1 Member

    Joined:
    Nov 13, 2014
    Messages:
    12
    Likes Received:
    2
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Hello WHM & cPanel support and community,

    I encountered the tailwatchd problem. I got a lot of emails with the following messages:


    Code:
    ---------- Forwarded message ----------
    From: cPanel Monitoring <cpanel@***my_server***>
    Date: Wed, Nov 29, 2017 at 6:37 AM
    Subject: [***my_server***] FAILED : tailwatchd
    (10.0.131.33)
    To: my@email_here
    
    
    The service “tailwatchd” appears to be down.
    
    Server ***my_server***
    Primary IP Address 10.0.131.33
    Service Name tailwatchd
    Service Status failed
    Notification The service “tailwatchd” appears to be down.
    Service Check Raw Output Cpanel::Exception::Timeout/(XID d5nfna) The system
    failed to lock the file “/var/run/restartsrv_tailwatchd” after 121 seconds.
    at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 336.
    Cpanel::Exception::create("Timeout", "The system failed to lock the file
    \x{e2}\x{80}\x{9c}[_1]\x{e2}\x{80}\x{9d} after [quant,_2"...,
    ARRAY(0x2928f60)) called at /usr/local/cpanel/Cpanel/Exception.pm line 46
    Cpanel::Exception::__ANON__(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__...,
    ARRAY(0x2928f60)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 710
    Cpanel::SafeFile::_timeout_exception("/var/run/restartsrv_tailwatchd", 121)
    called at /usr/local/cpanel/Cpanel/SafeFile.pm line 742
    Cpanel::SafeFile::_die_if_file_is_flocked_cuz_already_
    waited_a_while("/var/run/restartsrv_tailwatchd", 121) called at
    /usr/local/cpanel/Cpanel/SafeFile.pm line 811
    Cpanel::SafeFile::_lock_wait("/var/run/restartsrv_tailwatchd") called at
    /usr/local/cpanel/Cpanel/SafeFile.pm line 339
    Cpanel::SafeFile::_safelock("/var/run/restartsrv_tailwatchd") called at
    /usr/local/cpanel/Cpanel/SafeFile.pm line 538
    Cpanel::SafeFile::_safe_open(undef, ">>", "/var/run/restartsrv_tailwatchd",
    CODE(0xc0ea38), "safeopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm
    line 107
    Cpanel::SafeFile::safeopen(undef, ">>", "/var/run/restartsrv_tailwatchd")
    called at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25
    Cpanel::RestartSrv::Lock::new("Cpanel::RestartSrv::Lock", "tailwatchd")
    called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 183
    Cpanel::ServiceManager::Base::lock(Cpanel::ServiceManager::
    Services::Tailwatchd=HASH(0x29036e8)) called at
    /usr/local/cpanel/Cpanel/ServiceManager/Base.pm
    line 459
    Cpanel::ServiceManager::Base::BUILD(Cpanel::ServiceManager::
    Services::Tailwatchd=HASH(0x29036e8), HASH(0xc4bd58)) called at
    /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_
    lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 49
    Cpanel::Class::Object::BUILDALL(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x29036e8),
    HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/
    perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm
    line 41
    Cpanel::Class::Object::new("Cpanel::ServiceManager::Services::Tailwatchd",
    "service", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager.pm
    line 99
    Cpanel::ServiceManager::new("Cpanel::ServiceManager", "service",
    "tailwatchd") called at bin/restartsrv_base.pl line 62
    main::__ANON__() called at /usr/local/cpanel/3rdparty/
    perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
    eval {...} called at
    /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm
    line 90
    Try::Tiny::try(CODE(0xc7c400), Try::Tiny::Catch=REF(0xbe3310)) called at
    bin/restartsrv_base.pl line 109
    Startup Log No startup log
    Memory Information
    Used 3.04 GB
    Available 12.48 GB
    Installed 15.52 GB
    Load Information 1.80 1.54 1.40
    Uptime 11 days, 10 hours, 18 minutes, and 20 seconds
    IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 26.47
    0.67 3.81 31.23 0.05 37.77 Device: tps Blk_read/s Blk_wrtn/s Blk_read
    Blk_wrtn xvda 216.64 2090.67 932.57 2064534465 920912136 xvdg 11.34 97.29
    103.31 96072106 102014040 xvdh 0.73 18.93 6.78 18696002 6695808 xvdf 2.39
    112.79 24.97 111379016 24658896 xvdi 6.53 70.58 526.69 69696360 520108328
    Top Processes
    PID Owner CPU % Memory % Command
    85284 msarts 8.53 0.40 php-fpm: pool msarts_org
    85056 factnotfiction 4.44 0.32 php-fpm: pool factnotfiction_com
    84481 factnotfiction 3.91 0.32 php-fpm: pool factnotfiction_com
    83298 factnotfiction 3.59 0.32 php-fpm: pool factnotfiction_com
    83214 factnotfiction 3.30 0.32 php-fpm: pool factnotfiction_com
    
    Configure Monitor Settings:
    https://***my_server***:2087/scripts/srvmng#service-chkservd
    
    Configure chkservd:
    https://***my_server***:2087/scripts2/tweaksettings?find=
    chkservd
    
    Disable HTML notifications:
    https://***my_server***:2087/scripts2/tweaksettings?find=
    chkservd_plaintext_notify
    
    The system generated this notice on 2017-11-29 at 12:37:02 UTC.
    
    “Service failures (ChkServd)” notifications are currently configured to
    have an importance of “High”. You can change the importance or disable this
    type of notification in WHM’s Contact Manager at: https://***my_server***
    :2087/scripts2/editcontact?event=chkservd
    
    Do not reply to this automated message.
    [image: cP]
    
    Copyright© 2017 cPanel, Inc.
    
    The command /scripts/restartsrv_tailwatchd --status throws the next:

    Code:
    ################
    Cpanel::Exception::Timeout/(XID 8k6uua) The system failed to lock the file “/var/run/restartsrv_tailwatchd” after 121 seconds.
     at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 336.
            Cpanel::Exception::create("Timeout", "The system failed to lock the file \x{e2}\x{80}\x{9c}[_1]\x{e2}\x{80}\x{9d} after [quant,_2"..., ARRAY(0x1696ce8)) called at /usr/local/cpanel/Cpanel/Exception.pm line 46
            Cpanel::Exception::__ANON__(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__..., ARRAY(0x1696ce8)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 710
            Cpanel::SafeFile::_timeout_exception("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 742
            Cpanel::SafeFile::_die_if_file_is_flocked_cuz_already_waited_a_while("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 811
            Cpanel::SafeFile::_lock_wait("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 339
            Cpanel::SafeFile::_safelock("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 538
            Cpanel::SafeFile::_safe_open(undef, ">>", "/var/run/restartsrv_tailwatchd", CODE(0xc0ea38), "safeopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 107
            Cpanel::SafeFile::safeopen(undef, ">>", "/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25
            Cpanel::RestartSrv::Lock::new("Cpanel::RestartSrv::Lock", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 183
            Cpanel::ServiceManager::Base::lock(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x16729a0)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 459
            Cpanel::ServiceManager::Base::BUILD(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x16729a0), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 49
            Cpanel::Class::Object::BUILDALL(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x16729a0), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 41
            Cpanel::Class::Object::new("Cpanel::ServiceManager::Services::Tailwatchd", "service", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager.pm line 99
            Cpanel::ServiceManager::new("Cpanel::ServiceManager", "service", "tailwatchd") called at bin/restartsrv_base.pl line 62
            main::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
            eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
            Try::Tiny::try(CODE(0xc7d078), Try::Tiny::Catch=REF(0xbe3310)) called at bin/restartsrv_base.pl line 109
    
    ################
    The /etc/init.d/cpanel restart command throws the same error.
    Also I tried to update 'cPanel' server with "
    /scripts/upcp" command. The cpanel was updated successfully but there still the problem with updating tailwatchd, here is output:

    Code:
    ################
    *** Running restart_tailwatchd_if_updated ***
    tailwatchd has newer code than the current running version.  Restarting
    Cpanel::Exception::Timeout/(XID ym38rd) The system failed to lock the file <E2><80><9C>/var/run/restartsrv_tailwatchd<E2><80><9D> after 121 seconds.
     at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 336.
            Cpanel::Exception::create("Timeout", "The system failed to lock the file \x{e2}\x{80}\x{9c}[_1]\x{e2}\x{80}\x{9d} after [quant,_2"..., ARRAY(0x1a874c8)) called at /usr/local/cpanel/Cpanel/Exception.pm line 46
            Cpanel::Exception::__ANON__(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__..., ARRAY(0x1a874c8)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 710
            Cpanel::SafeFile::_timeout_exception("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 742
            Cpanel::SafeFile::_die_if_file_is_flocked_cuz_already_waited_a_while("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 811
            Cpanel::SafeFile::_lock_wait("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 339
            Cpanel::SafeFile::_safelock("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 538
            Cpanel::SafeFile::_safe_open(undef, ">>", "/var/run/restartsrv_tailwatchd", CODE(0xc0ea38), "safeopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 107
            Cpanel::SafeFile::safeopen(undef, ">>", "/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25
            Cpanel::RestartSrv::Lock::new("Cpanel::RestartSrv::Lock", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 183
            Cpanel::ServiceManager::Base::lock(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x1a5caa8)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 459
            Cpanel::ServiceManager::Base::BUILD(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x1a5caa8), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 49
            Cpanel::Class::Object::BUILDALL(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x1a5caa8), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 41
            Cpanel::Class::Object::new("Cpanel::ServiceManager::Services::Tailwatchd", "service", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager.pm line 99
            Cpanel::ServiceManager::new("Cpanel::ServiceManager", "service", "tailwatchd") called at bin/restartsrv_base.pl line 62
            main::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
            eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
            Try::Tiny::try(CODE(0xc7d078), Try::Tiny::Catch=REF(0xbe3310)) called at bin/restartsrv_base.pl line 109
    *** Done restart_tailwatchd_if_updated ***
    ################
    
    I'm sorry for the long explanation.
    Can someone explain how to fix the abowe issue?

    Thanks.
     
    #1 ally1, Nov 29, 2017
    Last edited by a moderator: Nov 29, 2017
  2. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,214
    Likes Received:
    1,937
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    Could you open a support ticket using the link in my signature so we can take a closer look?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. ally1

    ally1 Member

    Joined:
    Nov 13, 2014
    Messages:
    12
    Likes Received:
    2
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Hello,

    I opened the new support ticket. Ticket ID is: 9073163

    Thanks.
     
  4. ally1

    ally1 Member

    Joined:
    Nov 13, 2014
    Messages:
    12
    Likes Received:
    2
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Hello,

    Just for those who encountered the same problem, the issue was with eximstats database that had grown too large (about 13 Gb), and tailwatchd was timing out when attempting to start up. The eximstats_db.sqlite3 files was re-initialized and now tailwatchd is up and working.

    Thanks cPanel support for the help!

    Best Regards,
    Alexander Lymar
     
    did-vmonroig likes this.
  5. did-vmonroig

    did-vmonroig Active Member

    Joined:
    Feb 6, 2012
    Messages:
    40
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Root Administrator
    Hello,

    Thanks for your report. We had a spam event and same problem started because that database has become huge. Time for cleaning!

    Regards,
     
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice