tailwatchd service failure after update to cPanel & WHM version 88

Operating System & Version
CentOS Linux release 7.8.2003 virtuozzo
cPanel & WHM Version
11.88.0.7

Michael-Inet

Well-Known Member
Feb 20, 2014
104
13
68
Austin, TX, USA
cPanel Access Level
Root Administrator
Getting one of these emails every hour after updatenow to '11.88.0.7.' yum update and reboot didn’t help.

tl;dr Says tailwatchd is already running so fails to start it.

Fair enough, but what process is saying it’s down then that it tries to restart it?

Or, more relevant, how do I get them to stop?

Code:
Service Check Raw Output

(XID y4jnv3) The “tailwatchd” service is down.

Startup Log

Jun 09 02:20:45 srv03 systemd[1]: Starting tailwatchd...
Jun 09 02:20:45 srv03 restartsrv_tailwatchd[16039]: tailwatchd is already running (tailwatchd) with PID 804 by root
Jun 09 02:20:45 srv03 systemd[1]: tailwatchd.service: control process exited, code=exited status=1
Jun 09 02:20:45 srv03 systemd[1]: Failed to start tailwatchd.
Jun 09 02:20:45 srv03 systemd[1]: Unit tailwatchd.service entered failed state.
Jun 09 02:20:45 srv03 systemd[1]: tailwatchd.service failed.

Log Messages

The system could not provide log messages for “tailwatchd” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/secure”: “No such file or directory”, Error while attempting to open “/var/log/maillog”: “No such file or directory”, Error while attempting to open “/var/log/messages”: “No such file or directory”

Memory Information

Used 1,002 MB
Available 7.49 GB
Installed 8 GB
Load Information 0.62 1.06 0.64
Uptime 17 days, 12 hours, 21 minutes, and 1 second
rofl
# /usr/local/cpanel/bin/whmapi1 version
[whmapi1] This program can only be run by root!

CentOS Linux release 7.8.2003 virtuozzo
Target version set to '11.88.0.7'

==== Edit ===

Two updates in one day? Uhm, okay...

Still broken, still getting the tailwatchd.service failed every hour...

And the update to '11.88.0.9' went FUBAR as well...

Code:
=> Log opened from /usr/local/cpanel/scripts/post_sync_cleanup (2424) at Tue Jun  9 18:12:26 2020
[2020-06-09 18:12:26 -0500]   Processing: Restarting queueprocd before taskrun
[2020-06-09 18:12:26 -0500]    - Processing command `/usr/local/cpanel/scripts/restartsrv_queueprocd`
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Waiting for “queueprocd” to start ………failed.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Cpanel::Exception::Services::StartError
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Service Status
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  undefined status from Cpanel::ServiceManager::Services::Queueprocd
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Service Error
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  (XID dye4e2) The “queueprocd” service failed to start.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Startup Log
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:12:26 srv03 systemd[1]: Starting cPanel Queue services...
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:12:26 srv03 startqueueprocd[2430]: ==> cPanel TaskQueue Processing Daemon version 1.0
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:12:26 srv03 systemd[1]: Can't open PID file /var/run/queueprocd.pid (yet?) after start: Too many levels of symbolic links
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:13:56 srv03 systemd[1]: queueprocd.service start operation timed out. Terminating.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:13:56 srv03 systemd[1]: Failed to start cPanel Queue services.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:13:56 srv03 systemd[1]: Unit queueprocd.service entered failed state.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:13:56 srv03 systemd[1]: queueprocd.service failed.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] queueprocd has failed. Contact your system administrator if the service does not automagically recover.
=> Log closed Tue Jun  9 18:13:56 2020
Time to hire some QA people????
 
Last edited:

Michael-Inet

Well-Known Member
Feb 20, 2014
104
13
68
Austin, TX, USA
cPanel Access Level
Root Administrator
WTF?

TierCurrentlyDescription
LTS86.0.21 A well-tested and proven version that cPanel, L.L.C. supports for one full year.
STABLE86.0.21 A version that is feature-complete, fully tested, and is in widespread use.
RELEASE
Recommended
88.0.9 A version that is feature-complete and well tested.
CURRENT88.0.9 A “release candidate” version that is tested but may not be feature-complete.
EDGE88.0.9 A development version for testing only and not for production servers.


This server is suppose to be on "Stable" but "Release" is selected which is also Edge? Based on the entire thread about “v88 Known Issues,” v88 is known to be not ready for production machines, yet it’s allowed to be a "Recommended" tier?

Seriously, who’s bright idea is it to have production servers running an experimental release?

Is there any other way than completely rebuilding the server to get this back to an actual production level release?
 

jigster

Member
May 17, 2007
6
0
151
We have exactly the same issue with tailwatchd errors every hour, on just one server - a cpanel DNSONLY server . Same errors exactly. Any progress on a fix for this?
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,304
1,250
313
Houston
The program that can't restart *is* tailwatchd - it runs in the background in order to manage other services. I was going to have you check /var/log/messages for the systemd output when it is restarted or run:

systemctl start tailwatchd.service

But it is best this be handled in the ticket since they will have access to the system
 
  • Like
Reactions: Michael-Inet

Michael-Inet

Well-Known Member
Feb 20, 2014
104
13
68
Austin, TX, USA
cPanel Access Level
Root Administrator
The program that can't restart *is* tailwatchd
Thanks for that! This log line is just really confusing "tailwatchd is already running (tailwatchd) with PID 804 by root!"

Someone's in the box now looking at it, if he doesn't update here with results, I'll do so when he's finished.

Thanks again,
Michael
 
  • Like
Reactions: cPanelLauren

Michael-Inet

Well-Known Member
Feb 20, 2014
104
13
68
Austin, TX, USA
cPanel Access Level
Root Administrator
Hi All,

Not a cPanel problem... For details read the reply below from cPanel support. Including my ticket to my hosting company if you just want to cut&paste:


Hi Ya,

Okay, I admit I don’t fully understand the fine level details with this issue, but apparently recent CentOS updates are causing errors with older kernels (symlinks specifically?).

Request: Please update the kernel on the node and reboot.

I’m copying the email cPanel support sent me below as background information, with links to relevant bug reports, etc.

Thanks,
Michael

Hmm, it expanded links, but should still work...

====

Subject: [cPanel] Re: tailwatchd service failure after update to cPanel & WHM version 88
Date: 06/11/20 14:40:35
From: "Jeremy (cPanel Technical Support)"
To: Michael

----------------------------------------------

Jeremy, Jun 11, 2020, 2:40:34 PM CDT

Hello again,

Thank you for your time and patience. I am sorry for any inconvenience, but at this time I do believe I have some information to share.

The issue you are facing is not specific to the recent cPanel updates, but it also related to recent CentOS updates and how CentOS now manages symlinks. If your kernel does not support the new method that systemd uses to open symlinks, you will see errors like these, i.e., “Too many levels of symbolic links”:

For example, I tried to restart MySQL on your server, and encountered several of this error:

Jun 09 02:25:04 srv03 systemd[1]: Stopped MySQL Server.
Jun 09 02:25:04 srv03 systemd[1]: Starting MySQL Server...
Jun 09 02:25:04 srv03 systemd[1]: Can't open PID file /var/run/mysqld/mysqld.pid (yet?) after start: Too many levels of symbolic links
Jun 09 08:52:07 srv03 systemd[1]: Stopped MySQL Server.
Jun 10 20:35:10 srv03 systemd[1]: Starting MySQL Server...
Jun 10 20:35:10 srv03 systemd[1]: Can't open PID file /var/run/mysqld/mysqld.pid (yet?) after start: Too many levels of symbolic links

This problem will be seen on systems with the following configurations:


- CentOS 7.7 and OpenVZ/Virtuozzo with kernel prior to 2.6.32-042stab134.7

- CentOS 7.7 with kernel prior to 2.6.39

Your server is currently running the kernel `2.6.32-042stab113.21`

**Do not** reboot your server without updating the kernel first. Rebooting the server will likely cause further issues with systemd due to the outdated kernel.

I can see your server is running on OpenVZ/Virtuozzo:

# cat /var/cpanel/envtype
virtuozzo

This means that your host will need to update the kernel on the node and reboot to correct this issue. Or, if you have access to the host node you may be able to do this yourself.

Please understand this is an operating-system issue with the kernel and systemd, not an issue with cPanel. While the following page is a bug reported against Ubuntu, it is still relevant, and contains a response from OpenVZ developers about the issue:


We also have further information about the issue in our forums post below:


If you have any more questions or issues on this subject, please let us know. I am grateful for your time and patience.

Thank you,


* * *

Jeremy
Technical Analyst
cPanel, L.L.C.

--------------------------------
This email is a service from cPanel.
 
  • Like
Reactions: cPanelLauren