tailwatchd service failure after update to cPanel & WHM version 88

[Michael-Inet]

Getting one of these emails every hour after updatenow to '11.88.0.7.' yum update and reboot didn’t help.

tl;dr Says tailwatchd is already running so fails to start it.

Fair enough, but what process is saying it’s down then that it tries to restart it?

Or, more relevant, how do I get them to stop?

Service Check Raw Output

(XID y4jnv3) The “tailwatchd” service is down.

Startup Log

Jun 09 02:20:45 srv03 systemd[1]: Starting tailwatchd...
Jun 09 02:20:45 srv03 restartsrv_tailwatchd[16039]: tailwatchd is already running (tailwatchd) with PID 804 by root
Jun 09 02:20:45 srv03 systemd[1]: tailwatchd.service: control process exited, code=exited status=1
Jun 09 02:20:45 srv03 systemd[1]: Failed to start tailwatchd.
Jun 09 02:20:45 srv03 systemd[1]: Unit tailwatchd.service entered failed state.
Jun 09 02:20:45 srv03 systemd[1]: tailwatchd.service failed.

Log Messages

The system could not provide log messages for “tailwatchd” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/secure”: “No such file or directory”, Error while attempting to open “/var/log/maillog”: “No such file or directory”, Error while attempting to open “/var/log/messages”: “No such file or directory”

Memory Information

Used 1,002 MB
Available 7.49 GB
Installed 8 GB
Load Information 0.62 1.06 0.64
Uptime 17 days, 12 hours, 21 minutes, and 1 second

rofl
# /usr/local/cpanel/bin/whmapi1 version
[whmapi1] This program can only be run by root!

CentOS Linux release 7.8.2003 virtuozzo
Target version set to '11.88.0.7'

==== Edit ===

Two updates in one day? Uhm, okay...

Still broken, still getting the tailwatchd.service failed every hour...

And the update to '11.88.0.9' went FUBAR as well...

=> Log opened from /usr/local/cpanel/scripts/post_sync_cleanup (2424) at Tue Jun  9 18:12:26 2020
[2020-06-09 18:12:26 -0500]   Processing: Restarting queueprocd before taskrun
[2020-06-09 18:12:26 -0500]    - Processing command `/usr/local/cpanel/scripts/restartsrv_queueprocd`
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Waiting for “queueprocd” to start ………failed.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Cpanel::Exception::Services::StartError
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Service Status
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  undefined status from Cpanel::ServiceManager::Services::Queueprocd
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Service Error
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  (XID dye4e2) The “queueprocd” service failed to start.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] Startup Log
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:12:26 srv03 systemd[1]: Starting cPanel Queue services...
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:12:26 srv03 startqueueprocd[2430]: ==> cPanel TaskQueue Processing Daemon version 1.0
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:12:26 srv03 systemd[1]: Can't open PID file /var/run/queueprocd.pid (yet?) after start: Too many levels of symbolic links
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:13:56 srv03 systemd[1]: queueprocd.service start operation timed out. Terminating.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:13:56 srv03 systemd[1]: Failed to start cPanel Queue services.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:13:56 srv03 systemd[1]: Unit queueprocd.service entered failed state.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]  Jun 09 18:13:56 srv03 systemd[1]: queueprocd.service failed.
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd]
[2020-06-09 18:13:56 -0500]      [/usr/local/cpanel/scripts/restartsrv_queueprocd] queueprocd has failed. Contact your system administrator if the service does not automagically recover.
=> Log closed Tue Jun  9 18:13:56 2020

Time to hire some QA people????

 

[Michael-Inet]

WTF?

Tier	Currently		Description
	LTS	86.0.21		A well-tested and proven version that cPanel, L.L.C. supports for one full year.
	STABLE	86.0.21		A version that is feature-complete, fully tested, and is in widespread use.
	RELEASE Recommended	88.0.9		A version that is feature-complete and well tested.
	CURRENT	88.0.9		A “release candidate” version that is tested but may not be feature-complete.
	EDGE	88.0.9		A development version for testing only and not for production servers.

This server is suppose to be on "Stable" but "Release" is selected which is also Edge? Based on the entire thread about “v88 Known Issues,” v88 is known to be not ready for production machines, yet it’s allowed to be a "Recommended" tier?

Seriously, who’s bright idea is it to have production servers running an experimental release?

Is there any other way than completely rebuilding the server to get this back to an actual production level release?

 

[jigster]

We have exactly the same issue with tailwatchd errors every hour, on just one server - a cpanel DNSONLY server . Same errors exactly. Any progress on a fix for this?

 

[Michael-Inet]

Your Support Ticket ID is: 93497725

Addendum:

- Error is happening on a full WHM/cPanel install.
- I know of 2 other DNSONLY servers this is not happening on.

 

[cPanelLauren]

The program that can't restart *is* tailwatchd - it runs in the background in order to manage other services. I was going to have you check /var/log/messages for the systemd output when it is restarted or run:

systemctl start tailwatchd.service

But it is best this be handled in the ticket since they will have access to the system

 

[Michael-Inet]

The program that can't restart *is* tailwatchd

Thanks for that! This log line is just really confusing "tailwatchd is already running (tailwatchd) with PID 804 by root!"

Someone's in the box now looking at it, if he doesn't update here with results, I'll do so when he's finished.

Thanks again,
Michael

 

[Michael-Inet]

Hi All,

Not a cPanel problem... For details read the reply below from cPanel support. Including my ticket to my hosting company if you just want to cut&paste:

Hi Ya,

Okay, I admit I don’t fully understand the fine level details with this issue, but apparently recent CentOS updates are causing errors with older kernels (symlinks specifically?).

Request: Please update the kernel on the node and reboot.

I’m copying the email cPanel support sent me below as background information, with links to relevant bug reports, etc.

Thanks,
Michael

Hmm, it expanded links, but should still work...

====

Subject: [cPanel] Re: tailwatchd service failure after update to cPanel & WHM version 88
Date: 06/11/20 14:40:35
From: "Jeremy (cPanel Technical Support)"
To: Michael

----------------------------------------------

Jeremy, Jun 11, 2020, 2:40:34 PM CDT

Hello again,

Thank you for your time and patience. I am sorry for any inconvenience, but at this time I do believe I have some information to share.

The issue you are facing is not specific to the recent cPanel updates, but it also related to recent CentOS updates and how CentOS now manages symlinks. If your kernel does not support the new method that systemd uses to open symlinks, you will see errors like these, i.e., “Too many levels of symbolic links”:

For example, I tried to restart MySQL on your server, and encountered several of this error:

Jun 09 02:25:04 srv03 systemd[1]: Stopped MySQL Server.
Jun 09 02:25:04 srv03 systemd[1]: Starting MySQL Server...
Jun 09 02:25:04 srv03 systemd[1]: Can't open PID file /var/run/mysqld/mysqld.pid (yet?) after start: Too many levels of symbolic links
Jun 09 08:52:07 srv03 systemd[1]: Stopped MySQL Server.
Jun 10 20:35:10 srv03 systemd[1]: Starting MySQL Server...
Jun 10 20:35:10 srv03 systemd[1]: Can't open PID file /var/run/mysqld/mysqld.pid (yet?) after start: Too many levels of symbolic links

This problem will be seen on systems with the following configurations:


- CentOS 7.7 and OpenVZ/Virtuozzo with kernel prior to 2.6.32-042stab134.7

- CentOS 7.7 with kernel prior to 2.6.39

Your server is currently running the kernel `2.6.32-042stab113.21`

**Do not** reboot your server without updating the kernel first. Rebooting the server will likely cause further issues with systemd due to the outdated kernel.

I can see your server is running on OpenVZ/Virtuozzo:

# cat /var/cpanel/envtype
virtuozzo

This means that your host will need to update the kernel on the node and reboot to correct this issue. Or, if you have access to the host node you may be able to do this yourself.

Please understand this is an operating-system issue with the kernel and systemd, not an issue with cPanel. While the following page is a bug reported against Ubuntu, it is still relevant, and contains a response from OpenVZ developers about the issue:

Comment #18 : Bug #1804847 : Bugs : systemd package : Ubuntu

Dear All, my name is Vasily Averin, I'm maintainer of RHEL6-based OpenVz kernel. We have backported required patches and going to release updated kernel with support for openat(O_PATH|O_NOFOLLOW) for symlinks (internal bug id PSBM-90060) and fchownat for empty path (PSBM-89993). Fixed kernel...

bugs.launchpad.net

We also have further information about the issue in our forums post below:

[UPS-177] Service Failures after CentOS 7.7 update

Hello Everyone, We have received a number of reports regarding service failures with the error "Too many levels of symbolic links". This is an upstream issue involving the latest update to systemd included in CentOS 7.7. Technical Summary The recent CentOS 7.7 update includes an update to...

go.cpanel.net

If you have any more questions or issues on this subject, please let us know. I am grateful for your time and patience.

Thank you,


* * *

Jeremy
Technical Analyst
cPanel, L.L.C.

--------------------------------
This email is a service from cPanel.