cpanllogd won't restart

[maestroc]

My server was down for about eight hours yesterday. Since it came back up I have been receiving messages every few minutes that say:


The subprocess “/usr/local/cpanel/scripts/restartsrv_cpanellogd” reported error number 255 when it ended.
Nov 10 04:28:10 vps systemd[1]: Starting cPanel Log services...
Nov 10 04:28:10 vps cpanellogd[2208]: ==> cPanel Log Daemon version 26.0
Nov 10 04:28:10 vps systemd[1]: Can't open PID file /var/run/cpanellogd.pid (yet?) after start: Too many levels of symbolic links
Nov 10 04:29:40 vps systemd[1]: cpanellogd.service start operation timed out. Terminating.
Nov 10 04:29:40 vps systemd[1]: Failed to start cPanel Log services.
Nov 10 04:29:40 vps systemd[1]: Unit cpanellogd.service entered failed state.
Nov 10 04:29:40 vps systemd[1]: cpanellogd.service failed.

Nov 10 12:36:33 vps systemd[1]: Starting cPanel Log services...
Nov 10 12:36:33 vps cpanellogd[3636]: ==> cPanel Log Daemon version 26.0
Nov 10 12:36:33 vps systemd[1]: Can't open PID file /var/run/cpanellogd.pid (yet?) after start: Too many levels of symbolic links
Nov 10 12:38:03 vps systemd[1]: cpanellogd.service start operation timed out. Terminating.
Nov 10 12:38:03 vps systemd[1]: Failed to start cPanel Log services.
Nov 10 12:38:03 vps systemd[1]: Unit cpanellogd.service entered failed state.
Nov 10 12:38:03 vps systemd[1]: cpanellogd.service failed.

If I try systemctl restart cpanellogd it does not restart.

I am having the same problem with another service (osmd) which I know is not a cpanel service but the error message is similar:

ConfigServer Outgoing Spam Monitor daemon starting - logging to /var/log/osmd.log
Nov 10 12:04:40 vps systemd[1]: Can't open PID file /var/run/osmd.pid (yet?) after start: Too many levels of symbolic links

Any idea what I should do to fix these issues?

 

[cPanelAnthony]

Hello! This error typically only occurs when there is a recursive symbolic link configured. Can you provide me with the output of the following commands?

ls -la /var/|grep run
cd /run ; find -L ./ -type l
df -h

Please censor any hostnames or information that could trace your server back to us.

 

[maestroc]

Here is the output. Thank you for your help!

[[email protected] ~]# ls -la /var/|grep run
lrwxrwxrwx 1 root root 11 Aug 31 2017 lock -> ../run/lock
lrwxrwxrwx 1 root root 6 Aug 31 2017 run -> ../run
[[email protected] ~]# cd /run ; find -L ./ -type l
[[email protected] run]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/simfs 325G 200G 126G 62% /
devtmpfs 4.0G 0 4.0G 0% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 50M 4.0G 2% /run
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup

 

[cPanelAnthony]

I'm not seeing any recursive linking in place with this setup.

I also saw that this issue can be caused by the server kernel not support the O_PATH flag with open(). This can cause service failures similar to the ones you are experiencing. Could you provide me the output from these commands to confirm?

perl -E 'our $p = "/proc/self"; $! = 0;$fd = syscall 2, $p, 0x2e0000, 0x7fff98b7c438; ($!) ? say $! : say "Not UPS-177"'

uname -a

 

[maestroc]

[[email protected] ~]# perl -E 'our $p = "/proc/self"; $! = 0;$fd = syscall 2, $p, 0x2e0000, 0x7fff98b7c438; ($!) ? say $! : say "Not UPS-177"'
Too many levels of symbolic links
[[email protected] ~]# uname -a
Linux vps 2.6.32-042stab128.2 #1 SMP Thu Mar 22 10:58:36 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux

 

[cPanelAnthony]

Thanks! It looks to me like your kernel is pretty outdated; the time stamp shows March of 2018. The recommended solution to this behavior is to update the server kernel to a more recent version. The following article explains this in greater detail and also provides a workaround in case you are not able to upgrade the server kernel:

Service failures and outdated kernels

 

[maestroc]

It sounds like they run kernel care but they will not upgrade the base kernel. Short of migrating all 50+ accounts to another VPS do I have any other options? Is there no other way to clear this error?

 

[cPanelAnthony]

We would generally suggest a migration in situations like this; I do apologize for that inconvenience. The following article offers some info.

Too many levels of symbolic links

 

[maestroc]

Will this cpanel provided workaround be applicable to this case? If I am reading correctly I believe my kernel matches the description given on this related post:

[UPS-177] Service Failures after CentOS 7.7 update

Hello Everyone, We have received a number of reports regarding service failures with the error "Too many levels of symbolic links". This is an upstream issue involving the latest update to systemd included in CentOS 7.7. Technical Summary The recent CentOS 7.7 update includes an update to...

forums.cpanel.net

 

[cPanelAnthony]

Will this cpanel provided workaround be applicable to this case? If I am reading correctly I believe my kernel matches the description given on this related post:

[UPS-177] Service Failures after CentOS 7.7 update

Hello Everyone, We have received a number of reports regarding service failures with the error "Too many levels of symbolic links". This is an upstream issue involving the latest update to systemd included in CentOS 7.7. Technical Summary The recent CentOS 7.7 update includes an update to...

forums.cpanel.net

Just to confirm, are you on CentOS 7.7 as described h ere?

 

[maestroc]

My kernel is this one:

Linux vps 2.6.32-042stab128.2 #1 SMP Thu Mar 22 10:58:36 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux

In the other post it said the fix was applicable to this:

CentOS 7.7 and OpenVZ/Virtuozzo with kernels older than 2.6.32-042stab134.7

In any case I already tried it last night and it seemed to work perfectly. All services are back up and running. Eventually I am going to have to migrate but at least now I feel like I have some wiggle room on doing so.

 

[cPanelAnthony]

Thanks for this confirmation! I am happy to see everything is working.