Service notifications showing down but is not.

[Madmaxdallas]

I am receiving notifications of services down, but I check they are up, also the service
monitor page shows pending.

Here is output from:

/usr/local/cpanel/bin/whmapi1 installed_versions packages=0 |egrep 'operating_system_name|operating_system_version|cpanel_and_whm'


cpanel_and_whm: 11.90.0.10
  operating_system_name: centos
  operating_system_version: '7.8'

    Server    ns1sf1.foodhost.net
Primary IP Address    138.68.41.125
Service Name    cpanellogd
Service Status    failed ⛔
Notification    The service “cpanellogd” appears to be down.
Service Check Method    The system’s command to check or to restart this service failed.
Number of Restart Attempts    1
Service Check Raw Output    (XID h32j6t) The “cpanellogd” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_cpanellogd” reported error number 255 when it ended.
Startup Log    Oct 01 19:29:28 ns1sf1.foodhost.net systemd[1]: Starting cPanel Log services...
Oct 01 19:29:30 ns1sf1.foodhost.net cpanellogd[1796]: ==> cPanel Log Daemon version 26.0
Oct 01 19:29:30 ns1sf1.foodhost.net systemd[1]: Started cPanel Log services.
Oct 01 20:18:01 ns1sf1.foodhost.net systemd[1]: Stopping cPanel Log services...
Oct 01 20:18:01 ns1sf1.foodhost.net systemd[1]: Stopped cPanel Log services.
Memory Information    Used    2.15 GB
Available    1.55 GB
Installed    3.7 GB

Load Information    2.05 0.57 0.19


---------------------------
Restart of cpanellogd

[[email protected] panel]# /usr/local/cpanel/scripts/restartsrv_cpanellogd
Waiting for “cpanellogd” to restart ………waiting for “cpanellogd” to initialize ………finished.

Service Status
        cpanellogd (cpanellogd - master) is running as root with PID 5531 (systemd+/proc check method).

Startup Log
        Oct 01 20:42:53 ns1sf1.foodhost.net systemd[1]: Starting cPanel Log services...
        Oct 01 20:42:53 ns1sf1.foodhost.net cpanellogd[5530]: ==> cPanel Log Daemon version 26.0
        Oct 01 20:42:53 ns1sf1.foodhost.net systemd[1]: Started cPanel Log services.

Log Messages
        Oct  1 20:42:53 ns1sf1 cpanellogd: ==> cPanel Log Daemon version 26.0

cpanellogd restarted successfully.
[[email protected] panel]#

---------------------

I also get report of IMAP, SQL etc being down but are not, I feel it may be a result of OS update or WMM.
This server been running virtual for years.

I tried to open a ticket, and also granted access but when it got down to the last few steps it failed an I gave up.

Thanks,
MMD

 

[andrew.n]

Looks like cpanellogd is failing for some reason. What does

service cpanellogd status

shows?

Have you checked the logs under /usr/local/cpanel/logs yet?

 

[Madmaxdallas]

[[email protected] panel]# service cpanellogd status
Redirecting to /bin/systemctl status cpanellogd.service
● cpanellogd.service - cPanel Log services
Loaded: loaded (/etc/systemd/system/cpanellogd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-10-02 06:18:31 CDT; 4min 13s ago
Process: 1779 ExecStart=/usr/local/cpanel/cpanellogd (code=exited, status=0/SUCCESS)
Main PID: 1944 (cpanellogd - sl)
CGroup: /system.slice/cpanellogd.service
└─1944 cpanellogd - sleeping for logs

Oct 02 06:18:29 ns1sf1.foodhost.net systemd[1]: Starting cPanel Log services...
Oct 02 06:18:30 ns1sf1.foodhost.net cpanellogd[1779]: ==> cPanel Log Daemon v...
Oct 02 06:18:31 ns1sf1.foodhost.net systemd[1]: Started cPanel Log services.
Hint: Some lines were ellipsized, use -l to show in full.
[[email protected] panel]#



---------------------------
[[email protected] logs]# tail tailwatchd_log
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Restored /var/log/exim_mainlog to position 6546909
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Will resume /var/log/maillog to 5300244
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Reading back thirty lines of /var/log/maillog starting at 5283860
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Restoring /var/log/maillog to catch up position 5300244
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Restored /var/log/maillog to position 5300244
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Will resume /etc/apache2/logs/modsec_audit.log to 0
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Reading back thirty lines starting at 0 (small file)
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Restoring /etc/apache2/logs/modsec_audit.log to catch up position 0
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch] [INFO] Restored /etc/apache2/logs/modsec_audit.log to position 0
[1963] [2020-10-02 06:19:01 -0500] [Cpanel::TailWatch::ChkServd] [INFO] Opening /var/log/chkservd.log in append mode
[[email protected] logs]#


---
Noticed this in one of the logs it was trying to check on a non routable address. I didnt add this to the system but deleted it. Maybe somehow digital ocean added it. Not sure at all.

eth0	<IPAddress>	Main/shared IP for: names deleted for post. Nameserver: ns1.foodhost.net
eth0	10.46.0.5			Delete

 

[andrew.n]

I'm not really see any issue here. The services still fails? You can find cPanel certified system administrators at System Administration Services if you want a professional to have a closer look for you or maybe somebody from cPanel (@cPanelLauren) can advise further here.

 

[cPanelLauren]

I'd wager that the service notifications are valid when they're sent. By the time you log in, you're seeing the service back up because chksrvd has already restarted them. The service performs checks on services you have set to monitored every 5 minutes and attempts to restart them. If they don't restart within chksrvd's timeout for this then the notice is sent. Even in the cpanellogd output it's only been running for 4 minutes.

Loaded: loaded (/etc/systemd/system/cpanellogd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-10-02 06:18:31 CDT; 4min 13s ago

Looking at the times for restart in the notice in your original response it looks like a long time goes by before it's restarted:

Oct 01 20:18:01 ns1sf1.foodhost.net systemd[1]: Stopping cPanel Log services...
Oct 01 20:18:01 ns1sf1.foodhost.net systemd[1]: Stopped cPanel Log services.

then:

   Oct 01 20:42:53 ns1sf1.foodhost.net systemd[1]: Starting cPanel Log services...
        Oct 01 20:42:53 ns1sf1.foodhost.net cpanellogd[5530]: ==> cPanel Log Daemon version 26.0
        Oct 01 20:42:53 ns1sf1.foodhost.net systemd[1]: Started cPanel Log services.

24 minutes between when it was stopped and when it did start again.

I also see that at that specific point the load average jumped:

Load Information    2.05 0.57 0.19

What, if anything is noted in /var/log/messages around Oct 01 20:18:01