Many cPanel Monitoring notices: services failed.

[virtualreality]

Hi cPanel Support,

Our server (run CentOS 7) keeps sending us the cPanel Monitoring notices:

Cron <[email protected]> /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check

Failed to get properties: Connection timed out

Service Name queueprocd
Service Status failed

Service Name tailwatchd
Service Status failed

Service Name mysql
Service Status failed

[server name] HANG: : chkservd

The chkservd process has become non-responsive.

The chkservd subprocess with PID “37196” ran for “10 minutes and 2 seconds”. The system terminated this sub-process when it exceeded the time allowed between checks, which is “5 minutes”. To determine why, check the “ /var/log/chkservd.log ” and “ /usr/local/cpanel/logs/tailwatchd_log ” files.

You likely received this notification as a symptom of a larger problem. If your server is experiencing a high load, we recommend that you investigate the cause. If you continue to receive this notification, it is likely that your system is unable to handle demand or there is a misconfiguration that delays restarts.

If you are sure that no misconfigurations exist, you should consider gradually increasing the following options in WHM’s “Tweak Settings” feature: “The number of times chkservd will allow a previous check to complete before terminating the check”, “The number of seconds between chkservd service checks”, or both.

Cron <[email protected]> /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check

Failed to get properties: Activation of org.freedesktop.systemd1 timed out

Now our WHM and cPanel (site) accounts pages fail to open. They are down, although we can login in our server via SSH.

We don't know what has caused all this.

We've tried to restart some services via SSH, such as:

systemctl restart queueprocd.service

systemctl restart tailwatchd.service

systemctl restart cpanel.service

However, all the restarts failed, and we've noticed the restart results all have this:

Failed to restart cpanel.service: Activation of org.freedesktop.systemd1 timed out

Failed to ..............................: Activation of org.freedesktop.systemd1 timed out

What does "Activation of org.freedesktop.systemd1 timed out" really mean here?

How to properly get org.freedesktop.systemd1 activated?

We've searched some search engines for the problem. Some people suggest to reboot server (such as: sudo systemctl --force --force reboot) to fix the problem, but we feel the forceful server reboot may cause some data loss in our server or our website database.

What is the correct way/best way to fix this problem?

Need your professional response and solution.

Thank you very much in advance for your support!

VR

 

[dalem]

Tutorial - Troubleshooting high server loads on Linux servers

 

[virtualreality]

Thank you dalem for help!

We think that the problem has been highly probably caused by a short period of time when there was an abrupt traffic surge or more probably there were heavy hacking hits to our website (Our website has got more hacking hits for over a year) and thus the server RAM was becoming inadequately low, making the problem happen since around 23:15PM, March 27.

Today, the problem still goes on. We've got tons of cPanel Monitoring notices saying some important services failed on our server, including MySQL server fails. In our previous post we forgot to mention that in WHM we've already set auto restarting MySQL server if MySQL server fails. Every time MySQL server fails it restarts and enables, so we can still browse our website.

For a long time as for today, from the newest cPanel Monitoring notice showing minutes ago we can see our server still has:

Available RAM 1.83 GB
Load Information 0.00 0.03 0.05.
IOStat Information
avg-cpu: %user %nice %system %iowait %steal %idle
1.29 0.03 0.25 0.00 0.00 98.43
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
ploop60838 6.79 164.28 13.36 3328556697 270637220

For most of the past hours, the server should have adequate RAM, and all those restarts should be successful.

As the problem has "Activation of org.freedesktop.systemd1 timed out" involved, the problem may be partly spurred by some CentOS 7 bug, that may also cause "Activation of org.freedesktop.systemd1 timed out"?

During all these days, we have never touched, never changed anything in the server. And our Apache server and MySQL server with our long-time stable, unchanged server settings have been running smoothly and stably for years.

We only keep our WHM/cPanel updating patches once a week.

This is the first time we've come across with this problem happening all of sudden. Previously, we never saw and never knew "Activation of org.freedesktop.systemd1 timed out".

Need more guides and exact fixing please!

Thank you very much in advance!

 

[virtualreality]

We have submitted the Technical Support Ticket (ID: 11796131) to the cPanel Customer Technical Support.

Thank you very much for support!

 

[cPanelLauren]

Hello @virtualreality

I checked in on that ticket and it appears we weren't able to access the server but that the hosting provider you use was able to resolve the issue. If you're able to, please let us know what was done in order to resolve this.


Thanks!

 

[northbridge]

Hi,

We have quite a major issue resulting to crash various services like sshd , nscd, pop, queueprocd, tailwatchd and more.

The issue appeared unexpectedtly.

The errors is:

Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out

[[email protected] ~]# journalctl -xn
-- Logs begin at Wed 2019-09-18 16:41:44 CEST, end at Fri 2019-11-15 22:43:47 CET. --
Nov 15 22:42:26 host1 CROND[7968]: (root) CMD (/etc/nginx/utilities/https_vhosts.sh >> /dev/null 2>&1)
Nov 15 22:42:31 host1 systemd-logind[2921]: Failed to start session scope session-36906.scope: Connection timed out
Nov 15 22:42:31 host1 dbus[2948]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Nov 15 22:42:56 host1 dbus[2948]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Nov 15 22:43:22 host1 dbus[2948]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Nov 15 22:43:22 host1 systemd-logind[2921]: Failed to start user slice user-0.slice, ignoring: Failed to activate service 'org.freedesktop.systemd1': timed out (org.fr
Nov 15 22:43:26 host1 crond[8027]: pam_systemd(crond:session): Failed to create session: Connection timed out
Nov 15 22:43:26 host1 CROND[8065]: (root) CMD (/etc/nginx/utilities/https_vhosts.sh >> /dev/null 2>&1)
Nov 15 22:43:47 host1 dbus[2948]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Nov 15 22:43:47 host1 systemd-logind[2921]: Failed to start session scope session-36907.scope: Connection timed out
lines 1-11/11 (END)


Could you help us to resolve the issues?

Thank you!
Sotiris

 

[cPanelLauren]

Hi,

We have quite a major issue resulting to crash various services like sshd , nscd, pop, queueprocd, tailwatchd and more.

The issue appeared unexpectedtly.

The errors is:

Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out

[[email protected] ~]# journalctl -xn
-- Logs begin at Wed 2019-09-18 16:41:44 CEST, end at Fri 2019-11-15 22:43:47 CET. --
Nov 15 22:42:26 host1 CROND[7968]: (root) CMD (/etc/nginx/utilities/https_vhosts.sh >> /dev/null 2>&1)
Nov 15 22:42:31 host1 systemd-logind[2921]: Failed to start session scope session-36906.scope: Connection timed out
Nov 15 22:42:31 host1 dbus[2948]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Nov 15 22:42:56 host1 dbus[2948]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Nov 15 22:43:22 host1 dbus[2948]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Nov 15 22:43:22 host1 systemd-logind[2921]: Failed to start user slice user-0.slice, ignoring: Failed to activate service 'org.freedesktop.systemd1': timed out (org.fr
Nov 15 22:43:26 host1 crond[8027]: pam_systemd(crond:session): Failed to create session: Connection timed out
Nov 15 22:43:26 host1 CROND[8065]: (root) CMD (/etc/nginx/utilities/https_vhosts.sh >> /dev/null 2>&1)
Nov 15 22:43:47 host1 dbus[2948]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Nov 15 22:43:47 host1 systemd-logind[2921]: Failed to start session scope session-36907.scope: Connection timed out
lines 1-11/11 (END)


Could you help us to resolve the issues?

Thank you!
Sotiris

Hello,

I'd suggest opening a ticket if you haven't done so already and are still experiencing issues. You can do so using the link in my signature.

Thanks!