Getting regular update failure in upcp script emails

[cseufert]

I am running AlmaLinux 8 + cloudlinux and 3~5 days a week getting cPanel & WHM update failure in upcp script emails from our whm.

It is stating the reason as:
Maintenance ended; however, it did not exit cleanly (256). The following events were logged: “scripts/find_outdated_services”. Review the update logs to determine why the update failed.

Manually Running find_outdated_services scripts results in failure more often that success with the following error, which looks to be an error caused be a process terminating before the needs-restarting command completes, resuling in the folliwng error:

STDERR: Traceback (most recent call last):
  File "/usr/bin/needs-restarting", line 101, in <module>
    main.user_main(MAPPING[command] + args, exit_code=True)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 201, in user_main
    errcode = main(args)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 122, in cli_run
    cli.run()
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 1055, in run
    return self.command.run()
  File "/usr/lib/python3.6/site-packages/dnf-plugins/needs_restarting.py", line 278, in run
    names = set([get_service_dbus(pid) for pid in sorted(stale_pids)])
  File "/usr/lib/python3.6/site-packages/dnf-plugins/needs_restarting.py", line 278, in <listcomp>
    names = set([get_service_dbus(pid) for pid in sorted(stale_pids)])
  File "/usr/lib/python3.6/site-packages/dnf-plugins/needs_restarting.py", line 142, in get_service_dbus
    systemd_manager_interface.GetUnitByPID(pid)
  File "/usr/lib64/python3.6/site-packages/dbus/proxies.py", line 70, in __call__
    return self._proxy_method(*args, **keywords)
  File "/usr/lib64/python3.6/site-packages/dbus/proxies.py", line 145, in __call__
    **keywords)
  File "/usr/lib64/python3.6/site-packages/dbus/connection.py", line 651, in call_blocking
    message, timeout)
dbus.exceptions.DBusException: org.freedesktop.systemd1.NoUnitForPID: PID 753212 does not belong to any loaded unit.


Cpanel::Exception::ProcessFailed::Error/(XID ud4fta) “/usr/bin/needs-restarting” reported error code “1” when it ended: Traceback (most recent call last):
  File "/usr/bin/needs-restarting", line 101, in <module>
    main.user_main(MAPPING[command] + args, exit_code=True)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 201, in user_main
    errcode = main(args)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 122, in cli_run
    cli.run()
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 1055, in run
    return self.command.run()
  File "/usr/lib/python3.6/site-packages/dnf-plugins/needs_restarting.py", line 278, in run
    names = set([get_service_dbus(pid) for pid in sorted(stale_pids)])
  File "/usr/lib/python3.6/site-packages/dnf-plugins/needs_restarting.py", line 278, in <listcomp>
    names = set([get_service_dbus(pid) for pid in sorted(stale_pids)])
  File "/usr/lib/python3.6/site-packages/dnf-plugins/needs_restarting.py", line 142, in get_service_dbus
    systemd_manager_interface.GetUnitByPID(pid)
  File "/usr/lib64/python3.6/site-packages/dbus/proxies.py", line 70, in __call__
    return self._proxy_method(*args, **keywords)
  File "/usr/lib64/python3.6/site-packages/dbus/proxies.py", line 145, in __call__
    **keywords)
  File "/usr/lib64/python3.6/site-packages/dbus/connection.py", line 651, in call_blocking
    message, timeout)
dbus.exceptions.DBusException: org.freedesktop.systemd1.NoUnitForPID: PID 753212 does not belong to any loaded unit.

at /usr/local/cpanel/Cpanel/ChildErrorStringifier.pm line 161.
        Cpanel::ChildErrorStringifier::to_exception(Cpanel::SafeRun::Object=HASH(0x2773518)) called at /usr/local/cpanel/Cpanel/ChildErrorStringifier.pm line 140
        Cpanel::ChildErrorStringifier::die_if_error(Cpanel::SafeRun::Object=HASH(0x2773518)) called at /usr/local/cpanel/Cpanel/SafeRun/Object.pm line 681
        Cpanel::SafeRun::Object::die_if_error(Cpanel::SafeRun::Object=HASH(0x2773518)) called at /usr/local/cpanel/Cpanel/ProcessCheck/Outdated.pm line 274
        Cpanel::ProcessCheck::Outdated::_outdated_services_default() called at /usr/local/cpanel/Cpanel/ProcessCheck/Outdated.pm line 68
        Cpanel::ProcessCheck::Outdated::outdated_services() called at /scripts/find_outdated_services line 87
        scripts::find_outdated_services::__ANON__() called at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/Try/Tiny.pm line 96
        eval {...} called at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/Try/Tiny.pm line 91
        Try::Tiny::try(CODE(0x1e8f2d8), Try::Tiny::Catch=REF(0x1a927d0)) called at /scripts/find_outdated_services line 96
        scripts::find_outdated_services::run(scripts::find_outdated_services=HASH(0x1a5ab18)) called at /scripts/find_outdated_services line 73

 

[andrew.n]

Can you paste here the full error which you get when you run the script?

 

[cseufert]

I have narrowed it down to running

# needs-restarting

which randomly fails reading a path like

/proc/xxxx/cmdline

So I guess it could be a bug in DNF, but unsure who to go about solving it, as its intermittant, and running needs-restarting manually can work fine somtimes, and fail others.

 

[Rachel S]

"Try to run the following command and verify:

$ sudo cat /etc/cpupdate.conf
$ sudo cat /usr/local/cpanel/version

$ sudo cat /var/cpanel/updatelogs/update.$$$.log
>>In above command you have to replace the ""update.$$$.log"" with the most recent update log, indicating that the update failed.

Also, try to run the following command and verify:

$ sudo /usr/local/cpanel/scripts/find_outdated_services

Now, try to rebuild the RPM database to check if it works:
""WHM Home » Software » Rebuild RPM Database""

Lastly, you can run the following command manually to check if it still continues to fail, and verify the message for the failure.

$ sudo /usr/local/cpanel/scripts/post_sync_cleanup

If this does not work, you need to force another cPanel update with the following command

""/scripts/upcp --force""

Still, if that doesn't work, then the last option

Run the following command:

$ sudo yum clean all
$ sudo yum update

Just for information:

$ sudo yum clean all :-

The command ""yum clean"" clears the cache of repositories that have been enabled in /etc/yum. repos. In the commands below, ""all files"" refers to ""all files in presently enabled repositories."" Use the —enablerepo='*' option if you wish to clean any (temporarily) disabled repositories as well.


$ sudo yum update

""yum update"" brings all currently installed packages up to date with the latest versions available in the repository."

 

[cPRex]

I would second the "yum update" option since these errors seem to be related to dnf. I would run this and see if that helps the situation:

dnf clean all && dnf makecache
yum update