SOLVED [CPANEL-24261] find_outdated_services performs soft restarts

[Michael-Inet]

Can't restart cpsrvd (on VPS, CENTOS 6.9 virtuozzo v68.0.29).

New Security Advisor notifications with High importance
Detected 1 process that is running outdated executables: 16076 You must take one of the following actions to ensure the system is up-to-date:

Solutions in this thread don't work:

SOLVED - Manually start services shown by /bin/needs-restarting

root@srv04 [~]# /scripts/restartsrv_cpsrvd
Waiting for "cpsrvd" to restart gracefully ??waiting for "cpsrvd" to initialize ???finished.

Service Status
        cpanel (cpsrvd (SSL) - dormant mode - accepting connections) is running as root with PID 16076 (pidfile+/proc check method).

Startup Log
        License is valid and has already updated recently.
        Starting PID 16076: /usr/local/cpanel/libexec/cpsrvd-dormant

cpsrvd restarted successfully.

root@srv04 [~]# /bin/needs-restarting
-bash: /bin/needs-restarting: No such file or directory

root@srv04 [~]# systemctl restart cpanel
-bash: systemctl: command not found

WHM » Security Center » Security Advisor

Detected 1 process that is running outdated executables: 16076

  You must take one of the following actions to ensure the system is up-to-date:

    Restart the listed process.
    Reboot the server.

Restart the listed process has no link.

# # #

Okay, what do?

 

[cPanelMichael]

Hello,

Can you let us know the output from the "/scripts/find_outdated_services" command on this system?

Thank you.

 

[Michael-Inet]

Hi Michael,

Sure!

root@srv04 [~]# /scripts/find_outdated_services
Looking for outdated services …
The system did not find any outdated services.

So, there aren't any? Uhm, well, maybe there are? I re-scaned in WHM's cPanel Security Advisor.

Start Time: Wed Feb 14 2018 20:27:15 GMT-0600 (CST)
Version: 1.04
Finish Time: Wed Feb 14 2018 20:27:28 GMT-0600 (CST)

...
Detected 1 process that is running outdated executables: 16076
...

And if I'm reading ps right, it's been running since Jan30?

root@srv04 [~]# ps -p 16076 -o pid,vsz=MEMORY -o user,group=GROUP -o comm,args=ARGS
  PID MEMORY USER     GROUP    COMMAND         ARGS
16076  25848 root     root     cpsrvd (SSL) -  cpsrvd (SSL) - dormant mode - accepting connections

root@srv04 [~]# ps -p 16076 -F
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root     16076     1  0 25334 24476   2 Jan30 ?        00:06:13 cpsrvd (SSL) - waiting for connections

What I found odd originally is that the PID stayed the same when running restartsrv_cpsrvd, but then I don't know enough about cPanel to know if that's normal.

I'd rather be able to re-start this, as I can't really reboot except around 2am Sat or Sun.

Anything else you want me to look at?

Best,
Michael

 

[cPanelMichael]

Hello,

Can you try killing that process and then restarting cPanel again? EX:

kill -9 16076
/scripts/restartsrv_cpsrvd

Let us know if that helps.

Thank you.

 

[Michael-Inet]

That did it! Now doesn't show up in cPanel Security Advisor and the PID changed too

Thanks Michael!

Best,
Michael

PS: Since the prior thread is Google's favorite, would you add the kill as a last resort to it? And maybe a "pgrep -l cpsrvd" so they know they have the correct PID to kill? Thanks tons, m..

root@srv04 [~]# kill -9 16076

root@srv04 [~]# ps -p 16076 -F
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD

root@srv04 [~]# /scripts/restartsrv_cpsrvd
Waiting for “cpsrvd” to start ……waiting for “cpsrvd” to initialize ………finished.

Service Status
        cpanel (cpsrvd (SSL) - dormant mode - accepting connections) is running as root with PID 397 (pidfile+/proc check method).

Startup Log
        Starting PID 397: /usr/local/cpanel/libexec/cpsrvd-dormant
        License is valid and has already updated recently.

cpsrvd started successfully.

root@srv04 [~]# pgrep -l cpsrvd
397 cpsrvd (SSL) -

 

[cPanelMichael]

Hello,

I'm glad to see that helped. I've updated the response in the linked thread as well.

Thanks!

 

[Michael-Inet]

Follow up.

cpsrvd not restarting did this same thing on another server of mine (dedicated, not vps). Which seems odd, hence the follow up.

If it's random chance, no biggie, but if multiple people are having the issue, should I file a bug?

Code that 'fixed' it below.

Best,
Michael

Edit: PID 3548 is the PID in the email stating an outdated executable. Didn't really need to pgrep that

root@srv01 [~]# ll /bin/needs-restarting
ls: cannot access /bin/needs-restarting: No such file or directory

root@srv01 [~]# pgrep cpsrvd
3548

root@srv01 [~]# ps -p 3548 -F
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root      3548     1  0 25269 24424   6 Feb19 ?        00:01:26 cpsrvd (SSL) - waiting for connections

root@srv01 [~]# /scripts/find_outdated_services
Looking for outdated services …
The system did not find any outdated services.

root@srv01 [~]# /scripts/restartsrv_cpsrvd
Waiting for “cpsrvd” to restart gracefully ……failed.

Waiting for “cpsrvd” to restart ………waiting for “cpsrvd” to initialize ………finished.

Service Status
        cpanel (cpsrvd (SSL) - dormant mode - accepting connections) is running as root with PID 24768 (pidfile+/proc check method).

Startup Log
        License is valid and has already updated recently.
        Starting PID 24768: /usr/local/cpanel/libexec/cpsrvd-dormant

cpsrvd restarted successfully.

root@srv01 [~]# ps -p 3548 -F
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD

root@srv01 [~]# ps -p 24768 -F
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root     24768     1  0 49482 24504   1 09:22 ?        00:00:00 cpsrvd (SSL) - waiting for connections

 

[cPanelMichael]

Hello,

Feel free to open a support ticket using the link in my signature if you'd like us to take a closer look to rule out any issues with the product.

Thank you.

 

[Michael-Inet]

Okay, this no longer works? I've got 2 VPSes that won't restart cpsrvd or dnsadmin using the above (/scripts/restartsrv_cpsrvd, /scripts/restartsrv_dnsadmin).

Both scripts claim to work, but show the original PID in the output (Starting PID 28396: /usr/local/cpanel/libexec/cpsrvd-dormant)

I had to physically kill the PID under root to get them to restart...

kill -9 $PID

Edit:
For the truly paranoid (me!) run this after the restartsrv* commands to double verify you've got a new PID.

ps aux|grep cpsrvd
ps aux|grep dnsadmin

# # #

Also, what/where is the /bin/needs-restarting script called now?

If it is /scripts/find_outdated_services , that script is broken and reports "The system did not find any outdated services." while WHM » Security Center » Security Advisor shows the proper output.

Best,
Michael

 

[cPanelMichael]

Also, what/where is the /bin/needs-restarting script called now?

If it is /scripts/find_outdated_services , that script is broken and reports "The system did not find any outdated services." while WHM » Security Center » Security Advisor shows the proper output.

This seems like a separate issue that requires it's own investigation. Can you create a new thread for this topic so we can take a closer look?

Both scripts claim to work, but show the original PID in the output (Starting PID 28396: /usr/local/cpanel/libexec/cpsrvd-dormant)

Soft restarts are performed by default. You'll need to append "--restart --hard" to the restart command if you want to force a reboot of the service with a new PID. EX:

/scripts/restartsrv_cpsrvd --restart --hard
/scripts/restartsrv_dnsadmin --restart --hard

Let me know if this helps.

Thank you.

 

[Michael-Inet]

This seems like a separate issue that requires it's own investigation. Can you create a new thread for this topic so we can take a closer look?

Done: New Thread - /scripts/find_outdated_services broken?

Soft restarts are performed by default. You'll need to append "--restart --hard" to the restart command if you want to force a reboot of the service with a new PID. EX:

/scripts/restartsrv_cpsrvd --restart --hard
/scripts/restartsrv_dnsadmin --reboot --hard

I know this isn’t your doing, I’m just venting my frustration, but what @&&^))# changed that? Wheee, lets have a script called restart, not actually restart something, nor even check if the object it owns is out of date?!?!? Uhg!! Sorry, Anyway...

As opposed to trying to keep these type threads updated, as there are 50+ of the restart scripts and having to guess which ones use --restart and which uses --reboot seems a bit problematic... (Or that’s a typo? But still who knows when it’ll change next???)

Would you please dig up the official cPanel documentation for restarting services? I’ll update my posts with that link so people aren’t reading, and getting frustrated by, solutions that no longer work.

Thanks Michael!

Best,
Michael

 

[cPanelMichael]

Hi Michael,

I believe the better approach here is to address the problem at the source of the issue. I've opened internal case CPANEL-24261 to report the behavior where /scripts/find_outdated_services performs soft restarts on services such as cpsrvd and tailwatchd by default. Thus, using the restart option included with the /scripts/find_outdated_services script is ineffective for these services. I'll monitor this case and update this thread with more information as it becomes available.

As opposed to trying to keep these type threads updated, as there are 50+ of the restart scripts and having to guess which ones use --restart and which uses --reboot seems a bit problematic... (Or that’s a typo? But still who knows when it’ll change next???)

The "reboot" flag was simply a typo. I've updated that post to reflect "restart" instead of "reboot".

Thank you.

 

[Michael-Inet]

I believe the better approach here is to address the problem at the source of the issue. I've opened internal case CPANEL-24261 ...

Before I get distracted once again, Happy whatever you celebrate at this time of year May next year be wonderful for you!

And yes, that is a much better approach, so thank you for that as well.

For anyone else reading this, the '--restart --hard' does work, as I just needed it on a different server.

Best All, Thanks Michael,
Michael

 

[cPanelMichael]

Hi @Michael-Inet,

Likewise!

Internal case CPANEL-24261 is still open at this time, but I do see it was noted that hard restarts are not performed by default in order to avoid service downtime (it's important that we avoid downtime whenever possible, even if it's just a couple of seconds). Thus, the case will aim to address the issue in the detection aspect of the script as opposed to modifying the script to perform hard restarts by default. I'll update this thread again as soon as more information is available.

Thank you.

 

[cPanelMichael]

Hello,

To update, we're aiming to publish a solution for this issue to cPaenl & WHM version 80 before it reaches the CURRENT release tier. I'll update this thread with more information as soon as it's available.

Thank you.

 

[cPanelMichael]

Hello Everyone,

This is fixed in cPanel & WHM version 80 (build 80.0.9 was published to the RELEASE tier earlier today):

Fixed case CPANEL-24261: Fix issues which prevents proper detection of hot service restarts, causing services to be marked as needing restart when they have already been successfully restarted.

Let us know if you have any questions.

Thanks!