SOLVED [CPANEL-23845] cpsrvd fails after 76 upgrade when running a custom Apache service

[inteldigital]

At around 0700 hours this morning, we were alerted to one of our redundant DNSOnly servers had failed, it has been trying to restart itself for the past 9 hours and we have tried everything to get it restarted but nothing is working.

On running the command "/scripts/restartsrv_cpsrvd" we get the following response:

Cpanel::Exception::Services::StartError
Service Status

Service Error
    (XID 44mh23) The “cpsrvd” service failed to start.

Startup Log
    Nov 08 14:39:08 jade.example.com restartsrv_cpsrvd[18140]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....
    Nov 08 14:39:09 jade.example.com restartsrv_cpsrvd[18140]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....
    Nov 08 14:39:10 jade.example.com restartsrv_cpsrvd[18140]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....
    Nov 08 14:39:11 jade.example.com restartsrv_cpsrvd[18140]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....
    Nov 08 14:39:12 jade.example.com restartsrv_cpsrvd[18140]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....
    Nov 08 14:39:13 jade.example.com restartsrv_cpsrvd[18140]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....
    Nov 08 14:39:13 jade.example.com systemd[1]: cpanel.service start operation timed out. Terminating.
    Nov 08 14:39:13 jade.example.com systemd[1]: Failed to start cPanel services.
    Nov 08 14:39:13 jade.example.com systemd[1]: Unit cpanel.service entered failed state.
    Nov 08 14:39:13 jade.example.com systemd[1]: cpanel.service failed.

Our message log is filled with stuff like this:

Nov  8 01:00:06 jade systemd: Stopping User Slice of nmonagent.
Nov  8 01:00:23 jade systemd: Stopping tailwatchd...
Nov  8 01:00:24 jade restartsrv_tailwatchd: [7708] [2018-11-08 01:00:24 +0000] [main] Current process '6671' stopped
Nov  8 01:00:24 jade systemd: Stopped tailwatchd.
Nov  8 01:00:24 jade systemd: Starting tailwatchd...
Nov  8 01:00:24 jade restartsrv_tailwatchd: [Thu Nov  8 01:00:24 2018] Starting /usr/local/cpanel/libexec/tailwatch/tailwatchd daemon
Nov  8 01:00:24 jade restartsrv_tailwatchd: Log is at /usr/local/cpanel/logs/tailwatchd_log
Nov  8 01:00:24 jade systemd: PID file /var/run/tailwatchd.pid not readable (yet?) after start.
Nov  8 01:00:24 jade systemd: Started tailwatchd.
Nov  8 01:00:40 jade systemd: Reloading.
Nov  8 01:01:01 jade systemd: Started Session 211372 of user root.
Nov  8 01:01:01 jade systemd: Starting Session 211372 of user root.
Nov  8 01:01:01 jade systemd: Created slice User Slice of nmonagent.
Nov  8 01:01:01 jade systemd: Starting User Slice of nmonagent.
Nov  8 01:01:01 jade systemd: Started Session 211371 of user nmonagent.
Nov  8 01:01:01 jade systemd: Starting Session 211371 of user nmonagent.
Nov  8 01:01:06 jade systemd: Removed slice User Slice of nmonagent.
Nov  8 01:01:06 jade systemd: Stopping User Slice of nmonagent.
Nov  8 01:01:25 jade systemd: Reloading.
Nov  8 01:01:43 jade systemd: Reloading.
Nov  8 01:01:43 jade systemd: Stopping PowerDNS Authoritative Server...
Nov  8 01:01:43 jade systemd: Starting PowerDNS Authoritative Server...
Nov  8 01:01:43 jade pdns_server: Reading random entropy from '/dev/urandom'
Nov  8 01:01:43 jade pdns_server: Loading '/usr/lib64/pdns/libbindbackend.so'
Nov  8 01:01:43 jade pdns_server: This is a standalone pdns
Nov  8 01:01:43 jade pdns_server: Listening on controlsocket in '/var/run/pdns.controlsocket'
Nov  8 01:01:43 jade pdns_server: UDP server bound to 0.0.0.0:53
Nov  8 01:01:43 jade pdns_server: UDPv6 server bound to [::]:53
Nov  8 01:01:43 jade pdns_server: TCP server bound to 0.0.0.0:53
Nov  8 01:01:43 jade pdns_server: TCPv6 server bound to [::]:53
Nov  8 01:01:43 jade pdns_server: PowerDNS Authoritative Server 4.1.4 (C) 2001-2018 PowerDNS.COM BV
Nov  8 01:01:43 jade pdns_server: Using 64-bits mode. Built using gcc 4.8.2 20140120 (Red Hat 4.8.2-16) on Oct 29 2018 15:56:48 by [email protected]
Nov  8 01:01:43 jade pdns_server: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Nov  8 01:01:43 jade pdns_server: Creating backend connection for TCP
Nov  8 01:01:43 jade pdns_server: [bindbackend] Parsing 227 domain(s), will report when done
Nov  8 01:01:44 jade pdns_server: [bindbackend] Warning! Skipping zone '.' because type 'hint' is invalid
Nov  8 01:01:44 jade pdns_server: [bindbackend] Warning! Skipping zone '.' because type 'hint' is invalid
Nov  8 01:01:44 jade pdns_server: [bindbackend] Warning! Skipping zone '.' because type 'hint' is invalid
Nov  8 01:01:45 jade pdns_server: [bindbackend] Done parsing domains, 3 rejected, 115 new, 0 removed
Nov  8 01:01:45 jade systemd: Started PowerDNS Authoritative Server.
Nov  8 01:01:45 jade pdns_server: Only asked for 1 backend thread - operating unthreaded
Nov  8 01:02:01 jade systemd: Created slice User Slice of nmonagent.
Nov  8 01:02:01 jade systemd: Starting User Slice of nmonagent.
Nov  8 01:02:01 jade systemd: Started Session 211373 of user nmonagent.
Nov  8 01:02:01 jade systemd: Starting Session 211373 of user nmonagent.
Nov  8 01:02:03 jade systemd: Stopping cPanel services...
Nov  8 01:02:03 jade systemd: Stopped cPanel services.
Nov  8 01:02:03 jade systemd: Cannot add dependency job for unit mailman.service, ignoring: Unit not found.
Nov  8 01:02:03 jade systemd: Starting cPanel services...
Nov  8 01:02:03 jade restartsrv_cpsrvd: License is valid and has already updated recently.
Nov  8 01:02:04 jade restartsrv_cpsrvd: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....
Nov  8 01:02:05 jade systemd: Removed slice User Slice of nmonagent.
Nov  8 01:02:05 jade systemd: Stopping User Slice of nmonagent.
Nov  8 01:02:05 jade restartsrv_cpsrvd: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....
Nov  8 01:02:06 jade restartsrv_cpsrvd: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to port 80 (Address already in use)....

EDIT: I might add, we have Apache running on these servers too. We've had 0 problems running cPanel DNS Only and Apache until now. However, it seems when I disable httpd I can restart cPanel, but httpd won't run as something is taking up port 80? If I use netstat to try to locate what is running on port 80 it doesn't show anything, do you have any experience with this @cPanelLauren

 

[cPanelMichael]

Hello @inteldigital,

It looks like the issue you are facing stems from the following change noted in the cPanel & WHM version 76 Release Notes:

The Prevent cpsrvd from serving standard HTTP ports Tweak Setting
In cPanel & WHM version 76, we added the Prevent cpsrvd from serving standard HTTP ports setting to the Systems tab of WHM's Tweak Settings interface (WHM >> Home >> Server Configuration >> Tweak Settings). This setting prevents the cpsrvd daemon from taking over the standard HTTP ports when you disable the system’s web server services, such as Apache. This action renders any cPanel & WHM features that depend on the standard HTTP ports partially or entirely unusable. These features include service subdomains, AutoSSL, Mailman, and BoxTrapper.

To enable this option on a DNSONLY system, you can manually edit the the following value in the /var/cpanel/cpanel.config file:

disable_cphttpd=0

To:

disable_cphttpd=1

Then, save the file and restart cpsrvd with the following command:

/usr/local/cpanel/scripts/restartsrv_cpsrvd

Note that internal case CPANEL-23845 is open to report this behavior and determine if there's a better practice to prevent the unexpected downtime upon the upgrade to version 76. I'll update this thread with more information on the status of this case as it becomes available.

Thank you.

 

[inteldigital]

@cPanelMichael Thank you so much for this, I didn't even check the changelog. I noticed that the auto upgrade had stopped for some reason but decided to run it manually. Perhaps I should have checked beforehand

EDIT: This worked great, thanks again.

 

[cPanelMichael]

Hello @inteldigital,

I'm glad to see the solution helped.

Regarding CPANEL-23845, the change from this case will ensure that cpsrvd only listens on HTTP ports when the Web Server role is disabled. You can read more about roles and profiles at:

How to Use Server Profiles - cPanel Knowledge Base - cPanel Documentation

This will prevent the issue you encountered from happening on standard cPanel & WHM servers. However, since you are using DNSOnly, manually changing the disable_cphttpd setting using the instructions from the post above remains a requirement. You shouldn't have to make any additional changes now that you've made the change.

Thank you.

 

[inteldigital]

@cPanelMichael

Thanks for updating me on this. Much appreciated. The changes implemented have worked and we've had no further issues with our production DNS Only servers.

 

[cPanelMichael]

Hello @inteldigital,

I'm glad to see the suggested change helped.

For anyone facing this issue on standard cPanel & WHM server, this is fixed in cPanel & WHM version 76.0.7:

Fixed case CPANEL-23845: Make cpsrvd listen on HTTP ports only if WebServer role is disabled.

You can learn more about our release tiers by clicking here and see which build is published to each release tier by clicking here.

Thank you.

 

[Juan R. Pozo]

As a suggestion, if there is something listening on port 80/443, maybe you shouldn't stop it and make cpsrvd listen on HTTP ports, even if WebServer role is disabled.

 

[cPanelMichael]

As a suggestion, if there is something listening on port 80/443, maybe you shouldn't stop it and make cpsrvd listen on HTTP ports, even if WebServer role is disabled.

Hello @Juan R. Pozo,

The disable_cphttpd setting referenced here is available if you have a web server listening on those ports on cPanel & WHM servers setup as DNS nodes, Database nodes, or Mail nodes. The default behavior is to enable cpsrvd to run on these ports on these node types. This ensures that the cPanel & WHM features that depend on these ports continue to function, such as service subdomains, AutoSSL, Mailman, and BoxTrapper.

Thank you.