In Progress CPANEL-40292 - testing installation and uninstallation of Nginx sometimes leave Nginx in a error wont start

Operating System & Version
Ubuntu
cPanel & WHM Version
102.08

Spirogg

Well-Known Member
Feb 21, 2018
700
163
43
chicago
cPanel Access Level
Root Administrator
Ive hit this issue 2 times already

Has anyone else had this issue while testing Ubuntu cPanel 102.08


Ive installed Nginx then clciked restart Nginx and it works. then Uninstall Nginx soft reboot server and install again. then click restart Nginx and get this error

Code:
  Error: [2022-03-16 22:13:18 -0500] info [restartsrv_nginx] systemd failed to start the service “nginx” (The “/usr/bin/systemctl restart nginx.service --no-ask-password” command (process 1898) reported error number 1 when it ended.): Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details. Waiting for “nginx” to start ………failed. Cpanel::Exception::Services::StartError Service Status undefined status from Cpanel::ServiceManager::Services::Nginx Service Error (XID nnxpmc) The “nginx” service failed to start. Startup Log Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Mar 16 22:13:17 server2.e.com nginx[1899]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use) Mar 16 22:13:18 server2.e.com nginx[1899]: nginx: [emerg] still could not bind() Mar 16 22:13:18 server2.e.com systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Mar 16 22:13:18 server2.e.com systemd[1]: nginx.service: Failed with result 'exit-code'. Mar 16 22:13:18 server2.e.com systemd[1]: Failed to start nginx - high
I also get an email that it failed


The service “nginx” appears to be down.
Serverserver2
Primary IP Addressxxx.xx.xx.xx
Service Namenginx
Service Statusfailed ⛔
NotificationThe service “nginx” appears to be down.
Service Check MethodThe system’s command to check or to restart this service failed.
Number of Restart Attempts6
Service Check Raw Output(XID zup3y4) The “nginx” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_nginx” reported error number 255 when it ended.
Startup LogMar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
Mar 16 20:04:31 server2 nginx[5588]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
Mar 16 20:04:32 server2 nginx[5588]: nginx: [emerg] still could not bind()
Mar 16 20:04:32 server2 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Mar 16 20:04:32 server2 systemd[1]: nginx.service: Failed with result 'exit-code'.
Mar 16 20:04:32 server2 systemd[1]: Failed to start nginx - high performance web server.
Log MessagesThe system could not provide log messages for “nginx” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/messages”: “No such file or directory”, Error while attempting to open “/var/log/secure”: “No such file or directory”
Memory Information
Used870 MB
Available10.79 GB
Installed11.64 GB
Load Information4.40 1.47 0.60
Uptime41 minutes and 13 seconds
IOStat Informationavg-cpu: %user %nice %system %iowait %steal %idle 0.71 0.03 0.43 2.12 0.00 96.71 Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd dm-0 21.63 555.17 119.17 0.00 1372508 294624 0 loop0 0.02 0.14 0.00 0.00 338 0 0 loop1 0.02 0.14 0.00 0.00 338 0 0 loop2 0.02 0.14 0.00 0.00 350 0 0 loop3 0.60 0.72 0.00 0.00 1788 0 0 loop4 0.02 0.43 0.00 0.00 1074 0 0 loop5 0.02 0.14 0.00 0.00 336 0 0 loop6 7.25 7.37 0.00 0.00 18219 0 0 loop7 0.03 0.44 0.00 0.00 1083 0 0 loop8 0.47 0.80 1.40 0.00 1976 3456 0 loop9 0.03 0.14 0.00 0.00 349 8 0 sda 15.39 558.98 118.39 0.00 1381917 292692 0 scd0 0.08 2.97 0.00 0.00 7336 0 0
Top Processes
PID​
Owner​
CPU %​
Memory %​
Command​
5929root0.460.11/usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/bin/leechprotect
6297root0.380.03/lib/systemd/systemd-udevd
1141mysql0.361.29/usr/sbin/mysqld
1087cpanelconnecttrack0.360.03/usr/local/cpanel/3rdparty/sbin/p0f -i any -u cpanelconnecttrack -d -s /var/cpanel/userhomes/cpanelconnecttrack/p0f.socket less 400 and not dst port 80 and not dst port 443 and tcp[13] & 8==0
1137root0.341.07/usr/local/cpanel/3rdparty/perl/532/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 --pidfile=/var/run/spamd.pid --listen=5 --listen=6
The chkservd process attempts to connect to “127.0.0.1:80” in order to validate that this service is functioning. If you blocked connections with iptables or the “Host Access Control” interface in WHM, this failure may be a false positive.
To resolve this issue, either open the firewall to allow connections as the root user to “127.0.0.1:80” or disable checks for this service in WHM’s “Service Manager” interface with the “Configure Monitor Settings” link below.
The system generated this notice on Thursday, March 17, 2022 at 1:13:44 AM UTC.
“Service failures (ChkServd)” notifications are currently configured to have an importance of “High”. You can change the importance or disable this type of notification in WHM’s Contact Manager at:
Do not reply to this automated message.


then I uninstalled and reinstalled and it starts ?

also the first time it removed POP3 and IMAP in service Manager
(Both selections = enable and monitor check marks) ?
and got LMTP Error Failed email. (second time Nginx would not start everything else seems to be intact. IMAP POP3 LMTP


The service “lmtp” appears to be down.
Serverserver2
Primary IP Addressxxx.xx.xxx.xxx
Service Namelmtp
Service Statusfailed ⛔
NotificationThe service “lmtp” appears to be down.
Service Check MethodThe system’s command to check or to restart this service failed.
Number of Restart Attempts1
Service Check Raw Output(XID xnkcbd) The “dovecot” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_lmtp” reported error number 255 when it ended.
Startup LogNo startup log
Log MessagesThe system could not provide log messages for “lmtp” because it failed to read all of the potential log files with the following errors: Error while attempting to open “/var/log/messages”: “No such file or directory”, Error while attempting to open “/var/log/secure”: “No such file or directory”
Memory Information
Used792 MB
Available10.86 GB
Installed11.64 GB
Load Information0.25 0.59 0.47
Uptime8 minutes and 34 seconds
IOStat Informationavg-cpu: %user %nice %system %iowait %steal %idle 1.56 0.03 1.04 7.65 0.00 89.71 Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd dm-0 75.27 2169.09 211.83 0.00 1113416 108732 0 loop0 0.08 0.66 0.00 0.00 338 0 0 loop1 0.11 0.68 0.00 0.00 350 0 0 loop2 0.08 0.66 0.00 0.00 338 0 0 loop3 2.91 3.48 0.00 0.00 1788 0 0 loop4 0.10 2.09 0.00 0.00 1074 0 0 loop5 0.08 0.65 0.00 0.00 336 0 0 loop6 1.39 3.84 3.27 0.00 1969 1676 0 loop7 34.92 35.49 0.00 0.00 18219 0 0 loop8 0.13 2.11 0.00 0.00 1083 0 0 loop9 0.14 0.68 0.02 0.00 349 8 0 sda 53.99 2186.99 210.53 0.00 1122605 108068 0 scd0 0.16 6.12 0.00 0.00 3144 0 0
Top Processes
PID​
Owner​
CPU %​
Memory %​
Command​
1121root1.751.07/usr/local/cpanel/3rdparty/perl/532/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 --pidfile=/var/run/spamd.pid --listen=5 --listen=6
2057root0.761.02spamd child
795root0.670.31/usr/lib/snapd/snapd
1068mysql0.601.31/usr/sbin/mysqld
2093root0.440.03/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
The chkservd process attempts to connect to “127.0.0.1:/var/run/dovecot/lmtp” in order to validate that this service is functioning. If you blocked connections with iptables or the “Host Access Control” interface in WHM, this failure may be a false positive.
To resolve this issue, either open the firewall to allow connections as the root user to “127.0.0.1:/var/run/dovecot/lmtp” or disable checks for this service in WHM’s “Service Manager” interface with the “Configure Monitor Settings” link below.
The system generated this notice on Thursday, March 17, 2022 at 1:59:14 AM UTC.
“Service failures (ChkServd)” notifications are currently configured to have an importance of “High”. You can change the importance or disable this type of notification in WHM’s Contact Manager at:
Do not reply to this automated message.



@cPRex

Have you ever run into this issue where nginx won’t start even if you click restart
You have to totally uninstall and reinstall

also if there ever was this issue on a production server. How would this affect users if you had to uninstall and reinstall
Would it ruin there websites. Or you just clear cache and it’s all good ?
Thanks

Kind Regards,
Spiro
 
Last edited by a moderator:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,570
2,612
363
cPanel Access Level
Root Administrator
Hey there! I did the following:

-built an Edge server on Ubuntu
-installed Nginx through the EA4 interface

At this point it restarted normally

-removed Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager

-reinstalled Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager - no odd restart issues either

At this point, I can't reproduce, but you're welcome to create a ticket with our team if you'd like us to check out that specific machine.
 

Spirogg

Well-Known Member
Feb 21, 2018
700
163
43
chicago
cPanel Access Level
Root Administrator
Hey there! I did the following:

-built an Edge server on Ubuntu
-installed Nginx through the EA4 interface

At this point it restarted normally

-removed Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager

-reinstalled Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager - no odd restart issues either

At this point, I can't reproduce, but you're welcome to create a ticket with our team if you'd like us to check out that specific machine.
- uninstall and
- soft reboot server
and then install Nginx again is what I did,

I don't think it happens all the time.
possibly Not waiting till the process finished before soft reboot ?
not sure.
will try it again later and see if I can nail down the replication and if it happens I will open ticket with the team

Thanks
 
  • Like
Reactions: cPRex

Spirogg

Well-Known Member
Feb 21, 2018
700
163
43
chicago
cPanel Access Level
Root Administrator
Hey there! I did the following:

-built an Edge server on Ubuntu
-installed Nginx through the EA4 interface

At this point it restarted normally

-removed Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager

-reinstalled Nginx through EA4

I confirmed I still see POP3 and IMAP in service manager - no odd restart issues either

At this point, I can't reproduce, but you're welcome to create a ticket with our team if you'd like us to check out that specific machine.
ok It happened again I have Proxmox Server and this is a VM Ubuntu and cPanel 102.08 everything was running fine with serer for a few days and today i updated the Proxmox server some apt updates and installed them and had to reboot server. then I got email that Nginx is down. went to Home/Software/Nginx Manager click reboot Nignx and got the error again, So I just put in a ticket for them to look at why this is happening only in Ubuntu. the Other 2 VM's are using AlmaLinux and cPanel 102.08 and they are up and running. ?

here is the Ticket #94427991
if you want to Follow along
Thanks
Spiro
 

Spirogg

Well-Known Member
Feb 21, 2018
700
163
43
chicago
cPanel Access Level
Root Administrator
Thanks so much - following now!
@cPRex

Once again David H. has replicated this issue on a test server and found the issue


It appears that on Ubuntu servers, the "nginx-debug" service is set to be enabled when the ea-nginx package is installed. You can see this here:
Code:
[18:32:12 server2 root@94427991 ~]cPs# systemctl status nginx-debug
● nginx-debug.service - nginx - high performance web server
     Loaded: loaded (/lib/systemd/system/nginx-debug.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-03-22 01:37:20 CDT; 17h ago
       Docs: nginx documentation
    Process: 1001 ExecStart=/usr/sbin/nginx-debug -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
   Main PID: 1289 (nginx-debug)
On CentOS servers, this service is disabled as expected:
Code:
# systemctl status nginx-debug
● nginx-debug.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx-debug.service; disabled; vendor preset: disabled)
This "nginx-debug" process starts before "nginx", and binds to ports 80 and 443 as per usual. When "nginx" attempts to start, it fails as the ports are already bound to "nginx-debug". You can see this here in the server's log as well:
Code:
Mar 22 01:37:12 server2 systemd[1]: Starting nginx - high performance web server...
Mar 22 01:37:12 server2 systemd[1]: Starting nginx - high performance web server...
Mar 22 01:37:15 server2 nginx[1002]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Mar 22 01:37:15 server2 nginx[1002]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Mar 22 01:37:15 server2 nginx[1002]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
I have also performed the workaround on your server at this time to start the standard EA-NGINX process:
Code:
[18:41:26 server2 root@94427991 ~]cPs# systemctl stop nginx-debug
[19:14:33 server2 root@94427991 ~]cPs# systemctl disable nginx-debug
Removed /etc/systemd/system/multi-user.target.wants/nginx-debug.service.
[19:14:39 server2 root@94427991 ~]cPs# /scripts/restartsrv_nginx
Waiting for “nginx” to start ……waiting for “nginx” to initialize ………finished.

Service Status
        nginx (nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf) is running as root with PID 55124 (systemd+/proc check method).

Startup Log
        Mar 22 19:14:46 server2. e .com systemd[1]: Starting nginx - high performance web server...
        Mar 22 19:14:46 server2.e .com systemd[1]: Started nginx - high performance web server.

nginx started successfully.

I have filed a case with our developers regarding this with case ID CPANEL-40292.
I also have created the following article for additional information on this: EA-NGINX fails to start on Ubuntu after a server restart

You can follow the article to receive a notification when an update is available on this as well.

Workaround
Disable and stop the nginx-debug service:

Code:
systemctl stop nginx-debug
systemctl disable nginx-debug

Start the proper EA-NGINX process with the following:

Code:
/scripts/restartsrv_nginx


Thanks To David again for helping resolve the issue.
 
Last edited by a moderator:
  • Like
Reactions: kodeslogic