SOLVED Issues after interrupted update

maverickws

Member
Dec 13, 2017
22
4
3
Lisbon
cPanel Access Level
Root Administrator
Hi,

Recently we were updating the cPanel/WHM software on our development server and the process was interrupted.
We were having a number of issues with EA4 which we were able to fix by running
Code:
dnf reinstall ea-*
However, we are still getting a number of other issues, such as failed services at restart:

Code:
[[email protected] ~]# systemctl --failed
  UNIT                   LOAD   ACTIVE SUB    DESCRIPTION                                  
● cpanel_php_fpm.service loaded failed failed FPM service for cPanel Daemons               
● cpgreylistd.service    loaded failed failed cPanel Greylisting Daemon                    
● httpd.service          loaded failed failed Apache web server managed by cPanel EasyApache
● mailman.service        loaded failed failed mailman services                             

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

4 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
Code:
[[email protected] ~]# /usr/local/cpanel/scripts/restartsrv_httpd
Waiting for “httpd” to start ……info [restartsrv_httpd] systemd failed to start the service “httpd” (The “/usr/bin/systemctl restart httpd.service --no-ask-password” command (process 16134) reported error number 1 when it ended.): Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xe" for details.

…Waiting for httpd,litespeed,lscgid to shutdown ... not running.
info [restartsrv_httpd] systemd failed to start the service “httpd” (The “/usr/bin/systemctl restart httpd.service --no-ask-password” command (process 16155) reported error number 1 when it ended.): Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xe" for details.

…info [restartsrv_httpd] The service 'httpd' failed to restart at least three times. The system will now rebuild the httpd.conf file.
info [restartsrv_httpd] systemd failed to start the service “httpd” (The “/usr/bin/systemctl restart httpd.service --no-ask-password” command (process 16178) reported error number 1 when it ended.): Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xe" for details.

…failed.

Cpanel::Exception::Services::StartError
Service Status
    undefined status from Cpanel::ServiceManager::Services::Httpd

Service Error
    (XID uxvm5f) The “httpd” service failed to start.

Startup Log
    Oct 29 19:38:24 dev-cpanel.domain.io systemd[1]: Starting Apache web server managed by cPanel EasyApache...
    Oct 29 19:38:24 dev-cpanel.domain.io restartsrv_httpd[16179]: AH00534: httpd: Configuration error: No MPM loaded.
    Oct 29 19:38:24 dev-cpanel.domain.io systemd[1]: httpd.service: Control process exited, code=exited status=1
    Oct 29 19:38:24 dev-cpanel.domain.io systemd[1]: httpd.service: Failed with result 'exit-code'.
    Oct 29 19:38:24 dev-cpanel.domain.io systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.

Log Messages
    [Fri Oct 29 17:47:59.812593 2021] [mpm_prefork:notice] [pid 2246] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Fri Oct 29 13:35:19.461452 2021] [mpm_prefork:notice] [pid 2246] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Fri Oct 29 13:35:19.181521 2021] [:notice] [pid 2034] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
    [Fri Oct 29 13:25:48.381219 2021] [mpm_prefork:notice] [pid 259231] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Fri Oct 29 12:55:58.689969 2021] [mpm_prefork:notice] [pid 259231] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Fri Oct 29 12:55:56.582262 2021] [mpm_prefork:notice] [pid 259231] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Fri Oct 29 12:55:56.510133 2021] [:notice] [pid 259204] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
    [Fri Oct 29 12:55:22.358552 2021] [mpm_prefork:notice] [pid 258791] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Fri Oct 29 12:55:22.259967 2021] [:notice] [pid 258765] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
    [Wed Oct 27 19:58:22.397636 2021] [mpm_prefork:notice] [pid 1689] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Wed Oct 27 19:58:21.920144 2021] [:notice] [pid 1548] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
    [Wed Oct 27 19:46:24.999160 2021] [mpm_prefork:notice] [pid 1809] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Wed Oct 27 19:46:24.852232 2021] [:notice] [pid 1752] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
    [Thu Oct 14 12:17:06.913665 2021] [mpm_prefork:notice] [pid 1697] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Thu Oct 14 12:17:06.317291 2021] [:notice] [pid 1539] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
    [Thu Oct 14 12:14:56.085003 2021] [mpm_prefork:notice] [pid 1776] AH00163: Apache/2.4.51 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 Phusion_Passenger/6.0.11 configured -- resuming normal operations
    [Thu Oct 14 12:14:55.808285 2021] [:notice] [pid 1556] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.

httpd has failed. Contact your system administrator if the service does not automagically recover.
[[email protected] ~]#
About the error above
Code:
Oct 29 19:38:24 dev-cpanel.domain.io restartsrv_httpd[16179]: AH00534: httpd: Configuration error: No MPM loaded.
after recovering EA through dnf reinstall I provisioned Apache with MPM Event (default is prefork) and the output of EA4 was provision OK.

So... is any way of recovering this server, or should we just spin a new one? Thanks
 
Last edited by a moderator:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
12,498
1,971
363
cPanel Access Level
Root Administrator
Hey there! If you don't have accounts on the server, and you still see issues after running "/scripts/upcp --force" then it may just be easier to create a new system. I personally would always worry about lingering issues that may not be obvious, but could pop up later.
 

maverickws

Member
Dec 13, 2017
22
4
3
Lisbon
cPanel Access Level
Root Administrator
Hi @cPRex thank you for your quick reply. I have two accounts on this server (test accounts) one has a staging version of our website and the other has the development version of our WHMCS. Ofc I can always back them up and restore them afterwards, I was just thinking of trying to fix the problems well to get a hold of how to get around it.

I think I had ran /scripts/upcp --force but I will run it again.
 

maverickws

Member
Dec 13, 2017
22
4
3
Lisbon
cPanel Access Level
Root Administrator
Ok so I had done this before but I did it again. Originally it was set as MPM Prefork, I had changed it to MPM Event which we use on all our servers.
Now I went back and changed from MPM Event back to MPM Prefork ... figuring it will install what is required.

Code:
                    Provision process started.
[2021-10-29 20:09:30 +0100] info [xml-api] ================================================================================
[2021-10-29 20:09:30 +0100] info [xml-api] Package Arch Version Repo Size
[2021-10-29 20:09:30 +0100] info [xml-api] ================================================================================
[2021-10-29 20:09:30 +0100] info [xml-api] Installing:
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_cgi x86_64 2.4.51-1.2.3.cpanel EA4 49 k
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_mpm_prefork x86_64 2.4.51-1.2.3.cpanel EA4 51 k
[2021-10-29 20:09:30 +0100] info [xml-api] Removing:
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_cgid x86_64 2.4.51-1.2.3.cpanel @EA4 40 k
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_http2 x86_64 2.4.51-1.2.3.cpanel @EA4 258 k
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_mpm_event x86_64 2.4.51-1.2.3.cpanel @EA4 72 k
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_proxy_hcheck x86_64 2.4.51-1.2.3.cpanel @EA4 36 k
[2021-10-29 20:09:30 +0100] info [xml-api] Transaction Summary
[2021-10-29 20:09:30 +0100] info [xml-api] ================================================================================
[2021-10-29 20:09:30 +0100] info [xml-api] Install 2 Packages
[2021-10-29 20:09:30 +0100] info [xml-api] Remove 4 Packages
[2021-10-29 20:09:30 +0100] info [xml-api] Total download size: 100 k
[2021-10-29 20:09:30 +0100] info [xml-api] Downloading Packages:
[2021-10-29 20:09:30 +0100] info [xml-api] (1/2): ea-apache24-mod_cgi-2.4.51-1.2.3.cpanel. 98 kB/s | 49 kB 00:00
[2021-10-29 20:09:30 +0100] info [xml-api] (2/2): ea-apache24-mod_mpm_prefork-2.4.51-1.2.3 101 kB/s | 51 kB 00:00
[2021-10-29 20:09:30 +0100] info [xml-api] --------------------------------------------------------------------------------
[2021-10-29 20:09:30 +0100] info [xml-api] Total 90 kB/s | 100 kB 00:01
[2021-10-29 20:09:30 +0100] info [xml-api] Running transaction check
[2021-10-29 20:09:30 +0100] info [xml-api] Transaction check succeeded.
[2021-10-29 20:09:30 +0100] info [xml-api] Running transaction test
[2021-10-29 20:09:30 +0100] info [xml-api] Transaction test succeeded.
[2021-10-29 20:09:30 +0100] info [xml-api] Running transaction
[2021-10-29 20:09:30 +0100] info [xml-api] Preparing : 1/1
[2021-10-29 20:09:30 +0100] info [xml-api] Installing : ea-apache24-mod_mpm_prefork-2.4.51-1.2.3.cpanel.x86_ 1/6
[2021-10-29 20:09:30 +0100] info [xml-api] Installing : ea-apache24-mod_cgi-2.4.51-1.2.3.cpanel.x86_64 2/6
[2021-10-29 20:09:30 +0100] info [xml-api] Erasing : ea-apache24-mod_cgid-2.4.51-1.2.3.cpanel.x86_64 3/6
[2021-10-29 20:09:30 +0100] info [xml-api] Erasing : ea-apache24-mod_mpm_event-2.4.51-1.2.3.cpanel.x86_64 4/6
[2021-10-29 20:09:30 +0100] info [xml-api] Erasing : ea-apache24-mod_proxy_hcheck-2.4.51-1.2.3.cpanel.x86 5/6
[2021-10-29 20:09:30 +0100] info [xml-api] Erasing : ea-apache24-mod_http2-2.4.51-1.2.3.cpanel.x86_64 6/6
[2021-10-29 20:09:30 +0100] info [xml-api] Running scriptlet: ea-apache24-mod_http2-2.4.51-1.2.3.cpanel.x86_64 6/6
[2021-10-29 20:09:30 +0100] info [xml-api] Verifying : ea-apache24-mod_cgi-2.4.51-1.2.3.cpanel.x86_64 1/6
[2021-10-29 20:09:30 +0100] info [xml-api] Verifying : ea-apache24-mod_mpm_prefork-2.4.51-1.2.3.cpanel.x86_ 2/6
[2021-10-29 20:09:30 +0100] info [xml-api] Verifying : ea-apache24-mod_cgid-2.4.51-1.2.3.cpanel.x86_64 3/6
[2021-10-29 20:09:30 +0100] info [xml-api] Verifying : ea-apache24-mod_http2-2.4.51-1.2.3.cpanel.x86_64 4/6
[2021-10-29 20:09:30 +0100] info [xml-api] Verifying : ea-apache24-mod_mpm_event-2.4.51-1.2.3.cpanel.x86_64 5/6
[2021-10-29 20:09:30 +0100] info [xml-api] Verifying : ea-apache24-mod_proxy_hcheck-2.4.51-1.2.3.cpanel.x86 6/6
[2021-10-29 20:09:30 +0100] info [xml-api] Installed:
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_cgi-2.4.51-1.2.3.cpanel.x86_64
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_mpm_prefork-2.4.51-1.2.3.cpanel.x86_64
[2021-10-29 20:09:30 +0100] info [xml-api] Removed:
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_cgid-2.4.51-1.2.3.cpanel.x86_64
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_http2-2.4.51-1.2.3.cpanel.x86_64
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_mpm_event-2.4.51-1.2.3.cpanel.x86_64
[2021-10-29 20:09:30 +0100] info [xml-api] ea-apache24-mod_proxy_hcheck-2.4.51-1.2.3.cpanel.x86_64
[2021-10-29 20:09:30 +0100] info [xml-api] Complete!
[2021-10-29 20:09:30 +0100] info [xml-api] Last metadata expiration check: 0:13:36 ago on Fri Oct 29 19:55:53 2021.
[2021-10-29 20:09:30 +0100] info [xml-api] Dependencies resolved.
[2021-10-29 20:09:30 +0100] info [xml-api] Nothing to do.
[2021-10-29 20:09:30 +0100] info [xml-api] Complete!
[2021-10-29 20:09:30 +0100] info [xml-api] The downloaded packages were saved in cache until the next successful transaction.
[2021-10-29 20:09:30 +0100] info [xml-api] You can remove cached packages by executing 'dnf clean packages'.

Provision process finished.
But the issue persists. Is there a way to actually force the reinstallation of Apache? Thank you.

EDIT: other than "dnf reinstall ea-apache24" because that actually does nothing and the issuer persists as well.
 
Last edited:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
12,498
1,971
363
cPanel Access Level
Root Administrator
It's interesting that it's installing prefork just fine in that output, but you're still getting the same error. Could you create a support ticket with our team so we can check this out? If you're able to do that, please post the number here so I can follow along on my end.
 

maverickws

Member
Dec 13, 2017
22
4
3
Lisbon
cPanel Access Level
Root Administrator
Hi @cPRex ofc I don't mind doing that at all, just if you don't mind me making a question before that I found out one probable cause for the issue: the httpd.conf file is completely empty. 0 bytes empty.
I wonder if is there any way to restore or recreate the file? I see this would be a plausible motive for it not starting I guess.

EDIT: I was looking into it I probably can even recreate the httpd.conf file, taking another httpd.conf file from another server all I have to take in consideration would be the php-fpm handlers that are located at opt/cpanel .... php-fpm since each .sock is owned by the user of the account it belongs to.

But still, if there was some script that could recreate the file it would be so much better.
 

maverickws

Member
Dec 13, 2017
22
4
3
Lisbon
cPanel Access Level
Root Administrator
Ok so I did run that, it gives an OK, but nothing happens:

Code:
[[email protected] ~]# /scripts/rebuildhttpdconf
Built /etc/apache2/conf/httpd.conf OK
[[email protected] ~]# ls -la /etc/apache2/conf
total 84
drwxr-xr-x 2 root root    83 Oct 29 20:41 .
drwxr-xr-x 6 root root   103 Oct 29 20:15 ..
-rw------- 1 root root     0 Oct 29 20:41 httpd.conf
-rw------- 1 root root     2 Oct 29 19:24 httpd.conf.datastore
-rw-r--r-- 1 root root 13064 Oct 27 06:03 magic
-rw-r--r-- 1 root root 61489 Oct 29 19:55 mime.types
[[email protected] ~]# cat /etc/apache2/conf/httpd.conf.datastore
{}[[email protected] ~]#
I was wondering if manually creating the httpd.conf will create any issue if I leave the httpd.conf.datastore as is?
 

maverickws

Member
Dec 13, 2017
22
4
3
Lisbon
cPanel Access Level
Root Administrator
Ok so in the meanwhile I did as I mentioned, I took the httpd.conf from another cPanel server and edited the hostname / IP and virtual hosts to match what is on this server.

Code:
[[email protected] ~]# systemctl status httpd
● httpd.service - Apache web server managed by cPanel EasyApache
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-10-29 21:00:49 WEST; 1s ago
  Process: 6926 ExecStart=/usr/local/cpanel/scripts/restartsrv_httpd --no-verbose (code=exited, status=0/SUCCESS)
Main PID: 6969 (httpd)
    Tasks: 358 (limit: 23543)
   Memory: 82.2M
   CGroup: /system.slice/httpd.service
           ├─6969 /usr/sbin/httpd -k start
           ├─6970 /usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=dev-cpanel.domain.io --suffix=-bytes_log --sslport=444
           ├─6971 /usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=dev-cpanel.domain.io --mainout=/etc/apache2/logs/access_log --sslport=444
           ├─6972 /usr/sbin/httpd -k start
           ├─6973 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/bin/leechprotect
           ├─6974 /usr/sbin/httpd -k start
           ├─6976 Passenger watchdog
           ├─6980 Passenger core
           ├─7000 /usr/sbin/httpd -k start
           ├─7001 /usr/sbin/httpd -k start
           ├─7002 /usr/sbin/httpd -k start
           ├─7003 /usr/sbin/httpd -k start
           ├─7004 /usr/sbin/httpd -k start
           └─7005 /usr/sbin/httpd -k start

Oct 29 21:00:49 dev-cpanel.domain.io systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Oct 29 21:00:49 dev-cpanel.domain.io systemd[1]: httpd.service: Can't open PID file /run/apache2/httpd.pid (yet?) after start: No such file or directory
Oct 29 21:00:49 dev-cpanel.domain.io systemd[1]: Started Apache web server managed by cPanel EasyApache.
I was able to start httpd.

If you don't mind (because here it's Friday 8PM) and there's a long weekend ahead (Monday is an holiday) and to give access to you guys at cPanel I have to go and make changes to the firewalls etc... and I'm the last guy here today, I will open a ticket Tuesday morning and reference this issue. I really just want to wrap this up and leave! in the meanwhile got httpd running so that's ok, Tuesday you guys can have a field day on the server, that OK?

EDIT: and I really must because for some reason the AutoSSL isn't working either it always says it can't resolve the domain of the WHMCS development instance. The DNS is externally managed and is returning just fine... *shrug*

Thank you for all your time and feedback! Have a great weekend ahead.