Failed to start Apache web server error

[Kevin Vergara]

Hello,

I got a update for cpanel/whm and suddenly there's an issue with httpd.. it cannot start.

It says :

[[email protected] sbin]# systemctl status  httpd.service
● httpd.service - Apache web server managed by cPanel EasyApache
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2019-05-27 14:11:16 -04; 1min 32s ago

May 27 14:11:16 cpanel.example.cl systemd[1]: Starting Apache web server managed by cPanel EasyApache...
May 27 14:11:16 cpanel.example.cl restartsrv_httpd[29366]: /usr/sbin/httpd: symbol lookup error: /usr/sbin/httpd: undefined symbol: apr_thread_mutex_timedlock
May 27 14:11:16 cpanel.example.cl systemd[1]: httpd.service: control process exited, code=exited status=127
May 27 14:11:16 cpanel.example.cl systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
May 27 14:11:16 cpanel.example.cl systemd[1]: Unit httpd.service entered failed state.
May 27 14:11:16 cpanel.example.cl systemd[1]: httpd.service failed.

Any help is welcome.
Thank you

 

[dalem]

run
journalctl -xe
after attempting a restart

run
apachectl configtest
to check for errors

 

[Kevin Vergara]

run
journalctl -xe
after attempting a restart

run
apachectl configtest
to check for errors

When I did the journalctl -xe

Starting Apache web server managed by cPanel EasyApache...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: systemd-devel Info Page
--
-- Unit httpd.service has begun starting up.
May 28 09:24:30 cpanel.example.cl systemd[1]: Reloading.
May 28 09:24:30 cpanel.example.cl systemd[1]: Configuration file /etc/systemd/system/cpanel-dovecot-solr.service is marked executable. Please remove executable permission bits. Proceeding anyway.
May 28 09:24:30 cpanel.example.cl systemd[1]: Configuration file /etc/systemd/system/mysql.service.d/limits.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
May 28 09:24:30 cpanel.example.cl systemd[1]: Reloading.
May 28 09:24:30 cpanel.example.cl systemd[1]: Configuration file /etc/systemd/system/cpanel-dovecot-solr.service is marked executable. Please remove executable permission bits. Proceeding anyway.
May 28 09:24:30 cpanel.example.cl systemd[1]: Configuration file /etc/systemd/system/mysql.service.d/limits.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
May 28 09:24:30 cpanel.example.cl restartsrv_httpd[5740]: /usr/sbin/httpd: symbol lookup error: /usr/sbin/httpd: undefined symbol: apr_thread_mutex_timedlock
May 28 09:24:30 cpanel.example.cl systemd[1]: httpd.service: control process exited, code=exited status=127
May 28 09:24:30 cpanel.example.cl systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: systemd-devel Info Page
--
-- Unit httpd.service has failed.
--
-- The result is failed.
May 28 09:24:30 cpanel.example.cl systemd[1]: Unit httpd.service entered failed state.
May 28 09:24:30 cpanel.example.cl systemd[1]: httpd.service failed.
May 28 09:24:30 cpanel.example.cl polkitd[4784]: Unregistered Authentication Agent for unix-process:5734:7918100 (system bus name :1.3921, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
May 28 09:24:32 cpanel.example.cl sshd[5779]: Invalid user fake from 178.62.98.15 port 56420
May 28 09:24:32 cpanel.example.cl sshd[5779]: input_userauth_request: invalid user fake [preauth]
May 28 09:24:32 cpanel.example.cl sshd[5779]: pam_unix(sshd:auth): check pass; user unknown
May 28 09:24:32 cpanel.example.cl sshd[5779]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=178.62.98.15
May 28 09:24:34 cpanel.example.cl sshd[5779]: Failed password for invalid user fake from 178.62.98.15 port 56420 ssh2
May 28 09:24:34 cpanel.example.cl sshd[5779]: Received disconnect from 178.62.98.15 port 56420:11: Bye Bye [preauth]
May 28 09:24:34 cpanel.example.cl sshd[5779]: Disconnected from 178.62.98.15 port 56420 [preauth]

and when i run

[[email protected] ~]# apachectl configtest
/usr/sbin/httpd: symbol lookup error: /usr/sbin/httpd: undefined symbol: apr_thread_mutex_timedlock

 

[dalem]

check ea-apr-util for current version
rpm -qi ea-apr-util
if not
try
yum install ea-apr-util
or
yum update ea-apr-util

 

[Kevin Vergara]

check ea-apr-util for current version
rpm -qi ea-apr-util
if not
try
yum install ea-apr-util
or
yum update ea-apr-util

When i do this:

[[email protected] ~]# rpm -qi ea-apr-util
Name        : ea-apr-util
Version     : 1.6.1
Release     : 4.4.8.cpanel
Architecture: x86_64
Install Date: Thu Mar 14 00:33:17 2019
Group       : System Environment/Libraries
Size        : 214615
License     : ASL 2.0
Signature   : RSA/SHA256, Tue Mar 12 00:21:27 2019, Key ID 2378b5957e931c7c
Source RPM  : ea-apr-util-1.6.1-4.4.8.cpanel.src.rpm
Build Date  : Tue Mar 12 00:21:27 2019
Build Host  : build29-0
Relocations : (not relocatable)
Vendor      : cPanel, Inc.
URL         : [URL='http://apr.apache.org/']Welcome! - The Apache Portable Runtime Project[/URL]
Summary     : Apache Portable Runtime Utility library
Description :
The mission of the Apache Portable Runtime (APR) is to provide a
free library of C data structures and routines.  This library
contains additional utility interfaces for APR; including support
for XML, LDAP, database interfaces, URI parsing and more.


ackage ea-apr-util-1.6.1-4.4.8.cpanel.x86_64 already installed and latest version
Nothing to do

and

[[email protected] ~]# yum update ea-apr-util
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
 * EA4: 185.69.232.245
 * EA4-experimental: 185.69.232.245
 * cpanel-addons-production-feed: 185.69.232.245
 * cpanel-plugins: 185.69.232.245
 * base: mirror.checkdomain.de
 * epel: mirror.wiuwiu.de
 * extras: mirror.wiuwiu.de
 * updates: centosmirror.netcup.net
http://download.opensuse.org/repositories/home%3A/Jperkster%3A/ea-freetds/CentOS-7_standard/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/

No packages marked for update

 

[dalem]

that is your problem yum is broken

yum clean all
yum update

 

[Kevin Vergara]

that is your problem yum is broken

yum clean all
yum update

i did, but still nothing

[[email protected] ~]# yum clean all
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror, universal-hooks
Cleaning repos: EA4 EA4-experimental cpanel-addons-production-feed cpanel-plugins base epel extras updates
Cleaning up list of fastest mirrors
[[email protected] ~]# yum update
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror, universal-hooks
Determining fastest mirrors
epel/x86_64/metalink | 30 kB 00:00:00
* EA4: 185.69.232.245
* EA4-experimental: 185.69.232.245
* cpanel-addons-production-feed: 185.69.232.245
* cpanel-plugins: 185.69.232.245
* base: mirror.checkdomain.de
* epel: mirror.wiuwiu.de
* extras: mirror.checkdomain.de
* updates: mirror.ratiokontakt.de
EA4 | 2.9 kB 00:00:00
EA4-experimental | 2.9 kB 00:00:00
cpanel-addons-production-feed | 2.9 kB 00:00:00
cpanel-plugins | 2.9 kB 00:00:00
base | 3.6 kB 00:00:00
epel | 4.7 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/11): base/7/x86_64/group_gz | 166 kB 00:00:00
(2/11): base/7/x86_64/primary_db | 6.0 MB 00:00:00
(3/11): cpanel-addons-production-feed/7/x86_64/primary_db | 27 kB 00:00:00
(4/11): epel/x86_64/group_gz | 88 kB 00:00:00
(5/11): cpanel-plugins/x86_64/primary_db | 119 kB 00:00:00
(6/11): EA4-experimental/7/x86_64/primary_db | 212 kB 00:00:00
(7/11): extras/7/x86_64/primary_db | 200 kB 00:00:00
(8/11): epel/x86_64/primary_db | 6.7 MB 00:00:00
(9/11): epel/x86_64/updateinfo | 1.0 MB 00:00:00
(10/11): EA4/7/x86_64/primary_db | 961 kB 00:00:00
(11/11): updates/7/x86_64/primary_db | 5.0 MB 00:00:00
No packages marked for update

still nothing

 

[dalem]

its the correct version rebuild your httpd.conf

/scripts/rebuildhttpdconf

 

[Kevin Vergara]

its the correct version rebuild your httpd.conf

/scripts/rebuildhttpdconf

That happends

[[email protected] ~]# /scripts/rebuildhttpdconf
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
/usr/sbin/httpd: symbol lookup error: /usr/sbin/httpd: undefined symbol: apr_thread_mutex_timedlock
/usr/sbin/httpd: symbol lookup error: /usr/sbin/httpd: undefined symbol: apr_thread_mutex_timedlock
warn [rebuildhttpdconf] Invalid Apache Config File, /usr/sbin/httpd
/usr/sbin/httpd: symbol lookup error: /usr/sbin/httpd: undefined symbol: apr_thread_mutex_timedlock
/usr/sbin/httpd: symbol lookup error: /usr/sbin/httpd: undefined symbol: apr_thread_mutex_timedlock
warn [rebuildhttpdconf] Invalid Apache Config File, /usr/sbin/httpd
warn [rebuildhttpdconf] Unable to detect Apache version from binary
warn [rebuildhttpdconf] Failed to get apache configuration: Unable to detect Apache version from binary
Failed to build Apache configuration file (/etc/apache2/conf/httpd.conf.work.52e0fa5b)
Failed to get apache configuration: Unable to detect Apache version from binary

 

[dalem]

rpm -qa |grep ea-apache24-2.4
yum reinstall ea-apache24-2.4.XXXX

reinstall the version that is currently installed
then rebuilld if it stall does not start

------------------------------------------------------
on a separate note: error is not part of the original error ad does not affect apache

How to fix a locale setting warning from Perl?

 

[Kevin Vergara]

rpm -qa |grep ea-apache24-2.4
yum reinstall ea-apache24-2.4.XXXX

reinstall the version that is currently installed
then rebuilld if it stall does not start

[[email protected] ~]# rpm -qa |grep ea-apache24-2.4
ea-apache24-2.4.39-3.5.2.x86_64
[[email protected] ~]# yum reinstall ea-apache24-2.4.39-3.5.2.x86_64
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
epel/x86_64/metalink | 18 kB 00:00:00
* EA4: 185.69.232.245
* EA4-experimental: 185.69.232.245
* cpanel-addons-production-feed: 185.69.232.245
* cpanel-plugins: 185.69.232.245
* base: mirror.wiuwiu.de
* epel: ftp.plusline.net
* extras: mirror.wiuwiu.de
* updates: mirror.alpix.eu
EA4 | 2.9 kB 00:00:00
EA4-experimental | 2.9 kB 00:00:00
cpanel-addons-production-feed | 2.9 kB 00:00:00
cpanel-plugins | 2.9 kB 00:00:00
base | 3.6 kB 00:00:00
epel | 5.3 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/2): epel/x86_64/updateinfo | 1.0 MB 00:00:00
(2/2): epel/x86_64/primary_db | 6.7 MB 00:00:00
Installed package ea-apache24-2.4.39-3.5.2.x86_64 (from home_Jperkster_ea-freetds) not available.
Error: Nothing to do

 

[cPanelLauren]

This is part of a known upstream issue when using the Unofficial testing repository home_Jperkster_ea-freetds

There is a workaround which is as follows:

At the moment, the workaround involves removing Apache along with the modules that have been installed as part of the "home_Jperkster_ea-freetds" repository.

Usually performing a 'yum remove ea-apache24' should handle the removal of all of the modules, however some situations may require some more intervention, for example some servers will require 'ea-apache24-tools' to be removed separately.

Create staging directory and download profile

mkdir -vp /root/staging && ea_current_to_profile --output=$_/current.profile

Remove Apache and the additional tools package

yum remove ea-apache24 ea-apache24-tools

Disable the unsupported repository

yum-config-manager --disable home_Jperkster_ea-freetds

Use the EA4 profile to reinstall their packages from the official sources

awk -F '"' '/ea-apache/{printf $2" "}' /root/staging/current.profile | xargs yum  -y install

Update virtfs so that users can access PHP sessions again

/scripts/clear_orphaned_virtfs_mounts --clearall
/scripts/update_users_jail
/scripts/restartsrv_httpd --hard

 

[garconcn]

I got same issue. It happened just after cpanel upcp cronjob. I did not enable any auto update or any 3rd party yum.
This is ridiculous, this is not the 1st time cpanel upcp cronjob causing this issue within this 2 years and there's no way to prevent cpanel nightly cronjob to cause the problem.

 

[delalibera]

Hello

/usr/local/apache/bin/httpd: symbol lookup error: /usr/local/apache/bin/httpd: undefined symbol: apr_atomic_read64

The httpd not restart and return this error.

Can someone help me?

 

[cPanelLauren]

If you're getting the same error @garconcn you would have had to have enabled this 3rd party repository and the steps shown to resolve it should work for you. We can confirm this by checking your repositories in /etc/yum.repos.d/ or looking at the error when you attempt to update/reinstall apache:

yum reinstall ea-apache24

 

[GOT]

I've not encountered that before.

Are you still on EA3?

 

[garconcn]

If you're getting the same error @garconcn you would have had to have enabled this 3rd party repository and the steps shown to resolve it should work for you. We can confirm this by checking your repositories in /etc/yum.repos.d/ or looking at the error when you attempt to update/reinstall apache:

yum reinstall ea-apache24

Which one is 3rd party repo?

CentOS-Base.repo:enabled=0
CentOS-CR.repo:enabled=0
CentOS-Debuginfo.repo:enabled=0
CentOS-fasttrack.repo:enabled=0
CentOS-Media.repo:# yum --enablerepo=c7-media [command]
CentOS-Media.repo:# yum --disablerepo=\* --enablerepo=c7-media [command]
CentOS-Media.repo:enabled=0
CentOS-Sources.repo:enabled=0
CentOS-Sources.repo:enabled=0
CentOS-Sources.repo:enabled=0
CentOS-Sources.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
CentOS-Vault.repo:enabled=0
cPAddons.repo:enabled=1
EA4.repo:enabled=1
MariaDB100.repo:enabled=0
​

 

[garconcn]

RPMUP was set to manual, why cpanel still do YUM update?

# cat /etc/cpupdate.conf
CPANEL=stable
RPMUP=manual
SARULESUP=never
STAGING_DIR=/usr/local/cpanel
UPDATES=manual
​

 

[cPanelLauren]

Hello @garconcn

You don't have the repo in question here so the issue is not the same. Please show the output of the error you're receiving so we can assist you with resolving it.

RPMUP was set to manual, why cpanel still do YUM update?

I would suggest you read the documentation on this here:
Update Preferences - Version 80 Documentation - cPanel Documentation

Specifically:

Notes:

• This setting does not impact cPanel-provided RPMs.
• In cPanel & WHM version 68 and later, the /usr/local/cpanel/scripts/sysup script automatically updates the EasyApache 4 RPMs. The following settings will not prevent an automatic update:
  • The RPMUP variable set to never in the /etc/cpupdate.conf file.
  • The Operating System Package set to Never Update in WHM's Update Preferences interface (WHM >> Home >> Server Configuration >> Update Preferences).
• You can prevent the /usr/local/cpanel/scripts/upcp script from creating new EasyApache 4 yum repositories if they don't already exist. To do this, use the touch command to create the the /var/cpanel/skip-repo-setup file. For example:
  
  touch /var/cpanel/skip-repo-setup