SOLVED UPS-421 - Apache 2.4.52 issues?

sparek-3

Well-Known Member
Aug 10, 2002
2,138
260
388
cPanel Access Level
Root Administrator
Anybody seeing a lot of Apache dying with the latest EA4 Apache?

Upgraded to ea-apache24-2.4.52-1.1.3.cpanel.x86_64 earlier today across all of our servers, but I'm getting a lot of servers with the web server not responding.

Not sure if it's related or not - but seems to be a bit more than just a coincidence. This seems to be affecting a lot of our servers and all of our servers were upgraded to ea-apache24-2.4.52-1.1.3.cpanel.x86_64 earlier today (December 22).

Still we have some servers that were upgraded to ea-apache24-2.4.52-1.1.3.cpanel.x86_64 and they don't appear to be having this issue.

I've downgraded one server back to Apache 2.4.51 to see if the issue persists.

Just wondering if anyone else is seeing any similar behaviors after updating to ea-apache24-2.4.52-1.1.3.cpanel.x86_64
 
  • Like
Reactions: cpuser4234

sparek-3

Well-Known Member
Aug 10, 2002
2,138
260
388
cPanel Access Level
Root Administrator
Using the event MPM.

Still trying to work out exactly what might be happening, because I know it's hard to diagnose anything without any evidence to pick through.

Best I can tell from the Apache error_log with such events, the error log will report something like:

AH10291: h2_workers: cleanup, 5 idle workers did not exit after 5 seconds.

(numbers may be different)

Followed by, something like:

body.xml:2: parser error : XML declaration allowed only at the start of the document
<?xml version="1.0" encoding="iso-8859-1"?>
^
body.xml:2: parser error : XML declaration allowed only at the start of the document
<?xml version="1.0" encoding="iso-8859-1"?>
^
body.xml:2: parser error : Input is not proper UTF-8, indicate encoding !
Bytes: 0x8E 0x45 0x52 0x54
<data><value><array><data><value><string>XXXXX</string></value><value><string>A


And then Apache just stops handling new connections. It just sits there until it is restarted.

I'm noticing similar such entries in the Apache error_log for servers that are running Apache 2.4.51, but Apache seems to continue trucking afterwards.

I have no clue what that in the error_log is actually referring to, or if it has anything at all to do with the issue, but that seems to be where Apache 2.4.52 stops after that is reported in the error_log.

Just wondering if anybody else is seeing this issue? I figure more people may pop in today with the issue, since ea-apache24-2.4.52-1.1.3.cpanel.x86_64 was release during mid-day yesterday (December 22) and most folks probably have auto-updates set to run during the overnight hours.
 

sparek-3

Well-Known Member
Aug 10, 2002
2,138
260
388
cPanel Access Level
Root Administrator
I've actually run into this issue on an *ahem* other control panel that is also using Apache 2.4.52.

So I'm almost convinced that something within the code change from Apache 2.4.51 to Apache 2.4.52 is causing this.

Just really, really puzzled that nobody else is experiencing this issue.
 

voip

Member
Apr 8, 2004
17
2
153
And then Apache just stops handling new connections. It just sits there until it is restarted.
Same here. Since yesterday it has been necessary to restart Apache several times on several servers when it is not usual to do so.

When it stops accepting connections I don't see anything at /usr/local/apache/logs/error_log

I use event MPM.

Directdmin users are also reporting the issue:
 

coursevector

Well-Known Member
Feb 23, 2015
162
28
68
cPanel Access Level
Root Administrator
Been getting alerts from my servers all day today, about 8 times in one day when i usually have one a month. SOMETHING is up. And like @voip said, nothing in the logs at all.
 

gnusys

Well-Known Member
Jan 18, 2013
62
20
58
cPanel Access Level
DataCenter Provider
Same issue here too. seeing a lot of alerts regarding apache being unresponsive ( no error in the log). Then checkservd kicks in and restart apache and it goes fine until this is repeated after a few hours
 
Last edited:

sparek-3

Well-Known Member
Aug 10, 2002
2,138
260
388
cPanel Access Level
Root Administrator
You can downgrade Apache back down to Apache 2.4.51 - but you'll also need to be sure that you do not have automatic updates enabled, otherwise your server's just going to upgrade to Apache 2.4.52 on the next automatic package update run.

First you need to determine what RPM version of Apache you have installed:

rpm -qa ea-apache24

It would appear that cPanel released two versions of Apache 2.4.52:

ea-apache24-2.4.52-1.1.1.cpanel.x86_64
ea-apache24-2.4.52-1.1.3.cpanel.x86_64


If you are using Apache 2.4.52, then rpm -qa ea-apache24 will report one of these two versions. I would suspect that most everyone using Apache 2.4.52 on cPanel is using ea-apache24-2.4.52-1.1.3.cpanel.x86_64. I think ea-apache24-2.4.52-1.1.1.cpanel.x86_64 was released and then almost immediately overwritten by ea-apache24-2.4.52-1.1.3.cpanel.x86_64. So the vast majority of folks went from ea-apache24-2.4.51-2.4.1.cpanel.x86_64 straight to ea-apache24-2.4.52-1.1.3.cpanel.x86_64. But it's technically possible someone got ea-apache24-2.4.52-1.1.1.cpanel.x86_64 and never updated to ea-apache24-2.4.52-1.1.3.cpanel.x86_64.

The version you want to get down to is:

ea-apache24-2.4.51-2.4.1.cpanel.x86_64

Which would appear to be the latest version of Apache 2.4.51 that cPanel released

I'm going to assume that you are running ea-apache24-2.4.52-1.1.3.cpanel.x86_64 - and there may be a more eloquent way of doing this - I'm actually going to downgrade twice. The first downgrade will get me to ea-apache24-2.4.52-1.1.1.cpanel.x86_64 and the second downgrade will get me to ea-apache24-2.4.51-2.4.1.cpanel.x86_64 which is what I want. Probably is a more eloquent way of doing this, but by the time I could find it I could have already run the downgrade twice.

If by chance you are using ea-apache24-2.4.52-1.1.1.cpanel.x86_64 then you will only need to downgrade once.

To insure better results, simply do a rpm -qa ea-apache24 after each downgrade until you see ea-apache24-2.4.51-2.4.1.cpanel.x86_64.

Make a note of what version of Apache you have installed from rpm -qa ea-apache24. Remember there is a ea-apache24-2.4.52-1.1.3.cpanel.x86_64 and a ea-apache24-2.4.52-1.1.1.cpanel.x86_64. Again, I'm going to assume that you are using ea-apache24-2.4.52-1.1.3.cpanel.x86_64 but if you are using ea-apache24-2.4.52-1.1.1.cpanel.x86_64 then note the difference in versions.

To get a list of the packages affected, you will need to grep through your yum.log. Now... probably your yum logs have already been rotated, so what you're after is likely in /var/log/yum.log-20211223 - but for simplicity sake I'm just going to look through all of the yum logs:

cat /var/log/yum.log* | grep 'ea-apache24.*2.4.52-1.1.3.cpanel' | awk '{print $5}' | sed s/-2.4.52-1.1.3.cpanel.*//g | sort | uniq

(again - note the RPM version used here)

This is a list of all of the packages you will want to downgrade. The | sort | uniq is there just in the off chance you have duplicate packages listed. You can either copy and paste this list into a package list passed to yum downgrade or you can pass the output of the above directly into the package list to yum downgrade:

yum downgrade $(cat /var/log/yum.log* | grep 'ea-apache24.*2.4.52-1.1.3.cpanel' | awk '{print $5}' | sed s/-2.4.52-1.1.3.cpanel.*//g | sort | uniq)

If you'll note in this particular case yum will tell you that you are about to downgrade all of these to 2.4.52-1.1.1. Proceed with this downgrade.

See what version of Apache you are using:

rpm -qa ea-apache24

This should report:

ea-apache24-2.4.52-1.1.1.cpanel.x86_64

If it happens to report:

ea-apache24-2.4.51-2.4.1.cpanel.x86_64

then you are done.


If it reports

ea-apache24-2.4.52-1.1.1.cpanel.x86_64

then do another downgrade:

yum downgrade $(cat /var/log/yum.log* | grep 'ea-apache24.*2.4.52-1.1.3.cpanel' | awk '{print $5}' | sed s/-2.4.52-1.1.3.cpanel.*//g | sort | uniq)

This time yum should report that it's going to downograde to 2.4.51-2.4.1. Proceed with this downgrade.

Now when you run:

rpm -qa ea-apache24

This should report:

ea-apache24-2.4.51-2.4.1.cpanel.x86_64

And that's it. You're now using Apache 2.4.51, which does not have this not responding bug.

However, ultimately, the folks over at Apache will need to fix this issue in Apache 2.4.52 or otherwise you won't get any of the benefits of Apache versions after 2.4.52.
 

gnusys

Well-Known Member
Jan 18, 2013
62
20
58
cPanel Access Level
DataCenter Provider
A shorter version of the downgrade command is

Bash:
yum downgrade ea-apache24* liblsapi liblsapi-devel
Run the command again till you see the rpm version as ea-apache24-2.4.51 in the command

Bash:
rpm -qa ea-apache24
 
  • Like
Reactions: sparek-3

cpuser4234

Registered
Nov 13, 2012
2
0
51
cPanel Access Level
Root Administrator
We also started getting "httpd appears to be down" notifications starting Christmas Eve.

Here's our Apache server info:
  • Server Version: Apache/2.4.52 (cPanel) OpenSSL/1.1.1m mod_bwlimited/1.4 mod_fcgid/2.3.9
  • Server MPM: event
  • Server Built: Dec 22 2021 02:53:54
 

rainboy

Active Member
Mar 2, 2004
34
3
158
Eindhoven
same issue here. at random servers stop to accept connections, nothing in error logs, about 10min later they accept connections again. Started the day after 2.4.52 was installed with random servers. (7 & 8 based) with mod_mpm_event enabled.
 

kacsa

Member
Apr 6, 2003
19
0
151
We have same problem on some servers (servers are use event mpm, but not all cPanel servers have this problem)
 
Last edited: