SOLVED [EA-8886] Tomcat 8.5 upgrade breaks installations

[tizoo]

Hi,

I woke up to a few customers complaining that their Tomcat wasn't responding, resulting in 503 errors.

Symptoms
The error thrown by Tomcat (in catalina.out) is :

...
SEVERE [main] org.apache.catalina.core.StandardService.startInternal Failed to start connector [Connector[AJP/1.3-10019]]
    ...
    Caused by: java.lang.IllegalArgumentException: The AJP Connector is configured with secretRequired="true" but the secret attribute is either null or "". This combination is not valid.
   ...

Problem
This is linked to an upgrade that happened overnight:

# grep tomcat /var/log/yum.log
Mar 07 02:57:21 Updated: 1:ea-tomcat85-8.5.51-2.el7.cloudlinux.x86_64

The breaking change is documented here :

Apache Tomcat 8 (8.5.63) - Changelog

Solution
To solve this, one needs to add the secretRequired="false" to the AJP connector definition in server.xml.

Here is a script that does it for all users on the server, and then restart Tomcat for each user :

for USER in `ls /home/*/ea-tomcat85/conf/server.xml | cut -d/ -f3`
do
    echo "Fixing $USER's Tomcat"
    sed -i 's/protocol="AJP\/1.3" xpoweredBy="false"/protocol="AJP\/1.3" secretRequired="false" xpoweredBy="false"/g' /home/$USER/ea-tomcat85/conf/server.xml
    su - $USER -c "export PATH=$(dirname $(readlink /usr/local/cpanel/3rdparty/bin/perl)):$PATH && ubic restart ea-tomcat85"
    sleep 5
done;

Hope this helps someone else!

 

[MichaelCropper]

I have a feeling this issue is related to the one I'm seeing, but it's not obvious.

Previously I've always started/stopped Apache Tomcat via the command;

/etc/init.d/./easy-tomcat7 restart

But upon checking today, I'm getting 404s for all the Tomcat Servlet URLs oddly, which is pointing to an issue with Apache not routing 80 (HTTP) + 443 (HTTPS) traffic through Tomcat.

Is this related or a different issue? Any ideas how to resolve?

I'm really considering completely ditching the WHM + cPanel setup when critical things like this just break for no apparent reason. It's not good.

 

[cPanelLauren]

Hi @tizoo

Thanks for providing the workaround. I see we have EA-8886 - (Tomcat 8.5.51 when using AJP 1.3 requires secretRequired to be disabled) for this open and its reported as having been fixed with the next release of EasyApache which should be available in the next week or two.


@MichaelCropper

1. I'd suggest you read our Tomcat documentation here: Tomcat | cPanel & WHM Documentation

2. Based on their changelogs this was a change on Tomcat's end that would have occurred whether or not you were using the cPanel provided package for it.

Rename the requiredSecret attribute of the AJP/1.3 Connector to secret and add a new attribute secretRequired that defaults to true. When secretRequired is true the AJP/1.3 Connector will not start unless the secret attribute is configured to a non-null, non-zero length String. (markt)

Furthermore if you run into an issue with Tomcat, and you're unable to identify the cause, we can take a look at the configuration for you if you open a ticket, we're here 24/7

 

[tizoo]

Hi @cPanelLauren,

Thanks a lot for the update. Glad that cPanel fixed this in the official release.

Hi @MichaelCropper,

From your post, you're running Tomcat 7. From the change log of Tomcat 7, I see that the same change happened for version 7.0.100. Are you running this version ? If not, then the root cause of your problem is probably different.

If you're hit by this bug, you should see that Tomcat is not able to start and a connection to port 8080 (assuming it's open in your firewall and you're running Tomcat 7) should not be possible.

Hope this helps,
Florian