Summary of recent Let's Encrypt issues

Status
Not open for further replies.

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
7,496
1,009
313
cPanel Access Level
Root Administrator
Hello everyone! This post serves as a summary of the cPanel, Let's Encrypt, and OpenSSL issues that were experienced the week of October 1. This is going to be a bit long, but there were so many moving parts involved that isn't really avoidable, although I've tried to keep this down to the most important information.

What Happened:

When Let's Encrypt started their business in 2015, they needed a way to issue certificates that derive from trusted roots as they did not have their own. While waiting for their own, they signed certificates via another CA's trusted root, "IdenTrust DST Root R3". They did so by creating an intermediate certificate, "ISRG Root X1", signed by that IdenTrust root. This avoids the years of waiting needed to become a trusted certificate authority and allowed them to start distributing certificates almost immediately. Eventually Let's Encrypt obtained their own trusted root, which they named "ISRG Root X1" to match their IdenTrust-signed intermediate. Thus, any certificate that was valid via the intermediate could also be valid via Let's Encrypt's own new trusted root.

The "IdenTrust DST Root R3" certificate expired on 30 September 2021. Many clients no longer consider this a valid trusted root. Other clients, like Android phones, disregard trusted-root expiry and so continue to regard "IdenTrust DST Root R3" as valid. Most client libraries that contain Let's Encrypt's trusted root now ignore Let's Encrypt's IdenTrust-signed intermediate if provided in a TLS handshake. Old Android devices, though, can validate the chain ONLY if they receive that IdenTrust-signed intermediate. Thus, servers should ideally still send that intermediate certificate in order to maximize compatibility with older devices. This is why Let's Encrypt's API still includes that intermediate in the response when a client fetches a newly-issued certificate.

Let's Encrypt decided they would handle this by giving out the older certificate chain they have been using since 2015, along with the new data. A modern device would perform what is called Trusted-First verification, where it ignores unneeded intermediates when validation is possible using only part of the certificate chain that the server offers.

Now, let's talk about cPanel and OpenSSL.

SSL libraries usually perform trusted-first verification. However, the OpenSSL versions that ship with CentOS 7 and older operating systems when doing a standalone verification (i.e., not an actual connection handshake) require manual configuration in order to do trusted-first verification. cPanel did not configure OpenSSL to work that way, which caused the following to happen:

-as OpenSSL checked all portions of the certificate, it was able to be installed just fine to Apache
-Domain TLS, which includes the email services on the machine, failed, as that service does not allow the installation of invalid certificates. This caused users to receive an SSL error when connecting to the mail server as they would have been presented with the hostname SSL instead of the domain SSL.

When this happened AutoSSL kept running to try and replace the invalid certificates, causing them to refresh repeatedly.

The fixes we have applied

In order to get things working quickly, we did the following:

Autofixer Version 1: this fixed certificates that were already installed by removing the extra certificate provided by Let's Encrypt
Let's Encrypt Plugin Fix": this would completely omit the extra certificate on new installations as AutoSSL ran again

This is where we are currently at in this process.

Future Work


The previous fixes were designed to get sites working as quickly as possible as a method to "stop the bleeding" but this came at the cost of Android compatibility. The ideal solution that is planned for the future is to implement additional logic into OpenSSL to implement the Trusted-First verification process. However, this involves making several changes to the cPanel codebase, including backporting to previous versions. This would fix the compatibility issues with older Android devices. It's important to note that unsupported cPanel systems, such as version 96, 92, or even older machines, will have permanently degraded Android compatibility, so it would be best to update those systems where possible, or perform a migration to a modern machine.

Why the fixes took longer than we thought

Many users decided to switch from Let's Encrypt to the cPanel default Sectigo provider. Normally, we would have expected that to resolve the issues, but so many people attempted that switch at the same time that Sectigo also had to enforce ratelimits. Let's Encrypt was also inaccessible even after we pushed the initial autofixer, which caused additional delays to our testing due to ratelimiting, and delays to customers getting new certificates.

Some customers ran into the odd 644 permissions issue on directories inside /var/cpanel/ssl/domain_tls. We have a fix applied for this in version 100 and it will likely be backported to older supported versions soon.

What cPanel could have done differently

One of the common questions received is "why didn't cPanel test this when you had over one year of notice" and that is definitely a valid concern. Initially, when we heard about the impending change, we confirmed that our supported clients and operating systems trusted the new Let's Encrypt root certificate. Updates to root certificates do happen, and that is usually taken care of by the operating system. However, what we didn't expect was the fallout to cPanel services due to Let's Encrypt including an intermediate certificate that points to the old root certificate, causing problems with newly-issued certificates and OpenSSL verification. To work around this issue in the future, we are exploring the option of a formal testing policy for all root certificate expirations.
 
Status
Not open for further replies.
Thread starter Similar threads Forum Replies Date
I cPanel Announcements 2