In Progress [CPANEL-33077] Letsencrypt transition to ISRG’s Root (Important!!!!!)

[sparek-3]

I guess the part I'm confused about is

-Domain TLS, which includes the email services on the machine, failed, as that service does not allow the installation of invalid certificates. This caused users to receive an SSL error when connecting to the mail server as they would have been presented with the hostname SSL instead of the domain SSL.

What certificate in the chain was considered invalid? And what determines validity?

When issuing a Let's Encrypt certificate, the certificate for the domain is valid - yes?

ISRG Root X1 (expires September 15, 2025) is valid - yes?

DST Root CA X3 (expires September 30, 2024) is valid - yes?

So why isn't that being installed?

 

[cPRex]

The problem with that is the two-fold issue - the OpenSSL logic, and DomainTLS being much more strict about certificates than Apache. Let's Encrypt is still sending the older certificate at this time, IdenTrust DST Root R3, which fails the verification, so Domain TLS rejects that as being valid and not allowing the install to go through. This is what our first autofixer aimed to resolve.

 

[sparek-3]

No. I don't think that's true. I think that's where cPanel's confusing the two. And that's why I think it's faulty cPanel logic that's blocking this. But I'm man enough to admit that I may be wrong. I would just like to have it pointed out where I am wrong.


The DST Root CA X3 (expired September 29, 2021) certificate:
-----BEGIN CERTIFICATE-----
MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
/xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
-----END CERTIFICATE-----

The DST Root CA X3 (expires September 30, 2024) certificate:
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

Those are two different certificates.

Let's Encrypt is sending DST Root CA X3 (expires September 30, 2024)

If DST Root CA X3 (expires September 30, 2024) is included in the CA bundle then Domain TLS certificates don't get installed. But why is DST Root CA X3 (expires September 30, 2024) invalid?

The only thing I can surmise is that cPanel is seeing DST Root CA X3 and assuming it's invalid. It's not actually checking it. Thus faulty cPanel logic.

 

[cPanelFelipe]

LE is not sending DST Root CA X3, or any other root. They’re sending an intermediate that points back to the expired root.

You can see this if you grab CPAN’s Net::ACME2 and try out the example scripts to grab a certificate. (You’ll have to tweak them to point to LE’s production servers rather than the staging ones.)

 

[cPanelFelipe]

The DST-issued intermediate that LE sends has the same name (i.e., Subject) as LE’s new root: “ISRG Root X1”. Clients that have the actual root cert can thus ignore that extra intermediate.

The problem is that OpenSSL versions prior to 1.1.0—like the 1.0.2 that CentOS 7 uses—do not, by default, ignore the extra intermediate when doing a standalone verification (such as what AutoSSL does). We’ll be shipping a fix soon to supported cP versions that configures our standalone OpenSSL verifications always to ignore extra intermediates. This will allow us to install that extra intermediate again, which will restore compatibility with old Android devices that may not have the new LE root (and don’t care about the expiration of DST Root CA X3).

 

[eva2000]

The problem with that is the two-fold issue - the OpenSSL logic, and DomainTLS being much more strict about certificates than Apache. Let's Encrypt is still sending the older certificate at this time, IdenTrust DST Root R3, which fails the verification, so Domain TLS rejects that as being valid and not allowing the install to go through. This is what our first autofixer aimed to resolve.

You should be able to configure the shorter chain Long (default) and Short (alternate) Certificate Chains Explained without the intermediate reference to the expired DST ROOT CA X3 cert to avoid this. Some Acme protocol clients support this like acme.sh and certbot as a - - preferred-chain flag

There is unfortunately no fix for older client compatibility other that using a different SSL certificate provider who has an old enough cross signed root cert like ZeroSSL with their AAA Certificate Services CA root cert that supports all the way back to Android 2.3 and macOS 10.4.

Edit: ah cPanel Comodo/Sectigo SSL certs also have alternate chain to AAA Certificate Services CA - so easy just switch to that for older device support

 

[eva2000]

We’ll be shipping a fix soon to supported cP versions that configures our standalone OpenSSL verifications always to ignore extra intermediates

I guess that means you are rebuilding the Cpanel openssl 1.0.2 package with X509_V_FLAG_TRUSTED_FIRST flag enabled by default?

When X509_V_FLAG_TRUSTED_FIRST is set, which is always the case since OpenSSL 1.1.0, construction of the certificate chain in X509_verify_cert(3) searches the trust store for issuer certificates before searching the provided untrusted certificates. Local issuer certificates are often more likely to satisfy local security requirements and lead to a locally trusted root. This is especially important when some certificates in the trust store have explicit trust settings (see "TRUST SETTINGS" in openssl-x509(1)).

That flag does not ignore extra intermediates buts changes how it walks up the chain to verify so essentially will ignore the expired DST ROOT CA X3 cert

Or just using openssl verify with -trusted first flag which does the same thing

 

[cPanelFelipe]

I guess that means you are rebuilding the Cpanel openssl 1.0.2 package with X509_V_FLAG_TRUSTED_FIRST flag enabled by default?

No, just configuring our verification logic accordingly at runtime. We use the OS-provided OpenSSL rather than building one ourselves.

Contrary to what the OpenSSL team described in their “workaround #2”, we’ve found that connection-mode verification with OpenSSL 1.0.2 does not need to have trusted-first mode configured manually; only standalone verification seems to need it. You can see this if you `openssl s_client` to helloworld.letsencrypt.org.

 

[eva2000]

OpenSSL 1.0.2 does not need to have trusted-first mode configured manually

Yup same from my tests as long as expired root is removed from the trust store - ca-certificates yum updated

 

[[email protected]]

We’ll be shipping a fix soon to supported cP versions that configures our standalone OpenSSL verifications always to ignore extra intermediates. This will allow us to install that extra intermediate again, which will restore compatibility with old Android devices that may not have the new LE root (and don’t care about the expiration of DST Root CA X3).

Hello,
Can you please be more specific for "We’ll be shipping a fix soon".
You have any timeline when this problem will be fixed and all Android issues been correct?

Thank you.

 

[cPRex]

@[email protected] - I don't have any timelines at this point. I'll be sure to update this thread when I get them.

 

[jorbox]

@[email protected] - I don't have any timelines at this point. I'll be sure to update this thread when I get them.

Can you please check it,

days are passing and we lose a lot of customers

 

[cPanelAnthony]

@jorbox I just checked in and am also not seeing a timeline as Rex reported. I apologize for any inconvenience and thank you for your patience.

 

[cPRex]

At this point I know we weren't able to get to it in the next v98 release coming out. However, we're hoping it will get to the one after that. I'll be sure to post once I get more details.

 

[jandafields]

Can you please check it,

days are passing and we lose a lot of customers

Why don't you simply switch the AutoSSL provider to cPanel instead of LetsEncrypt? That's about a 5 minute job and your customers should all be happy again.

 

[monarobase]

Switching to cPanel causes issues for customers who have created a CAA record to only allow let’s Encrypt and will also cause problems for customers who require wildcard certificates.

 

[jorbox]

Why don't you simply switch the AutoSSL provider to cPanel instead of LetsEncrypt? That's about a 5 minute job and your customers should all be happy again.

Thank you but Sub domains, wildcard are not enabled with cPanel being an AutoSSL provider, even mail.domain.com, a lot of customers will have problems with their websites, its less than a 5 minute job but it will make hours of problems, my customers wont be happy they may kill me instead.

 

[spikey221]

I was wondering if there is any way to resolve this for those still on Centos 6? I've recently purchased a business that has about 130 Wordpress sites on a Centos 6 server running cPanel 86.0.40. I've run the steps here RHEL/CentOS 6 OpenSSL client compatibility after DST Root CA X3 expiration to get a newer version of openssl and updated certificates installed but I suspect this isn't the problem here from reading the post-mortem.

I full understand that I need to get these sites off this server but that takes time so in the meantime is there a way we can manually patch autossl so that it doesn't spam Let's Encrypt for new certificates? as this is also causing issues when trying to migrate sites to a new server when we're unable to issue a certificate on that server due to rate limiting!

 

[ciao70]

At this point I know we weren't able to get to it in the next v98 release coming out. However, we're hoping it will get to the one after that. I'll be sure to post once I get more details.

v100 release is out

You have news?

Thanks

 

[cPanelAnthony]

Hello! I will check on this with our team when they are in tomorrow and update this thread.