In Progress [CPANEL-33077] Letsencrypt transition to ISRG’s Root (Important!!!!!)

[monarobase]

The extending is done by including the expired certificate because older android don’t verify if a root certificate is expired, but that expired certificate causes issues with other devices, some security solutions block it and cPanel detected the certs as faulty.

We had random issues for different customers running up to date software when that root chain was included. For old android devices I believe that setting e-mail to “TLS accept all certificates” and installing Firefox should resolve the problem for most users. These users will have issues with lots of websites as lots of people will decide to not include the expired root certificate due to compatibility issues with other devices.

 

[[email protected]]

Hello.

The extending is done by including the expired certificate because older android don’t verify if a root certificate is expired, but that expired certificate causes issues with other devices, some security solutions block it and cPanel detected the certs as faulty.

Thank you for that kind of view on this particular problem.

These users will have issues with lots of websites as lots of people will decide to not include the expired root certificate due to compatibility issues with other devices.

That's correct I believe.

You know if we change to autossl cpanel powered by sectigo old devices like android before 7.1.1 will be ok as before or we have problems also?

Because Sectigo-AddTrust-External-CA-Root-Expired in May-30-2020 and I wonder if also Sectigo has the same exactly problem that we face with letsencrypt and for someone to change and make autossl from sectigo will have the same result...

For my point of view it's better stay with letsencrypt and go on without include the expired certificate. But in that case many clients that have website traffic from older android devices will be disappointed of losing views or sales from that particular audience.

But in the other hand I'm not sure if we change to autossl powered by sectigo will solve that problem. It's a bit of mess...

 

[ciao70]

The extending is done by including the expired certificate because older android don’t verify if a root certificate is expired, but that expired certificate causes issues with other devices, some security solutions block it and cPanel detected the certs as faulty.

We had random issues for different customers running up to date software when that root chain was included. For old android devices I believe that setting e-mail to “TLS accept all certificates” and installing Firefox should resolve the problem for most users. These users will have issues with lots of websites as lots of people will decide to not include the expired root certificate due to compatibility issues with other devices.

Hello,

What would be the devices that have problems? Cpanel doesn't seem to me to have communicated that it would remove DST Root.
As I wrote previously, we re-entered DST Root and disabled Autossl. I expect some press release from Cpanel to explain the situation

Thanks

 

[ciao70]

Hello,
As I have the same issue with older devices than Android 7.1.1 get error is possible to describe how we can manage to solve this? As most of us will have the exact same problem.

If we reinstall the certificate after cPanel plugin updated (2 days before - cpanel-letsencrypt-v2-1.02-1.2.1) will correct this problem or we must do something else?

The LetsEncrypt says: Extending Android Device Compatibility for Let's Encrypt Certificates how we can make that happened? cPanel letsencrypt plugin (after the update) it's not possible to insert a certificate that does that?

If we change to Sectigo as autossl provider will fix that kind of problem? The certificate will support android devices older than 7.1.1? Is a workaround or not?

Any suggestion? Thanks in advance!

Hello,

Honestly, I don't know at all, we put the DST Root certificate back manually.
Let's see what Cpanel says about it.

 

[jorbox]

Hello,

Honestly, I don't know at all, we put the DST Root certificate back manually.
Let's see what Cpanel says about it.

can you tell us how ?

 

[[email protected]]

Hello,

Honestly, I don't know at all, we put the DST Root certificate back manually.
Let's see what Cpanel says about it.

First of all thank you for your answer.
Can you tell us how you put DST Root certificate manually and if you face any error with dovecot/exim?

Is from this list?

Chain of Trust - Let's Encrypt

Root Certificates Our roots are kept safely offline. We issue end-entity certificates to subscribers from the intermediates in the next section. For additional compatibility as we submit our new Root X2 to various root programs, we have also cross-signed it from Root X1. Active ISRG Root X1...

letsencrypt.org

Certificate is not trusted on Android

You’re sending the intermediate certificate signed by the ISRG Root X1 root certificate in your certificate chain. While this root certificate is accepted in many modern root certificate stores, it isn’t in older stores like your old Android versions. You should use the intermediate certificate...

community.letsencrypt.org

 

[jorbox]

I think chrome & outlook should include there own list like Firefox do, one of my client same very angry because he uses an old android phone I told him to install firefox and the website worked, I told him also that chrome will release an update for this soon,,

 

[ciao70]

First of all thank you for your answer.
Can you tell us how you put DST Root certificate manually and if you face any error with dovecot/exim?

DST Root only for apache

What I wanted to know from Cpanel is why on Apache we ended up with Certificates provided 2 instead of 3?

DST Root CA X3 was gone

�

Certificates provided	3 (4032 bytes)
Chain issues	None
#2
Subject	R3
Valid until	Mon, 15 Sep 2025 16:00:00 UTC (expires in 3 years and 11 months)
Key	RSA 2048 bits (e 65537)
Issuer	ISRG Root X1
Signature algorithm	SHA256withRSA

#3
Subject	ISRG Root X1
Valid until	Mon, 30 Sep 2024 18:14:03 UTC (expires in 2 years and 11 months)
Key	RSA 4096 bits (e 65537)
Issuer	DST Root CA X3
Signature algorithm	SHA256withRSA

Now all devices even those older than Android 7.1.1 are able to connect again

Sorry for my English

 

[cPRex]

I'm not sure about the cert2 vs 3 issue being mentioned.

I didn't see a lot of updates over the weekend, but at this time we're still looking into the root cause of why some users encountered permission problems in the /var/cpanel/ssl/domain_tls/ directory. Once I hear more about that, I'll be sure to post.

As far as an "announcement" I believe the current plan is to update or create a support article once this is all said and done.

 

[dandadude]

In case this might help anyone:
- I have ran the permission fix 2 days ago but problems have arisen with some domains since then (regarding the common name again of course)
- I ran the permission fix now again and all is well (probably they had new certs generated since then, didn't check)
- I have put the permission fix in my cron.hourly, just in case (until we have a real solution from cPanel)

 

[cPRex]

Here's an article specifically for the permissions issue:

https://support.cpanel.net/hc/en-us/articles/4409996290071

so you can follow along with that for updates.

 

[ciao70]

Here's an article specifically for the permissions issue:

https://support.cpanel.net/hc/en-us/articles/4409996290071

so you can follow along with that for updates.

Requires login to read

 

[cPRex]

@ciao70 - that's correct. Articles that are linked to cases allow the end-user to "follow" the case for updates, which requires a login.

 

[cPRex]

Update - we *potentially* have a cause and a fix for the odd permissions in /var/cpanel/ssl/domain_tls. I can't say more just yet (well, really I don't want to accidentally lie or spread false hope if it turns out to not be true...) but I'm hoping we'll have something official in place soon.

 

[dolphyn]

Here's what I think @ciao70 is suggesting, as a solution to the Android problem, and I think it has worked for me to allow websites to be viewed. (This doesn't address any email or SNI issues such as ERR_CERT_COMMON_NAME_INVALID.)

Simply append the cross-signed ISRG Root X1 certificate (available here) to every "combined" file in /var/cpanel/ssl/apache_tls, then restart Apache.

But I'm worried that this solution might cause other problems, because it's cross-signed with an expired certificate! And we might need a cron job to execute regularly until a better solution is found.

I used this code:

#Download the cross-signed certificate file
cd ~
wget https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem

# Append to each "combined" file (that doesn't already include the cross-signed certificate)
for file in /var/cpanel/ssl/apache_tls/*/combined; do
  if [ ! -z $(grep "CCBEigAwIBAgIQQAF3ITfU6UK47naqPGQK" $file) ]; then
    echo "FOUND in $file"
  else
    echo "" >> $file
    cat ~/isrg-root-x1-cross-signed.pem >> $file
  fi
done

# Graceful Restart Apache
apachectl -k graceful

 

[Duplika]

Is there any news regarding the SSL and Android issue?
Many clients upset, pity cPanel didn't plan ahead on this.

 

[cPRex]

Here is what I know at this point:

-CPANEL-38838 has been created with a possible fix for the permissions problems we've seen with the /var/cpanel/ssl/domain_tls directory
-once that fix is official, we'll backport it to v98 and v94.
-we have an additional case as well that will help with the OpenSSL logic to verify what chains are trusted and what is not
-there is also a Let's Encrypt plugin update that will happen at some point soon

Once all of that happens, we do expect that to help with the Android issues some users have been seeing and to resolve the issues in general.

I'll be sure to post updates as I get them.

 

[sparek-3]

After looking into this, this is what I've been able to find. Someone feel free to correct me if I'm wrong any where in this.

There are two certificates issued by DST Root CA X3 and I don't know if cPanel is aware of this or really what they have done in regards to this.

There is (was) a DST Root CA X3 certificate:

Issuer: DST Root CA X3
Server Name: R3
Expires: September 29, 2021

and there's also a:

Issuer: DST Root CA X3
Server Name: ISRG Root X1
Expires: September 30, 2024

The one that expires on September 30, 2024 is the same one that @dolphyn (post #175) is referencing from https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem

I don't think cPanel realizes that these are two different certificates. They've just decided that all DST Root CA X3 certificates are bad. If you don't want to read any further, that's my best guess as to what happened with cPanel's "fix".

There's also a third certificate:

Issuer: ISRG Root X1
Server Name: R3
Expires: September 15, 2025

This is the primary, and proper CA Bundle that should be used with Let's Encrypt certificates for modern systems.

This is detailed at:

Let's Encrypt DST Root CA X3 expiry Sept 30th 2021 | Certify The Web Docs

Certificate trust mainly relies on the "root" issuing certificate (and intermediate certificates) being trusted by your computer.

docs.certifytheweb.com

if you scroll down to the green box where it says Valid Chains

Now... I'm not sure how cPanel's AutoSSL and installation of certificates really works.

I'm assuming that cPanel's certificate detection method or whatever it's called or how it function - when that detected that a domain's certificate was detected as being issued by Let's Encrypt - it just installed the September 29, 2021 expiring DST Root CA X3 certificate as the CA bundle. Even though this hasn't been the proper instructions from Let's Encrypt for sometime. That's why everyone that used Let's Encrypt AutoSSL found all of their certificates as being expired around September 30th.

Now... cPanel decided to "fix" this by going through and removing all DST Root CA X3 certificates from the CA bundles. And apparently unaware that there are two DST Root CA X3 certificates as I referenced above - they actually removed both - the one that expired on September 29, 2021 and the one that expires on September 30, 2024. Now, I can't be certain, but this may explain why some older Android devices are having trouble connecting securely - the server is missing the DST Root CA X3 certificate that expires on September 30, 2024 that chains back to a trusted certificate. (This would be the Chain 2 legacy chain as referenced in the green Valid Chains box on the certifytheweb.com website I referenced above).

Further from this... and this is really where the "fix" becomes head scratching... if you include the DST Root CA X3 (expiring on September 30, 2024) certificate in the CA Bundle list when installing a certificate with cPanel, then the cPanel install process will fail to create the /var/cpanel/ssl/domain_tls/%domain% directory and populate it with the necessary files. /var/cpanel/ssl/domain_tls/%domain% is what is used by Dovecot and Exim for SNI. Apparently, and I'm just guessing here, cPanel has decided to blacklist any certificate that identifies itself as DST Root CA X3 again apparently unaware that there are two such named certificates and it's just not installing it ... EXCEPT for...

The cPanel process will install the DST Root CA X3 (expires September 30, 2024) for Apache. Thus perpetuating the real head scratching nature of this cPanel "fix"

If the DST Root CA X3 (expires September 30, 2024) certificate is blacklisted by the cPanel installssl function ... why is it being allowed to be installed for Apache?

If the DST Root CA X3 (expires September 30, 2024) certificate is not blacklisted by the cPanel installssl function ... why is /var/cpanel/ssl/domain_tls/%domain% not being populated if it's included?

Now, I can't tell if the cPanel installssl always installed the DST Root CA X3 (expires September 30, 2024) in /var/cpanel/ssl/domain_tls/%domain% or not. I know the procedure I used to issue these certificates (which is not cPanel's AutoSSL) always passed the DST Root CA X3 (expires September 30, 2024) in the CA Bundle to installssl, but maybe cPanel always stripped that out. None of the certificates in /var/cpanel/ssl/domain_tls/%domain% are showing this, BUT... cPanel issued an autofixer that went through and presumably removed all of the DST Root CA X3 certificates (whether it was the one that expired on September 29, 2021 or the one that expires on September 30, 2024) from all combined certificate files on the system. That action has tainted my ability to check past history on this.

I do know that my procedure for issuing and installing certificates (which was passing both the ISRG Root X1 - Expires on September 15, 2025 and DST Root CA X3 - Expires on September 30, 2024 certificates to the CA Bundle paramter of installssl) was working fine up until October 1, 2021. After October 1, 2021 my procedure began to fail to create and populate the /var/cpanel/ssl/domain_tls/%domain% directory because I was passing both the ISRG Root X1 (expires September 15, 2025) and DST Root CA X3 (expires September 30, 2024) certificates in the CA Bundle. When I removed the DST Root CA X3 (expires September 30, 2024) from being installed in the CA Bundle - the /var/cpanel/ssl/domain_tls/%domain% started to be created again and populated correctly.

From my understanding the proper CA Bundle to be installed with Let's Encrypt certificates is to install:

ISRG Root X1 (expires September 15, 2025)

followed by

DST Root CA X3 (expires September 30, 2024)

But maybe I'm wrong there.

At any rate I'd like to know why cPanel isn't allowing me to install the DST Root CA X3 (expires September 30, 2024) certificate in the CA Bundle and have /var/cpanel/ssl/domain_tls/%domain% populated. Should cPanel really be stripping out any information that is passed through the CA Bundle paramter in the installssl function? If you don't want to include a CA Bundle parameter in the installssl function then cPanel can make a guess at what to install. But if something is passed in the CA Bundle parameter, shouldn't that be installed without prejudice? I think that is where a lot of the mess up has been made.

 

[cPRex]

I'm going to be posting a detailed summary soon

 

[cPRex]

Here's a link to the summary of the Let's Encrypt events:

Summary of recent Let's Encrypt issues

Hello everyone! This post serves as a summary of the cPanel, Let's Encrypt, and OpenSSL issues that were experienced the week of October 1. This is going to be a bit long, but there were so many moving parts involved that isn't really avoidable, although I've tried to keep this down to the...

forums.cpanel.net