In Progress [CPANEL-33077] Letsencrypt transition to ISRG’s Root (Important!!!!!)

[Irksome73]

I don't see a reference in this thread to https://support.cpanel.net/hc/en-us/articles/4409770365335 - and I'd also suggest this as it seems a lot of companies have been caught with their pants down https://twitter.com/search?q=%23letsencrypt

As far as I can see this only affects Dovecot, not Exim - we're getting loads of tickets emailed by affected customers ... but know they won't see our email replies!

 

[smurf]

This is what happen when cPanel try to focus on make money....

Fully agree.

It's possible to justify the price rises if the product 'Just works' but events like this show a huge hole in the way cPanel is being run. In the 4 hours since we lodged a priority ticket when this kicked off we've only received one reply from support. And that was over 2 hours ago asking us to check the rpm version of the CA certificates.

Communication is important. We need to know what's going on so we can relay it on to at least attempt to reassure our frustrated clients. Surely someone at cPanel can send out some ticket replies, keep us in the loop etc. Even if it's just a 'We're looking into this'. Having to come to the forum to work out what's happening doesn't exactly match the premium price tag.

 

[Irksome73]

To be fair - I went to raise a ticket and saw the annoucement at the top of the page ... which led me here to discover more.

 

[smurf]

I don't see a reference in this thread to https://support.cpanel.net/hc/en-us/articles/4409770365335 - and I'd also suggest this as it seems a lot of companies have been caught with their pants down https://twitter.com/search?q=#letsencrypt

As far as I can see this only affects Dovecot, not Exim - we're getting loads of tickets emailed by affected customers ... but know they won't see our email replies!

For us at least switching to cPanel Sectigo is not working as their server is not issuing SSL certificates:

The “cPanel (powered by Sectigo)” provider cannot currently accept incoming requests. The system will try again later.

Anyone else seeing this?

 

[mtindor]

I don't see a reference in this thread to https://support.cpanel.net/hc/en-us/articles/4409770365335 - and I'd also suggest this as it seems a lot of companies have been caught with their pants down https://twitter.com/search?q=#letsencrypt

As far as I can see this only affects Dovecot, not Exim - we're getting loads of tickets emailed by affected customers ... but know they won't see our email replies!

Interesting writeup by cPanel. I haven't used cPanel for SSLs (other than primary hostname). I've always used Letsencrypt. If we were to feel we absolutely had to try the cPanel option, is that free? Is there a charge? Do they rate limit?

So many issues combined. You've got the underlying issue that caused the whole mess, whatever that is. And then you have the issue of LetsEncrypt throttling mass cert renewals (such as when runs WHM --> AutoSSL --> Run AutoSSL for All Users.

Ok on Dovecot issues but not Exim. I haven't verified that myself. I know that Dovecot is definitely an issue. But I think Dovecot's issue was outlined by you or someone else earlier in this thread or another one -- the place where all the information is held for the dovecot SNI information isn't getting populated with the information from the new certs that were in fact renewed without error.

Mike

 

[Jcats]

You can't switch to cPanel for SSL

The “cPanel (powered by Sectigo)” provider cannot currently accept incoming requests. The system will try again later.

 

[cPRex]

As several of you have seen, the cPanel Sectigo provider is currently overwhelmed due to this issue. However, these errors will be intermittent as it is based on capcity. Switching to the cPanel SSL provider is free, and you can see the limits here:

Manage AutoSSL | cPanel & WHM Documentation

This interface allows you to manage the AutoSSL feature, which automatically installs domain-validated SSL certificates for the Apache®, Dovecot, Exim, Web Disk, and cPanel Server services for users' domains.

docs.cpanel.net

It's also important to note that the overall issue is only effecting Let's Encrypt, which isn't something that is provided by cPanel, and has a separate terms of service as outlined here:

Manage AutoSSL | cPanel & WHM Documentation

This interface allows you to manage the AutoSSL feature, which automatically installs domain-validated SSL certificates for the Apache®, Dovecot, Exim, Web Disk, and cPanel Server services for users' domains.

docs.cpanel.net

We are currently sending the following message to users that are submitting tickets to our team as it is the most effective workaround at this time:

Thank you for your patience. We are currently investigating this issue and are tracking it internally as UPS-403.

We will be publishing more information here:

https://support.cpanel.net/hc/en-us/articles/4409770365335

This is related to the recent expiration of the DST Root CA X3 Cert from Let's Encrypt. We believe this to be causing issues with the SNI configuration.

We are currently working with our developers on a more permanent solution that would correct the certificates already installed on the server. Once this is complete the page above will be updated.

However, if absolutely required you can bypass these errors by switching to using the cPanel Store as the AutoSSL certificate provider and issuing new certificates.

Running this command below will set cPanel as the AutoSSL provided and then run a check for all of the domains on the server:

whmapi1 set_autossl_provider provider='cPanel' ; /usr/local/cpanel/bin/autossl_check -all

If you have any questions, or if there is anything else we can assist you with, please let us know. We would be glad to help!

 

[mtindor]

@cPRex I followed your instructions for switching to cPanel/Sectigo and it went through everything successfully (or appears to have).

The issue is, Dovecot is not making use of any of those certs like it should be. So POP3/IMAP-over-SSL using the customer's mail.customerdomain.com is not working.

Mike

 

[tui]

It's also important to note that the overall issue is only effecting Let's Encrypt, which isn't something that is provided by cPanel, and has a separate terms of service as outlined here:

Yes... the root problem is Let's Encrypt, but this is nothing new as at least you (cPanel) where awarded about this change since more than one year, you had more than one year to investigate, update and push a update in time, not today that was the end day, instead of that you where focused on making ugly themes and increasing prices for that themes, in this thread there are so many questions, post and updates about this since past year and you ignore them... so the problem is caused by you and your lack of focusing in important things, if you where able to pick this in time nobody using cpanel would have this problem, why im not having this problem with my servers that dont use cpanel? thats because we prepared our servers for this in advance, so dont wash your hands saying that is Let's Encrypt problem, face up and accept that it is a problem for not doing your work and anticipating this at the time, you had more than a year to do what you are doing now

 

[cPRex]

@mtindor - I would expect that a certificate that was reinstalled would work normally with all services, as that would trigger the service to rebuild and restart. Just as a test, could you try manually restarting Dovecot to see if that gets things working?

If not, feel free to submit a ticket to our team so we can take a look.

 

[Irksome73]

I've done some further testing ... Exim is affected as well as Dovecot ... sorry for my mis-information previously.

@cPRex is a patch likely in the next few hours or do we need to switch to Sectigo and loose the wildcards some clients rely on?

 

[cPRex]

We are working on a more permanent patch right now, and it is in review. It will fix existing certificates, but we're actually seeing that Let's Encrypt still issuing new certificates that are having issues. So even once our fix is applied, we can't guarantee everything will work properly as some of it is still out of our control.

Our best recommendation at this time would be to switch to cPanel/Sectigo if it is absolutely critical, or wait for our patch to be released soon. I expect "soon" to mean "some point this evening" although the situation is still developing and it's hard for me to provide an accurate timeframe.

 

[mtindor]

@mtindor - I would expect that a certificate that was reinstalled would work normally with all services, as that would trigger the service to rebuild and restart. Just as a test, could you try manually restarting Dovecot to see if that gets things working?

If not, feel free to submit a ticket to our team so we can take a look.

Before, when I was using Lets Encrypt. It did go through and renew the expired SSLs. But none of them deposited appropriate information in /var/cpanel/ssl/domain_tls

I switched to cPanel/Sectigo. It appeared to only attempt to renew certs that weren't already valid on the system. So all the other ones that were renewed by Lets Encrypt that didn't cause information to be deposited into /var/cpanel/ssl/domain_tls stayed hte same.

I'm on cPanel/Sectigo now. But the only way I could figure out to get a new Sectigo SSL provisioned and resultant info put in /var/cpanel/ssl/domain_tls was to log into the user's cPanel account, UNInstall the SSL certificate for the site. Go to WHM --> AutoSSL --> and run AutoSSL for that user. The new cert would eventually be provisioned and information put in /var/cpanel/ssl/domain_tls. And I'm presuming that for those domains that I diid this, email service is probably restored. Yes, I just did this and verified that email service for that domain is restored.

So must be go into every cPanel account and (a) manually UNinstall every SSL certificate, (b) go and renew those certs in AutoSSL one at time for everything to work?

Mike

 

[cPRex]

That's part of the reason we're encouraging people to wait for an official fix - we're getting some reports of AutoSSL not properly replacing certificates, maybe due to ratelimiting, or maybe due to other issues.

 

[mtindor]

That's part of the reason we're encouraging people to wait for an official fix - we're getting some reports of AutoSSL not properly replacing certificates, maybe due to ratelimiting, or maybe due to other issues.

So if the automated process (WHM -- AutoSSL -- Run AutoSSL for all users) doesn't complete because of throttling by LetsEncrypt (or even Sectigo), then I'm guessing technically the "followthrough" doesn't occur where it sets all of the certificates up for email usage? Hmm. And if there are truly legitimate SSL certs on the server (supposedly I have a ton for a ton sites that do not show any issues in a web browser) for which there is no data in /var/cpanell/ssl/domain_tls . And apparently that's why there is no SSL for mail.thatdomain.com for email use.

Ok I guess I shall wait. In the meantime, for thoroughly pissed off customers, I'm UNinstalling their SSL via their cPanel interface and then having AutoSSL reprovision it (which seems to do the trick).

Mike

 

[ffeingol]

Ok I guess I shall wait. In the meantime, for thoroughly pissed off customers, I'm UNinstalling their SSL via their cPanel interface and then having AutoSSL reprovision it (which seems to do the trick).

That "may" backfire on you also, as you may run into domain or IP level rate limiting.

 

[mtindor]

That "may" backfire on you also, as you may run into domain or IP level rate limiting.

And it does / did backfire. A full run on all users definitely gets rate-limited with LetsEncrypt, didn't get throttled with Sectigo (but only had a couple invalid SSL to be reprovisioned). For single-user attempts, you are right -- Lets Encrypt is rate-limiting even those. I'm sure it is based upon traffic from the server IP. For Sectigo it's hit-or-miss. Works 1 or 2 times out of every three that I've tried so far. For the ones it didn't work for, it claims the server will re-attempt the provisioning at a later time.

Still, with this method, I was able to get 5 or 6 customers (the most vocal, who called in) taken care of. I'm going to try to take care of the remaining squeaky wheels in this fashion. Then wait for a 'fix' for the rest.

Of course, as long as Lets Encrypt and Sectigo are swamped / rate limiting, even when a "fix" comes out to correct all this, Lets Encrypt and Sectigo will be swamped and rate-limiting the hell out of us and that will end up being the hopefully final (and painful) issue to deal with.

 

[Duplika]

Was there something cPanel could have done before to avoid this?

Not sure how other control panels, even Plesk, managed to sort this changes with Let's Encrypt.

 

[mysterygang]

How about just run
yum update -y ca-certs
?
What do you think?

 

[brt]

If it fails to renew, I don't understand why it's not being added to the "Pending Queue". It appears that some domains just flat out fail and are never tried again. Clearly the Pending Queue exists, but doesn't function properly.

So glad to have the new f***ing themes though. Glad that's where our ever-increasing money is going.