In Progress [CPANEL-33077] Letsencrypt transition to ISRG’s Root (Important!!!!!)

Irksome73

Member
Oct 17, 2013
19
1
53
cPanel Access Level
Root Administrator
  • Like
Reactions: eva2000

smurf

Well-Known Member
Jun 4, 2009
56
10
58
This is what happen when cPanel try to focus on make money....
Fully agree.

It's possible to justify the price rises if the product 'Just works' but events like this show a huge hole in the way cPanel is being run. In the 4 hours since we lodged a priority ticket when this kicked off we've only received one reply from support. And that was over 2 hours ago asking us to check the rpm version of the CA certificates.

Communication is important. We need to know what's going on so we can relay it on to at least attempt to reassure our frustrated clients. Surely someone at cPanel can send out some ticket replies, keep us in the loop etc. Even if it's just a 'We're looking into this'. Having to come to the forum to work out what's happening doesn't exactly match the premium price tag.
 

smurf

Well-Known Member
Jun 4, 2009
56
10
58
I don't see a reference in this thread to https://support.cpanel.net/hc/en-us/articles/4409770365335 - and I'd also suggest this as it seems a lot of companies have been caught with their pants down https://twitter.com/search?q=#letsencrypt

As far as I can see this only affects Dovecot, not Exim - we're getting loads of tickets emailed by affected customers ... but know they won't see our email replies!
For us at least switching to cPanel Sectigo is not working as their server is not issuing SSL certificates:

Code:
The “cPanel (powered by Sectigo)” provider cannot currently accept incoming requests. The system will try again later.
Anyone else seeing this?
 

mtindor

Well-Known Member
Sep 14, 2004
1,454
110
193
inside a catfish
cPanel Access Level
Root Administrator
I don't see a reference in this thread to https://support.cpanel.net/hc/en-us/articles/4409770365335 - and I'd also suggest this as it seems a lot of companies have been caught with their pants down https://twitter.com/search?q=#letsencrypt

As far as I can see this only affects Dovecot, not Exim - we're getting loads of tickets emailed by affected customers ... but know they won't see our email replies!
Interesting writeup by cPanel. I haven't used cPanel for SSLs (other than primary hostname). I've always used Letsencrypt. If we were to feel we absolutely had to try the cPanel option, is that free? Is there a charge? Do they rate limit?

So many issues combined. You've got the underlying issue that caused the whole mess, whatever that is. And then you have the issue of LetsEncrypt throttling mass cert renewals (such as when runs WHM --> AutoSSL --> Run AutoSSL for All Users.

Ok on Dovecot issues but not Exim. I haven't verified that myself. I know that Dovecot is definitely an issue. But I think Dovecot's issue was outlined by you or someone else earlier in this thread or another one -- the place where all the information is held for the dovecot SNI information isn't getting populated with the information from the new certs that were in fact renewed without error.

Mike
 

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
807
160
168
New Jersey
cPanel Access Level
DataCenter Provider
You can't switch to cPanel for SSL :rolleyes:

The “cPanel (powered by Sectigo)” provider cannot currently accept incoming requests. The system will try again later.
 
  • Like
Reactions: Solokron

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
12,499
1,971
363
cPanel Access Level
Root Administrator
As several of you have seen, the cPanel Sectigo provider is currently overwhelmed due to this issue. However, these errors will be intermittent as it is based on capcity. Switching to the cPanel SSL provider is free, and you can see the limits here:


It's also important to note that the overall issue is only effecting Let's Encrypt, which isn't something that is provided by cPanel, and has a separate terms of service as outlined here:


We are currently sending the following message to users that are submitting tickets to our team as it is the most effective workaround at this time:

Code:
Thank you for your patience. We are currently investigating this issue and are tracking it internally as UPS-403.

We will be publishing more information here:

https://support.cpanel.net/hc/en-us/articles/4409770365335

This is related to the recent expiration of the DST Root CA X3 Cert from Let's Encrypt. We believe this to be causing issues with the SNI configuration.

We are currently working with our developers on a more permanent solution that would correct the certificates already installed on the server. Once this is complete the page above will be updated.

However, if absolutely required you can bypass these errors by switching to using the cPanel Store as the AutoSSL certificate provider and issuing new certificates.

Running this command below will set cPanel as the AutoSSL provided and then run a check for all of the domains on the server:

whmapi1 set_autossl_provider provider='cPanel' ; /usr/local/cpanel/bin/autossl_check -all

If you have any questions, or if there is anything else we can assist you with, please let us know. We would be glad to help!
 

mtindor

Well-Known Member
Sep 14, 2004
1,454
110
193
inside a catfish
cPanel Access Level
Root Administrator
@cPRex I followed your instructions for switching to cPanel/Sectigo and it went through everything successfully (or appears to have).

The issue is, Dovecot is not making use of any of those certs like it should be. So POP3/IMAP-over-SSL using the customer's mail.customerdomain.com is not working.

Mike
 

tui

Well-Known Member
Jun 15, 2007
144
40
78
Mexico
cPanel Access Level
Root Administrator
It's also important to note that the overall issue is only effecting Let's Encrypt, which isn't something that is provided by cPanel, and has a separate terms of service as outlined here:
Yes... the root problem is Let's Encrypt, but this is nothing new as at least you (cPanel) where awarded about this change since more than one year, you had more than one year to investigate, update and push a update in time, not today that was the end day, instead of that you where focused on making ugly themes and increasing prices for that themes, in this thread there are so many questions, post and updates about this since past year and you ignore them... so the problem is caused by you and your lack of focusing in important things, if you where able to pick this in time nobody using cpanel would have this problem, why im not having this problem with my servers that dont use cpanel? thats because we prepared our servers for this in advance, so dont wash your hands saying that is Let's Encrypt problem, face up and accept that it is a problem for not doing your work and anticipating this at the time, you had more than a year to do what you are doing now
 
  • Like
Reactions: eva2000

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
12,499
1,971
363
cPanel Access Level
Root Administrator
@mtindor - I would expect that a certificate that was reinstalled would work normally with all services, as that would trigger the service to rebuild and restart. Just as a test, could you try manually restarting Dovecot to see if that gets things working?

If not, feel free to submit a ticket to our team so we can take a look.
 

Irksome73

Member
Oct 17, 2013
19
1
53
cPanel Access Level
Root Administrator
I've done some further testing ... Exim is affected as well as Dovecot ... sorry for my mis-information previously.

@cPRex is a patch likely in the next few hours or do we need to switch to Sectigo and loose the wildcards some clients rely on?
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
12,499
1,971
363
cPanel Access Level
Root Administrator
We are working on a more permanent patch right now, and it is in review. It will fix existing certificates, but we're actually seeing that Let's Encrypt still issuing new certificates that are having issues. So even once our fix is applied, we can't guarantee everything will work properly as some of it is still out of our control.

Our best recommendation at this time would be to switch to cPanel/Sectigo if it is absolutely critical, or wait for our patch to be released soon. I expect "soon" to mean "some point this evening" although the situation is still developing and it's hard for me to provide an accurate timeframe.
 

mtindor

Well-Known Member
Sep 14, 2004
1,454
110
193
inside a catfish
cPanel Access Level
Root Administrator
@mtindor - I would expect that a certificate that was reinstalled would work normally with all services, as that would trigger the service to rebuild and restart. Just as a test, could you try manually restarting Dovecot to see if that gets things working?

If not, feel free to submit a ticket to our team so we can take a look.
Before, when I was using Lets Encrypt. It did go through and renew the expired SSLs. But none of them deposited appropriate information in /var/cpanel/ssl/domain_tls

I switched to cPanel/Sectigo. It appeared to only attempt to renew certs that weren't already valid on the system. So all the other ones that were renewed by Lets Encrypt that didn't cause information to be deposited into /var/cpanel/ssl/domain_tls stayed hte same.

I'm on cPanel/Sectigo now. But the only way I could figure out to get a new Sectigo SSL provisioned and resultant info put in /var/cpanel/ssl/domain_tls was to log into the user's cPanel account, UNInstall the SSL certificate for the site. Go to WHM --> AutoSSL --> and run AutoSSL for that user. The new cert would eventually be provisioned and information put in /var/cpanel/ssl/domain_tls. And I'm presuming that for those domains that I diid this, email service is probably restored. Yes, I just did this and verified that email service for that domain is restored.

So must be go into every cPanel account and (a) manually UNinstall every SSL certificate, (b) go and renew those certs in AutoSSL one at time for everything to work?

Mike
 

mtindor

Well-Known Member
Sep 14, 2004
1,454
110
193
inside a catfish
cPanel Access Level
Root Administrator
That's part of the reason we're encouraging people to wait for an official fix - we're getting some reports of AutoSSL not properly replacing certificates, maybe due to ratelimiting, or maybe due to other issues.
So if the automated process (WHM -- AutoSSL -- Run AutoSSL for all users) doesn't complete because of throttling by LetsEncrypt (or even Sectigo), then I'm guessing technically the "followthrough" doesn't occur where it sets all of the certificates up for email usage? Hmm. And if there are truly legitimate SSL certs on the server (supposedly I have a ton for a ton sites that do not show any issues in a web browser) for which there is no data in /var/cpanell/ssl/domain_tls . And apparently that's why there is no SSL for mail.thatdomain.com for email use.

Ok I guess I shall wait. In the meantime, for thoroughly pissed off customers, I'm UNinstalling their SSL via their cPanel interface and then having AutoSSL reprovision it (which seems to do the trick).

Mike
 

ffeingol

Well-Known Member
PartnerNOC
Nov 9, 2001
765
310
363
cPanel Access Level
DataCenter Provider
Ok I guess I shall wait. In the meantime, for thoroughly pissed off customers, I'm UNinstalling their SSL via their cPanel interface and then having AutoSSL reprovision it (which seems to do the trick).
That "may" backfire on you also, as you may run into domain or IP level rate limiting.
 

mtindor

Well-Known Member
Sep 14, 2004
1,454
110
193
inside a catfish
cPanel Access Level
Root Administrator
That "may" backfire on you also, as you may run into domain or IP level rate limiting.
And it does / did backfire. A full run on all users definitely gets rate-limited with LetsEncrypt, didn't get throttled with Sectigo (but only had a couple invalid SSL to be reprovisioned). For single-user attempts, you are right -- Lets Encrypt is rate-limiting even those. I'm sure it is based upon traffic from the server IP. For Sectigo it's hit-or-miss. Works 1 or 2 times out of every three that I've tried so far. For the ones it didn't work for, it claims the server will re-attempt the provisioning at a later time.

Still, with this method, I was able to get 5 or 6 customers (the most vocal, who called in) taken care of. I'm going to try to take care of the remaining squeaky wheels in this fashion. Then wait for a 'fix' for the rest.

Of course, as long as Lets Encrypt and Sectigo are swamped / rate limiting, even when a "fix" comes out to correct all this, Lets Encrypt and Sectigo will be swamped and rate-limiting the hell out of us and that will end up being the hopefully final (and painful) issue to deal with.
 
  • Like
Reactions: Kent Brockman

brt

Well-Known Member
Jul 9, 2015
105
10
68
US
cPanel Access Level
Root Administrator
If it fails to renew, I don't understand why it's not being added to the "Pending Queue". It appears that some domains just flat out fail and are never tried again. Clearly the Pending Queue exists, but doesn't function properly.

So glad to have the new f***ing themes though. Glad that's where our ever-increasing money is going.
 
  • Like
Reactions: SupraMario