SSL Notifications in cPanel 68

[rodpascoe]

[Moderator Note]

Here's the most recent update on this topic for anyone visiting this thread for the first time:

The expiry notification system is separate from the AutoSSL system so the confusion is understandable. This system is responsible for sending expiry notifications for all certificate types. The tweak setting disables the expiry notifications system (SSL::CertificateExpiring and AutoSSL::CertificateExpiring - except for related DCV problems).

The following command will disable the expiry notification system:

whmapi1 set_tweaksetting key=notify_expiring_certificates value=0

Its possible the cause of the unexpected notifications is the AutoSSL system sending them when a domain is failing DCV and is affecting the ability for it to renew before the expiry (AutoSSL::CertificateExpiring - when there are related DCV problems or AutoSSL::CertificateRenewalCoverage).

We opened up case CPANEL-16927 to move the all the expiry and related notifications for AutoSSL certificates to be controlled by the same options that were added in CPANEL-16842 (not yet released). Hopefully, this will reduce the confusion created by having two places where the notifications are controlled.

CPANEL-16842 shipped in 68.0.14 with these changes:

• AutoSSL options area will handle server-wide control for sending notifications for AutoSSL certificates except expiry. (AutoSSL::CertificateInstalled, AutoSSL::CertificateRenewalCoverage, and SSL::CertificateExpiring - when there are related DCV problems)
  
• If the notifications are enabled in the AutoSSL options area users will retain the option to disable them in cPanel. Once available the following command line options will be able to disable the notifications server-wide:
  • Turn off all the AutoSSL notifications and prevent AutoSSL from replacing invalid or expiring non-AutoSSL certificates:

    whmapi1 set_autossl_metadata metadata_json='{"clobber_externally_signed":0,"notify_autossl_expiry_coverage":0,"notify_autossl_renewal_coverage":0,"notify_autossl_renewal":0}'

  • Turn off all the AutoSSL notifications and allow AutoSSL to replace invalid or expiring non-AutoSSL certificates (not recommended):

    whmapi1 set_autossl_metadata metadata_json='{"clobber_externally_signed":1,"notify_autossl_expiry_coverage":0,"notify_autossl_renewal_coverage":0,"notify_autossl_renewal":0}'

When CPANEL-16927 is completed in a coming v70 release:

• Tweak Settings option will control sending notification non-AutoSSL certificates (SSL::CertificateExpiring) [Note: If AutoSSL is disabled we treat all certificates as non-AutoSSL certificates]
• AutoSSL options area will handle control for sending notifications for AutoSSL certificates. (AutoSSL::CertificateInstalled, AutoSSL::CertificateRenewalCoverage, AutoSSL::CertificateExpiryCoverage [partial DCV failure - NEW] and AutoSSL::CertificateExpiring [full DCV failure])
• We have also added some language in the WHM Contact Manager to clarify that the settings control which notifications the server administrator receives and where to adjust the settings for a cPanel user (in Contact Information)
• If the notifications are enabled in the AutoSSL options area users will retain the option to disable them in cPanel and administrators will have the option to disable them in the WHM Contact Manager

When CPANEL-16928 is completed in a coming v70 release:

• We are adding additional granularity to control to the AutoSSL::CertificateInstalled notification as AutoSSL::CertificateInstalledCovergeReduced [New] and
  AutoSSL::CertificateInstalledUncoveredDomains [NEW]
  for administrators who want to disable the AutoSSL::CertificateInstalled success notifications. This allows administrators to reduce the number of notifications but still stay informed when a certificate that reduces the SSL coverage is installed. This is an important distinction since this usually means that a DCV problem was not corrected in time to prevent interruption of service by having an expected domain removed from the certificate.

[End Moderator Note]


Hello, I hope someone can help me.

I upgraded to cPanel 68 and the instant I did so (and every day since) all my users have started receiving autoSSL error emails like this one :-

exampledomain.co.uk: The AutoSSL certificate expires on 2017-05-11 at 00:00:00 UTC. At the time of this notice, the certificate expired “159 days, 19 hours, 55 minutes, and 48 seconds” ago.
AutoSSL did not renew the certificate for “exampledomain.co.uk”. You must take action to keep this site secure.

The “cPanel” AutoSSL provider could not renew the SSL certificate without a reduction of coverage because of the following problems:

 webdisk.exampledomain.co.uk [ Last AutoSSL Run at “2017-10-16 at 23:54:07 UTC” ]
The system queried for a temporary file at “http://webdisk.exampledomain.co.uk/.well-known/pki-validation/C14A94680F46EA0B29D3DF1E93E14EFC.txt”, but the web server responded with the following error: 404 (Not Found). A DNS (Domain Name System) or web server misconfiguration may exist. The domain “webdisk.exampledomain.co.uk” resolved to an IP address “91.210.235.75” that does not exist on this server.

This is only part of the email, it's a long email listing failures for every cname like ftp, web disk etc.

I am getting loads of support tickets asking what the hell is going on as users don't understand the email and for the life of me I cannot find how to disable these emails.

This is where the cPanel/WHM documentation pages really let customers down, they are so difficult to navigate and find anything in, they really need an overhaul as the current plain text 1990's looking system just doesn't help anymore.

 

[quarterstaff]

Is there an answer for this one? This is a big problem. I too am starting to get panic support calls and email from everyone hosted on my server. We need the ability to turn off those emails, or find out why this is happening....

 

[quarterstaff]

I was also seeing expired certs even though they are up to date in the backend. I turned off cachewall (xvarnish) and *poof* they are back. Likely related. rodpascoe - are you running varnish?

 

[cPWilliamL]

Please check the release notes for 68 below:
68 Release Notes - Version 68 Documentation - cPanel Documentation

SSL and AutoSSL certificate renewal, expiry, failure, and success notifications
In cPanel & WHM version 68, by default, the system automatically sends users notifications about the status of SSL and AutoSSL certificates. These notifications include useful information and URLs users can access to correct a problem. You can enable or disable the following notifications:

In WHM's Contact Manager interface (WHM >> Home >> Server Contacts >> Contact Manager):

• AutoSSL certificates expiring — An account's AutoSSL certificate expires soon.
• Installation of AutoSSL certificates — AutoSSL installed an SSL certificate.
• Installation of purchased SSL certificates — The system installed SSL certificates that a user purchased through the cPanel Market.
• SSL Certificate Expiration — A service-level SSL certificate has expired.
• SSL Certificate Expires Soon — An account's SSL certificate expires soon.
• SSL certificates expiring — An account's SSL certificate expires soon.

In cPanel's Contact Information interface (cPanel >> Home >> Preferences >> Contact Information):

• AutoSSL has renewed a certificate — AutoSSL successfully completed a certificate renewal.
• AutoSSL certificate expiry — An AutoSSL certificate will expire soon.
• SSL certificate expiry — A non-AutoSSL certificate will expire soon.

Did you check the Contact Manager in WHM?

 

[quarterstaff]

OK, so which one do I turn off for this message:

The “cPanel” AutoSSL provider could not renew the SSL certificate without a reduction of coverage because of the following problems...

I don't see an option for that one.

 

[rodpascoe]

Thanks, I've disabled the option now.

You might want to do a check when a server is upgraded as it sent thousands of emails for certificates that expired months ago.

Perhaps it might be an idea to leave this disabled and allow server owners to make their own choice about what gets sent automatically.

 

[RobinMiller]

I'm also seeing this problem after the upgrade to cPanel 68. It appears that AutoSSL (using Let's Encrypt) has managed (up to now) to secure a number of cPanel related subdomains e.g. cpanel.user.server.com, webdisk.user.server.com etc which now fail the
/.well-known/acme-challenge/ check process (webdisk, for example, may require a login that is not available to AutoSSL).

In our case these domains aren't really important for the user's SSL Certificate, they can access them securely via the server's address and certificate. I do see that there is an option for the user to exclude them from AutoSSL using the SSL/TLS Status interface, however, I have a lot of users who don't understand what they're seeing and it would be helpful if there was a global interface where I could set which of the cPanel subdomains are included in the AutoSSL process for all users.

 

[rodpascoe]

I can't understand how a company as massive as cPanel with installs of their product in the millions worldwide (new domain created every six seconds according to their website) can't test adequately before releasing a change in functionality like this.

These forums are littered with threads like this one where something totally preventable with more testing has happened and caused a problem on real world servers.

 

[Infopro]

totally preventable with more testing

If you're running EDGE or CURRENT you might expect some issues.

 

[rodpascoe]

If you're running EDGE or CURRENT you might expect some issues.

Why? Why would I "expect some issues" on the CURRENT release?

CURRENT This version is tested and verified

Direct quote from your documentation at Product Versions and the Release Process - cPanel Knowledge Base - cPanel Documentation

 

[Infopro]

You've placed too much emphasis on the words tested and verified.

CURRENT is a Release Candidate.

Release Candidate (RC)
A stage of the software release cycle, in which feature development is complete and the software passes all known tests. We stage Release Candidates to become the next Production, or General Availability, Release. Release Candidate software may experience limited real-world testing.

EDGE is in Perpetual Development.

Perpetual Development
A stage of the software release cycle, in which software remains at the alpha or beta development stage for an indefinite period of time. Developers often use this stage to release new features early in order to encourage wide-spread testing and feedback by early adopters. Perpetual development software is not subject to the same quality standards as General Availability releases.

 

[rodpascoe]

You've placed too much emphasis on the words tested and verified.

CURRENT is a Release Candidate.


EDGE is in Perpetual Development.

So basically you're saying that although you say on your page I linked to above (which WHM itself links to from it's interface) "This version is tested and verified" on CURRENT you actually don't mean it?

You say on the same page EDGE is the only one not recommended on production servers, you're now implying CURRENT is too?

It's all very well you quoting that text above about what you consider current to be but you don't have that text on the page you give us to let us make the choice above what release we use.

Once again I'll post that link here :-

Product Versions and the Release Process - cPanel Knowledge Base - cPanel Documentation

That is the link you get to from within WHM when you go for help in choosing the level you're on.

It says in plain English that CURRENT is tested and verified. It does NOT say any of the things you're now saying.

Just admin your documentation is wrong/lacking and get it sorted rather than arguing semantics in a forum.

 

[Infopro]

I'm not arguing with you. When you go to your WebHost Manager to select your tier, you'll note the links to the right of each one.

Click them to be taken to the cPanel glossary page where I got the quotes above.



I hope this helps!

 

[Agics]

Beside who's fault this is, I have to say it took me a lot of work to get this resolved, basically logging in into every cpanel account on my servers and disable the notifications.
Yesterday I tried to get rid of the source of the messages. The messages vary from 403 access errors on the .well-known directory to resolve errors to the cpanel. subdomain :-S and "Size body exceeds..." errors. Removing manually the .well-known dir seems to solve the issue on some accounts but not all. Some problems disappear and come back after 1 day. The problem often relates to cpanel created subdomains, like "autodiscovery" or "mail" or "ipv6". Look at for example this log. Does not make much sense. The webdisk subdomain is cpanel created and does not have it's own dir path. Still it tries to make a certificate and sends a mail to owner of the account that it fails and looses coverage. (I removed the actual domain)

12:16:11 AM The website “[domain].nl”, owned by “web1153”, has a valid SSL certificate, but additional SSL coverage may be possible for the domain “ipv6.[domain].nl”. The system will attempt to replace this certificate with one that includes this additional domain.
12:16:11 AM WARN The domain “webdisk.[domain].nl” failed domain control validation: The system queried for a temporary file at “https://webdisk.[domain].nl/403.shtml”, which was redirected from “http://webdisk.[domain].nl/.well-known/pki-validation/78B8389E8CB1DFDE9D28D2BAF1D6EAE2.txt”. The web server responded with the following error: 401 (Unauthorized). A DNS (Domain Name System) or web server misconfiguration may exist.
12:16:11 AM WARN The current SSL certificate for “[domain].nl” secures the domain “webdisk.[domain].nl”. However, this domain failed local domain control validation. In order to maintain SSL domain coverage for this domain, the system will not attempt to replace the current certificate.

 

[RobinMiller]

That's actually very helpful, Infopro. I've just taken over running a number of servers and they should be set to "Release" but one of them, the one with problems is set to "Current". I shall manually disable the reports for the affected people and change the update cycle to something more appropriate.

 

[anton_latvia]

Just to follow up. I have disabled the following options:
[WHM - Tweak Settings] Send notifications when certificates approach expiry.
[WHM - Contact Manager] AutoSSL cannot add any additional domains because domains that fail validation exist on current certificate.
[WHM - Contact Manager] AutoSSL certificates expiring
[WHM - Contact Manager] Installation of AutoSSL certificates
[WHM - Contact Manager] Installation of purchased SSL certificates
[WHM - Contact Manager] SSL Certificate Expiration
[WHM - Contact Manager] SSL Certificate Expires Soon
[WHM - Contact Manager] SSL certificates expiring

but some notification emails are still being sent. I have checked and see, that in cPanel for users in [cPanel - Contact Information - Contact preferences] options for AutoSSL, SSL are enabled. Can that be a reason, why those emails are sent?

Question: is there easy option to disable it globally? Is there easy (API?) way to disable this on all accounts? These emails create huge confusion for most of our customers and give troubles to support.

 

[cPanelMichael]

Hello,

but some notification emails are still being sent.

Could you provide details and text from the specific notification that's still sent out?

Question: is there easy option to disable it globally? Is there easy (API?) way to disable this on all accounts? These emails create huge confusion for most of our customers and give troubles to support.

For WHM, the following WHM API 1 functions are available:

WHM API 1 Functions - get_all_contact_importances - Software Development Kit - cPanel Documentation
WHM API 1 Functions - set_application_contact_event_importance - Software Development Kit - cPanel Documentation

For cPanel, the following cPanel API 2 functions are available:

https://documentation.cpanel.net/display/SDK/cPanel+API+2+Functions+-+CustInfo::displaycontactinfo
https://documentation.cpanel.net/display/SDK/cPanel+API+2+Functions+-+CustInfo::savecontactinfo

The following WHM API 1 functions are also helpful for detecting AutoSSL problems:

WHM API 1 Functions - get_autossl_problems_for_user - Software Development Kit - cPanel Documentation
https://documentation.cpanel.net/display/SDK/WHM+API+1+Functions+-+get_autossl_problems_for_domain

Additionally, we should have a new blog post regarding the AutoSSL changes in cPanel version 68 published soon. Once published, you can find it at:

https://blog.cpanel.com/

To update, it's now available at https://blog.cpanel.com/new-ssl-notifications-in-v68/

Feel free to open a support ticket using the link in my signature if you are having trouble determining why AutoSSL is failing for a specific account or domain name so we can take a closer look.

Thank you.

 

[anton_latvia]

Here are screenshots.

Update: listed API function reference does not list parameters for SSL notification setting. Could you ask developers and update it?

 

[cPanelMichael]

Here are screenshots

In cPanel, under "Contact Information", you'd need to disable the AutoSSL notifications. The particular notification referenced in that screenshot is:

"AutoSSL cannot renew a certificate because domains that fail validation exist on the current certificate."

Update: listed API function reference does not list parameters for SSL notification setting. Could you ask developers and update it?

Documentation case DOC-9720 is open for this. I'll update this thread once the changes are published. In the meantime, here's a look at the new parameters:

notify_autossl_expiry
Boolean
Whether to send a notification when AutoSSL certificate expiry.

This parameter defaults to 1.

• 1 — Send notification.
• 0 — Do not send notification.

notify_autossl_expiry_coverage
Boolean
Whether to send a notification when AutoSSL cannot renew a certificate because domains that fail validation exist on the current certificate.

This parameter defaults to 1.

• 1 — Send notification
• 0 — Do not send notification.

notify_autossl_renewal
Boolean
Whether to send a notification when AutoSSL renews a certificate.

This parameter defaults to 1.

• 1 — Send notification.
• 0 — Do not send notification

notify_autossl_renewal_coverage
Boolean
Whether to send a notification when AutoSSL cannot add any additional domains because domains that fail validation exist on the current certificate.

This parameter defaults to 1.

• 1 — Send notification.
• 0 — Do not send notification

Update: The following document is now updated to include the additional parameters for the corresponding cPanel API 2 function:

cPanel API 2 Functions - CustInfo::savecontactinfo - Software Development Kit - cPanel Documentation

Thank you.

 

[MACscr]

Has anyone scripted anything up yet to disable these notifications server wide and by default for new accounts? Definitely a pain right now for me.