HTTP and DNS DCV checks failing

Operating System & Version
CloudLinux 6.10
cPanel & WHM Version
v86.0.21

Bretas

Active Member
PartnerNOC
Jun 10, 2018
30
7
8
Brazil
cPanel Access Level
Root Administrator
Hello!

We have been receiving the following alert:

"The system failed to acquire a signed certificate from the cPanel Store because of the following error: Neither HTTP nor DNS DCV preflight checks succeeded! "

When I run "/usr/local/cpanel/bin/checkallsslcerts", this was returned:

Code:
Setting up HTTP DCV (/var/www/html/.well-known/pki-validation/75D9F3BFE3E9D15ECC3931CB7A0F253F.txt) …
        … complete.
Setting up DNS DCV (CNAME _75d9f3bfe3e9d15ecc3931cb7a0f253f.[SERVER_HOSTNAME.OUR_DOMAIN.COM]) …
        … complete.
Attempting DNS DCV preflight check …
        FAILED: The DNS DCV check (_75d9f3bfe3e9d15ecc3931cb7a0f253f.[SERVER_HOSTNAME.OUR_DOMAIN.COM] IN CNAME) did not return the expected value (48b9c1eecf281fb400c6ebb605738587.f027e22b1fb6947ddb03aede2d086816.comodoca.com).
Attempting HTTP DCV preflight check …
        FAILED: Cpanel::Exception/(XID 3yfpvp) The system queried for a temporary file at “http://[SERVER_HOSTNAME.OUR_DOMAIN.COM]/.well-known/pki-validation/75D9F3BFE3E9D15ECC3931CB7A0F253F.txt”, but the web server responded with the following error: 406 (Not Acceptable). A DNS (Domain Name System) or web server misconfiguration may exist.
 at /usr/local/cpanel/Cpanel/SSL/DCV.pm line 356.
        Cpanel::SSL::DCV::__ANON__(Cpanel::Exception::HTTP::Server=HASH(0x4791670)) called at /usr/local/cpanel/3rdparty/perl/530/lib/perl5/cpanel_lib/Try/Tiny.pm line 118
        Try::Tiny::try(CODE(0x48cd5c8), Try::Tiny::Catch=REF(0x4170660)) called at /usr/local/cpanel/Cpanel/SSL/DCV.pm line 416
        Cpanel::SSL::DCV::_verify_http("http://[SERVER_HOSTNAME.OUR_DOMAIN.COM]/.well-known/pki-validation"..., "48b9c1eecf281fb400c6ebb605738587f027e22b1fb6947ddb03aede2d086"..., "COMODO DCV", 0, 4, ARRAY(0x4875730)) called at /usr/local/cpanel/Cpanel/SSL/DCV.pm line 261
        Cpanel::SSL::DCV::verify_http_with_dns_lookups("http://[SERVER_HOSTNAME.OUR_DOMAIN.COM]/.well-known/pki-validation"..., "48b9c1eecf281fb400c6ebb605738587f027e22b1fb6947ddb03aede2d086"..., "COMODO DCV", 0, undef) called at /usr/local/cpanel/Cpanel/Market/Provider/cPStore/Utils.pm line 97
        Cpanel::Market::Provider::cPStore::Utils::imitate_http_dcv_check_locally("[SERVER_HOSTNAME.OUR_DOMAIN.COM]", ".well-known/pki-validation/75D9F3BFE3E9D15ECC3931CB7A0F253F.txt", "48b9c1eecf281fb400c6ebb605738587f027e22b1fb6947ddb03aede2d086"...) called at /usr/local/cpanel/Cpanel/cPStore/HostnameCert/DCV.pm line 193
        eval {...} called at /usr/local/cpanel/Cpanel/cPStore/HostnameCert/DCV.pm line 189
        Cpanel::cPStore::HostnameCert::DCV::set_up("-----BEGIN CERTIFICATE REQUEST-----\x{a}MIICpDCCAYwCAQAwJjEkMCIGA"...) called at /usr/local/cpanel/Cpanel/cPStore/HostnameCert.pm line 172
        Cpanel::cPStore::HostnameCert::_request_new_certificate(Cpanel::cPStore::HostnameCert=HASH(0x38ca100)) called at /usr/local/cpanel/Cpanel/cPStore/HostnameCert.pm line 142
        Cpanel::cPStore::HostnameCert::get_hostname_cert_from_store(Cpanel::cPStore::HostnameCert=HASH(0x38ca100)) called at bin/checkallsslcerts.pl line 542
        bin::checkallsslcerts::_get_certificate_pem_from_store(bin::checkallsslcerts=HASH(0x3188118)) called at bin/checkallsslcerts.pl line 464
        bin::checkallsslcerts::__ANON__() called at /usr/local/cpanel/3rdparty/perl/530/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
        eval {...} called at /usr/local/cpanel/3rdparty/perl/530/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
        Try::Tiny::try(CODE(0x36a7908), Try::Tiny::Catch=REF(0x3828ff0)) called at bin/checkallsslcerts.pl line 468
        bin::checkallsslcerts::_replace_cert_with_ca_signed_cert_from_cpstore(bin::checkallsslcerts=HASH(0x3188118), "cpanel") called at bin/checkallsslcerts.pl line 320
        bin::checkallsslcerts::_check_notify_and_auto_renew_cert_for_service(bin::checkallsslcerts=HASH(0x3188118), "cpanel") called at bin/checkallsslcerts.pl line 86
        bin::checkallsslcerts::run(bin::checkallsslcerts=HASH(0x3188118)) called at bin/checkallsslcerts.pl line 50
Undoing HTTP DCV setup …
        … complete.
Undoing DNS DCV setup …
        … complete.
[WARN] The system failed to acquire a signed certificate from the cPanel Store because of the following error: Neither HTTP nor DNS DCV preflight checks succeeded!
The system will check for the certificate for the “dovecot” service.
The system will attempt to verify that the certificate for the “dovecot” service is still valid using OCSP (Online Certificate Status Protocol).
The “dovecot” service’s current certificate comes with the server’s cPanel license. This certificate expires in less than 25 days. The system will attempt to renew and install a new certificate to the “dovecot” service and any other services that use the old certificate.
The system will attempt to install a certificate for the “dovecot” service from the system ssl storage.
None of the certificates in the system ssl storage were acceptable to use for the “dovecot” service.
The system will check for the certificate for the “exim” service.
The system will attempt to verify that the certificate for the “exim” service is still valid using OCSP (Online Certificate Status Protocol).
The “exim” service’s current certificate comes with the server’s cPanel license. This certificate expires in less than 25 days. The system will attempt to renew and install a new certificate to the “exim” service and any other services that use the old certificate.
The system will attempt to install a certificate for the “exim” service from the system ssl storage.
None of the certificates in the system ssl storage were acceptable to use for the “exim” service.
The system will check for the certificate for the “ftp” service.
The system will attempt to verify that the certificate for the “ftp” service is still valid using OCSP (Online Certificate Status Protocol).
The “ftp” service’s current certificate comes with the server’s cPanel license. This certificate expires in less than 25 days. The system will attemptto renew and install a new certificate to the “ftp” service and any other services that use the old certificate.
The system will attempt to install a certificate for the “ftp” service from the system ssl storage.
None of the certificates in the system ssl storage were acceptable to use for the “ftp” service.
The server doesn't have an IPv6 (nor an AAAA record as its FQDN address).

It doesn't come as a surprise that DNS DCV failed given that this server's hostname is a subdomain to an external domain, but HTTP DCV should be working. This server recently had its mod_ruid2 and mod_cgi removed as its Apache Prefork MPM was replaced with Worker and HTTP/2 enabled, I though this might be related.

When querying that file using the methods GET and HEAD, it returns error 404 instead of 406 as seen in the error message.

Code:
$ curl -I -X GET https://[SERVER_HOSTNAME.OUR_DOMAIN.COM]/.well-known/pki-validation/75D9F3BFE3E9D15ECC3931CB7A0F253F.txt
HTTP/2 404 
date: Tue, 02 Jun 2020 00:34:11 GMT
server: Apache
accept-ranges: bytes
cache-control: no-cache, no-store, must-revalidate
pragma: no-cache
expires: 0
content-type: text/html

$ curl -I -X HEAD https://[SERVER_HOSTNAME.OUR_DOMAIN.COM]/.well-known/pki-validation/75D9F3BFE3E9D15ECC3931CB7A0F253F.txt
HTTP/2 404 
date: Tue, 02 Jun 2020 00:34:16 GMT
server: Apache
accept-ranges: bytes
cache-control: no-cache, no-store, must-revalidate
pragma: no-cache
expires: 0
content-type: text/html
Other than this issue regarding the server's FQDN hostname, no other domain on the server appears to be affected.

There are 3 files in the "/var/www/html/.well-known/pki-validation" dir but the newest of them dates back to June 20th 2019 (around the day when the Hostname's certificate was lastly renewed).

Is either mod_ruid2 or mod_cgi a dependency for HTTP DCV perhaps? If not, why is this happening now? This server has been running fine for about 4 years now and I don't recall any major changes in it over the past 12 months.

Thanks!
 

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
Can you add a test.txt file at /var/www/html/.well-known/pki-validation/ re-run the curl request as follows:

Code:
curl -kvv http://server.hostname.com/.well-known/pki-validation/test.txt
The HTTP DCV check will not function over https and the return that the internal DCV check is getting indicates an invalid Accept header response: 406 Not Acceptable

The DNS DCV check didn't succeed because the DNS record it queried for was not found, this is typical in the instance that you are using 3rd party DNS.

In both circumstances, the setup is cleaned up (meaning the CNAME record and .txt file are removed)
 

Bretas

Active Member
PartnerNOC
Jun 10, 2018
30
7
8
Brazil
cPanel Access Level
Root Administrator
Thanks for the clarification, Lauren!

I followed your instructions and test.txt is being served correctly (HTTP 200):

Code:
$ curl -kvv http://[SERVER_FQDN_HOSTNAME]/.well-known/pki-validation/test.txt
*   Trying [SERVER_IP]:80...
* TCP_NODELAY set
* Connected to [SERVER_FQDN_HOSTNAME] ([SERVER_IP]) port 80 (#0)
> GET /.well-known/pki-validation/test.txt HTTP/1.1
> Host: [SERVER_FQDN_HOSTNAME]
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 03 Jun 2020 04:09:14 GMT
< Server: Apache
< Upgrade: h2,h2c
< Connection: Upgrade
< Last-Modified: Wed, 03 Jun 2020 04:06:15 GMT
< ETag: "82191-5-5a72626329bc0"
< Accept-Ranges: bytes
< Content-Length: 5
< Cache-Control: no-cache, no-store, must-revalidate
< Pragma: no-cache
< Expires: 0
< Content-Type: text/plain
<
TEST
* Connection #0 to host [SERVER_FQDN_HOSTNAME] left intact
Code:
$ ls -laht /var/www/html/.well-known/pki-validation/
total 24K
drwxr-xr-x 2 root root 4.0K Jun  3 01:06 .
-rw-r--r-- 1 root root    5 Jun  3 01:06 test.txt
-rw-r--r-- 1 root root   77 Jun 20  2019 0DF7F3539F05CD000479D58A750007DE.txt
-rw-r--r-- 1 root root   77 Jul 14  2018 D6AFC2E5BF5339023476BDFFB0F4735C.txt
drwxr-xr-x 3 root root 4.0K Aug  7  2017 ..
-rw-r--r-- 1 root root   77 Aug  7  2017 13D967AB73920CBFC5385E16322E24CB.txt
 

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
Try it with the user agent flag now as follows;

Code:
curl -kvv --user-agent "COMODO DCV" --max-time 10 --retry 0 http://[SERVER_FQDN_HOSTNAME]/.well-known/pki-validation/test.txt
Also, let me know if you still get the same error when you run the check again. and if you do does the output of the following come back with your server's IP?

Code:
/scripts/cpdig server.hostname.tld a --verbose
 

Bretas

Active Member
PartnerNOC
Jun 10, 2018
30
7
8
Brazil
cPanel Access Level
Root Administrator
Sorry I'm late! Here's the output of the cURL:

Code:
$ curl -kvv --user-agent "COMODO DCV" --max-time 10 --retry 0 http://[SERVER_FQDN_HOSTNAME]/.well-known/pki-validation/test.txt
*   Trying [SERVER_IP]:80...
* TCP_NODELAY set
* Connected to [SERVER_FQDN_HOSTNAME] ([SERVER_IP]) port 80 (#0)
> GET /.well-known/pki-validation/test.txt HTTP/1.1
> Host: [SERVER_FQDN]
> User-Agent: COMODO DCV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 05 Jun 2020 22:40:59 GMT
< Server: Apache
< Upgrade: h2,h2c
< Connection: Upgrade
< Last-Modified: Wed, 03 Jun 2020 04:06:15 GMT
< ETag: "82191-5-5a72626329bc0"
< Accept-Ranges: bytes
< Content-Length: 5
< Cache-Control: no-cache, no-store, must-revalidate
< Pragma: no-cache
< Expires: 0
< Content-Type: text/plain
< 
TEST
* Connection #0 to host [SERVER_FQDN_HOSTNAME] left intact
Yes, the 406 error is still showing up when running "/usr/local/cpanel/bin/checkallsslcerts".

As for the second command:

Code:
$ /scripts/cpdig [SERVER_FQDN_HOSTNAME] a --verbose
[1591396623] libunbound[3436025:0] notice: init module 0: validator
[1591396623] libunbound[3436025:0] notice: init module 1: iterator
[1591396623] libunbound[3436025:0] info: resolving [SERVER_FQDN_HOSTNAME]. A IN
[1591396623] libunbound[3436025:0] info: priming . IN NS
[1591396623] libunbound[3436025:0] info: error sending query to auth server 2001:500:a8::e port 53
[1591396624] libunbound[3436025:0] info: response for . NS IN
[1591396624] libunbound[3436025:0] info: reply from <.> 192.58.128.30#53
[1591396624] libunbound[3436025:0] info: query response was ANSWER
[1591396624] libunbound[3436025:0] info: response for . NS IN
[1591396624] libunbound[3436025:0] info: reply from <.> 192.112.36.4#53
[1591396624] libunbound[3436025:0] info: query response was ANSWER
[1591396624] libunbound[3436025:0] info: priming successful for . NS IN
[1591396624] libunbound[3436025:0] info: response for [SERVER_FQDN_HOSTNAME]. A IN
[1591396624] libunbound[3436025:0] info: reply from <.> 199.7.83.42#53
[1591396624] libunbound[3436025:0] info: query response was REFERRAL
[1591396624] libunbound[3436025:0] info: response for [SERVER_FQDN_HOSTNAME]. A IN
[1591396624] libunbound[3436025:0] info: reply from <br.> 200.229.248.10#53
[1591396624] libunbound[3436025:0] info: query response was nodata ANSWER
[1591396624] libunbound[3436025:0] info: error sending query to auth server 2001:12f8:a::10 port 53
[1591396624] libunbound[3436025:0] info: response for [SERVER_FQDN_HOSTNAME]. A IN
[1591396624] libunbound[3436025:0] info: reply from <br.> 200.219.148.10#53
[1591396624] libunbound[3436025:0] info: query response was REFERRAL
[1591396624] libunbound[3436025:0] info: resolving ns2.[OUR_DOMAIN]. AAAA IN
[1591396624] libunbound[3436025:0] info: resolving ns.[OUR_DOMAIN]. AAAA IN
[1591396624] libunbound[3436025:0] info: response for [SERVER_FQDN_HOSTNAME]. A IN
[1591396624] libunbound[3436025:0] info: reply from <[OUR_DOMAIN].> [OUR_NS1_IP]#53
[1591396624] libunbound[3436025:0] info: query response was ANSWER
�9��
All IPs being shown are the recursive/authoritative DNS servers through which the query was performed. None of the IPs printed out belong the the affected server.

It's worth noticing that I ran the same command on two other cPanel servers that are not at all affected by this issue (and ran "/usr/local/cpanel/bin/checkallsslcerts" on them to confirm, both finished correctly). As it turns out, one of them also returned "�9��" as a result and the other one finished with its own IP as it was supposed to.

When I call the system DIG command rather than cPanel's cpdig, however, the affected server's IP is comes up correctly:

Code:
$ dig A +short [SERVER_FQDN_HOSTNAME]
[SERVER_MAIN_IP]
Thansk!
 

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
Hi @Bretas

I apologize for my own delay in responding to this. I was unexpectedly out for several days. In this case, I think it is indeed best if you open a ticket if you're still experiencing the issue. This way we can look at the configuration of the server. Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


Thanks!
 

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
I just checked in on this and it looks like there's an analyst currently working on it. I'll keep an eye on it and report back with what he finds
 
  • Like
Reactions: frankbret