SOLVED [CPANEL-30266] AutoSSL did not renew the certificate

[alphatls]

/scripts/cpdig $domain.com A --verbose

for i in {a..m}; do echo -n "$i.root-servers.net: "; dig -4 "$i".root-servers.net @"$i".root-servers.net +short;done

for gtld in {a..m}; do echo Trying $gtld.gtld-servers.net ; dig +trace +short $domain.com @$gtld.gtld-servers.net; done

/usr/local/cpanel/3rdparty/bin/perl -mCpanel::DnsRoots -e 'use Data::Dumper; print Dumper(Cpanel::DnsRoots->new()->get_ipv4_addresses_for_domain(@ARGV[0]));' $domain.com

Was there anything specific you were looking for with these?

I ran these and there were no errors, output was (afaik) as expected.

Ticket created as well 13894889

 

[cPanelLauren]

So, the only way I could make it work is by enabling promisc mode on the vm adapter:
ip link set vmbr1 promisc on

But you were able to get this working locally with your configuration?

 

[cPanelLauren]

Was there anything specific you were looking for with these?

I ran these and there were no errors, output was (afaik) as expected.

Ticket created as well 13894889

There was indeed, the output would have been helpful but I checked in on your ticket and it would appear that the analyst found the NS are not responding with an authoritative response for the domain.

 

[Jean Carlos Lausell]

Hello,

We've confirmed internally that the behavior persisting, in this case, is almost always a misconfiguration of the firewall resulting in incorrectly configured hairpinning. For reference on what this is the wiki may be helpful: Hairpinning - Wikipedia

For some reference you can try the commands I listed earlier in the thread but ultimately this will need to be resolved locally. We recently switched our DNS resolver to unbound which is less fault-tolerant than the internal method we were utilizing previously.

So Cpanel upgraded to a new "less fault-tolerant" without any regard as to how it would impact current customers. And now faced with the backlash the official answer is "ultimately this will need to be resolved locally" to say You the customer (small guys I know) just fix it as we wont do squat... That's the problem when companies monopolize a space. That's a horrible solution, from my particular case: it was working before, now its not. Its a VM running off KVM, 1:1 nat (even disabled all security to test), promiscuous mode and nothing works. Can't straight up install certbot and now left with expired certs, unsecured sites and our customers feeling like we have tricked them. But im sure the $5 dollars you guys saved by moving to a "less fault-tolerant" is totally worth screwing us over.

 

[phrogg]

I was just able to solve this issue on my CentOS 7 server. The only place this was happening for me was on domains where DNS was hosted on the same server. Basically, the AutoSSL check is trying to go out and validate the authoritative DNS server DCV records for the domains. Because I have NAT'ed IP addresses, when it looks up the authoritative DNS server it gets the external IP address and does not get the right answer because there is no hairpin NAT in place on the firewall. My Firewall seems to not support a hairpin NAT configuration.

So, I was able to add another interface to the box via VMware. My primary interface shows up as ens192, now there is a second one as ens224.

in /etc/sysconfig/network-scripts there is a config script for ens192 that I copied over to ens224 and edited. I changed the NAME, DEVICE, IPADDR, GATEWAY, DNS1 and DNS2 settings.

NAME should be the interface name
DEVICE should be the interface name
IPADDR should be the external IP address of your DNS server (obfuscated below)
GATEWAY should be blank
DNS1 should be blank
DNS2 should be blank

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens224"
UUID="5bc98952-958f-4d14-9594-0ef7ea44ead4"
DEVICE="ens224"
ONBOOT="yes"
IPADDR="110.110.110.110"
PREFIX="24"
GATEWAY=""
DNS1=""
DNS2=""
IPV6_PRIVACY="no"

You will then need to do a

[root]# service restart network
[root]# service restart ipaliases

that will restart the services. Verify that the interfaces are up with

ifconfig -a

This creates another interface so that when AutoSSL resolves my DNS server and gets the external IP address, this interface which is local will now answer for it and it will still connect to the DNS service on the box. Basically, I'm creating a hairpin NAT internal on the box.

After doing this, running AutoSSL renewed everything with no issues. You could create a more of these interfaces for any other DNS server addresses you have hosted as well. Hope that helps someone else!

 

[webo3com]

Having the same problem, but I figured out what was causing it...

I'm using clustered cPanel DNS only (bind server) and my cPanel is using another resolv.conf DNS so that if a zone is created or move to another providers, my servers will be able to get the "real" DNS result from the new NS server for that domain.

The problem with this is that queries done for local domains will also past from external DNS service and then the cPanel autossl script will not be able to verify authority on the domain and fail.

I think that a way to by-pass that verification should be implemented. I suppose this is for minimizing the number of queries done to the API of comodo or let's encrypt.

 

[LachM]

Just want to add that we got around this similar to phrogg by either adding the IP(s) that are on the public side of the device performing NAT to the server using /etc/ips in the format $IP:255.255.255.248:$GATEWAY and then running the aforementioned # service restart ipaliases

This adds the IP in the same way adding it through WHM does.

    inet 27.x.x.x/29 brd 27.x.x.x scope global secondary ens192:cp1
       valid_lft forever preferred_lft forever

If you go this route, there's an option tucked away in WHM config to set the default IP for new accounts - make sure it isn't this one, and is instead your NAT IP! Alternatively, you may have luck using iptables to NAT on itself...

/sbin/iptables -t nat -I PREROUTING -p udp -d $PUB_IP -j DNAT --to $PRIV_IP --dport 53 -i $INTERFACE

 

[Joe Gold]

I'm having these same issues on (1) of (4) identical sever configurations. Several different cPanel techs investigated over a 4 day peroid and they kept concluding that it was a network issue with the provider. The provider is AWS and after some investigation I concluded this was surely not the case. I did notice that when I disabled CSF firewall the issues disappear. I was able to fix a few of the issue by whitelisting some root server IP's in CSF but that did not fix the NAT and AutoSSL issues. There are also IP6 issues where users with IP6 IP's are getting blocked. It's extremely odd that only this (1) server out of (4) with identical configurations is suddenly having this issue... It's been a (2) weeks now and dozens of hours of troubleshooting with no solution...

 

[jpvanoosten]

I can confirm I am having similar issues:

# /usr/local/cpanel/bin/autossl_check --user DOMAIN
AutoSSL’s configured provider is “cPanel (powered by Sectigo)”.
This AutoSSL provider does not poll for certificate availability immediately after a certificate request submission. Instead, it submits certificate requests then periodically polls the cPanel Store for each requested certificate and installs it after a successful retrieval. The system will record all requests, retrievals, and installations for the current AutoSSL run in this log.
Analyzing “DOMAIN”’s domains …
        Analyzing “dev.discord.DOMAIN.nl” …
                TLS Status: OK
                Certificate expiry: 4/5/20, 12:00 AM UTC (45.57 days from now)
        Analyzing “discord.DOMAIN.nl” …
                TLS Status: Defective
                Certificate expiry: 2/18/20, 12:00 AM UTC (1.43 days ago)
                Defect: OPENSSL_VERIFY: The certificate chain failed OpenSSL’s verification (0:10:CERT_HAS_EXPIRED).
        Analyzing “forum.DOMAIN.nl” …
                TLS Status: Defective
                Certificate expiry: 2/18/20, 12:00 AM UTC (1.43 days ago)
                Defect: OPENSSL_VERIFY: The certificate chain failed OpenSSL’s verification (0:10:CERT_HAS_EXPIRED).
        Analyzing “DOMAIN.nl” …
                TLS Status: Defective
                Certificate expiry: 2/18/20, 12:00 AM UTC (1.43 days ago)
                Defect: OPENSSL_VERIFY: The certificate chain failed OpenSSL’s verification (0:10:CERT_HAS_EXPIRED).
        Attempting to ensure the existence of necessary CAA records …
                No CAA records were created.
        Verifying 12 domains’ DNS management …
        Verifying “cPanel (powered by Sectigo)”’s authorization on 12 domains via DNS CAA records …
        DNS query error (discord.DOMAIN.nl/NS): (XID 6j97ud) DNS request timeout: discord.DOMAIN.nl/NS
        DNS query error (DOMAIN.nl/NS): (XID jyqr2s) DNS request timeout: DOMAIN.nl/NS
                DNS does not manage “discord.DOMAIN.nl”.
                DNS does not manage “DOMAIN.nl”.
        DNS query error (www.discord.DOMAIN.nl/NS): (XID rumsaw) DNS request timeout: www.discord.DOMAIN.nl/NS
                DNS does not manage “www.discord.DOMAIN.nl”.
        DNS query error (forum.DOMAIN.nl/NS): (XID 3yv4re) DNS request timeout: forum.DOMAIN.nl/NS
                DNS does not manage “forum.DOMAIN.nl”.
        DNS query error (www.forum.DOMAIN.nl/NS): (XID thgt6a) DNS request timeout: www.forum.DOMAIN.nl/NS
                DNS does not manage “www.forum.DOMAIN.nl”.
        DNS query error (www.DOMAIN.nl/NS): (XID btp9c9) DNS request timeout: www.DOMAIN.nl/NS
                DNS does not manage “www.DOMAIN.nl”.
        DNS query error (mail.DOMAIN.nl/NS): (XID rz8m36) DNS request timeout: mail.DOMAIN.nl/NS
                DNS does not manage “mail.DOMAIN.nl”.
        DNS query error (cpanel.DOMAIN.nl/NS): (XID bmy8nn) DNS request timeout: cpanel.DOMAIN.nl/NS
                DNS does not manage “cpanel.DOMAIN.nl”.
        DNS query error (webdisk.DOMAIN.nl/NS): (XID ys9444) DNS request timeout: webdisk.DOMAIN.nl/NS
                DNS does not manage “webdisk.DOMAIN.nl”.
        DNS query error (webmail.DOMAIN.nl/NS): (XID pmbpzc) DNS request timeout: webmail.DOMAIN.nl/NS
                DNS does not manage “webmail.DOMAIN.nl”.
        DNS query error (forum.DOMAIN.nl/CAA): SERVFAIL (2)
        DNS query error (cpcontacts.DOMAIN.nl/NS): SERVFAIL (2)
                DNS does not manage “cpcontacts.DOMAIN.nl”.
        DNS query error (cpcalendars.DOMAIN.nl/NS): SERVFAIL (2)
                DNS does not manage “cpcalendars.DOMAIN.nl”.
                DNS does not manage any of this user’s 12 domains.
        DNS query error (mail.DOMAIN.nl/CAA): SERVFAIL (2)
        DNS query error (www.forum.DOMAIN.nl/CAA): SERVFAIL (2)
        DNS query error (DOMAIN.nl/CAA): SERVFAIL (2)
                CA authorized: “DOMAIN.nl”
                CA authorized: “forum.DOMAIN.nl”
                CA authorized: “mail.DOMAIN.nl”
                CA authorized: “www.forum.DOMAIN.nl”
        DNS query error (www.discord.DOMAIN.nl/CAA): SERVFAIL (2)
        DNS query error (cpanel.DOMAIN.nl/CAA): SERVFAIL (2)
                CA authorized: “cpanel.DOMAIN.nl”
        DNS query error (discord.DOMAIN.nl/CAA): SERVFAIL (2)
                CA authorized: “discord.DOMAIN.nl”
                CA authorized: “www.discord.DOMAIN.nl”
        DNS query error (www.DOMAIN.nl/CAA): SERVFAIL (2)
                CA authorized: “www.DOMAIN.nl”
        DNS query error (cpcontacts.DOMAIN.nl/CAA): (XID 7hdvqg) DNS request timeout: cpcontacts.DOMAIN.nl/CAA
                CA authorized: “cpcontacts.DOMAIN.nl”
        DNS query error (webmail.DOMAIN.nl/CAA): (XID 9dbwtq) DNS request timeout: webmail.DOMAIN.nl/CAA
                CA authorized: “webmail.DOMAIN.nl”
        DNS query error (webdisk.DOMAIN.nl/CAA): (XID ykawaa) DNS request timeout: webdisk.DOMAIN.nl/CAA
                CA authorized: “webdisk.DOMAIN.nl”
        DNS query error (cpcalendars.DOMAIN.nl/CAA): (XID t2fb5h) DNS request timeout: cpcalendars.DOMAIN.nl/CAA
                CA authorized: “cpcalendars.DOMAIN.nl”
                “cPanel (powered by Sectigo)” is authorized to issue certificates for 12 of this user’s 12 domains.
        AutoSSL cannot increase “DOMAIN”’s SSL coverage.

When running the command:

for i in {a..m}; do echo -n "$i.root-servers.net: "; dig -4 "$i".root-servers.net @"$i".root-servers.net +short;done

I get the result:

;; connection timed out; no servers could be reached

for every server.

Same result for this command:

for gtld in {a..m}; do echo Trying $gtld.gtld-servers.net ; dig +trace +short DOMAIN @$gtld.gtld-servers.net; done

;; connection timed out; no servers could be reached

I've disabled csf & lfd on both DNS servers in the DNS cluster but still no joy. I'm out of options.

 

[cPanelLauren]

This is clearly indicating that your server is unable to reach root nameservers, not just the AutoSSL scan but the subsequent queries you made, can you resolve any outbound domains? What is the output of the following:

sudo nmap -sT -sU <YOURIPADDRESS.or.HOSTNAME> -p 53,80,443

What is in your resolv.conf file?

What is the configuration on the server in terms of NAT routing?

 

[jpvanoosten]

This is clearly indicating that your server is unable to reach root nameservers, not just the AutoSSL scan but the subsequent queries you made, can you resolve any outbound domains? What is the output of the following:

sudo nmap -sT -sU <YOURIPADDRESS.or.HOSTNAME> -p 53,80,443

What is in your resolv.conf file?

What is the configuration on the server in terms of NAT routing?

It turns out that the University ICT department had blocked UDP port 53 preventing the AutoSSL from working correctly. They were convinced that DNS was working since TCP port 53 was open and the server was responding to DNS requests using dig and dnslookup. After opening UDP port 53, AutoSSL was working fine again.

I hope this helps others.

Regards,

Jeremiah van Oosten