SOLVED updatenow Tried to sync version /etc/cpanel/TIERS.json file but failed

[APatchworkBoy]

Currently upcp / updatenow are failing nightly on my CentOS6 / WHM 11.78.0.18 box...

=> Log opened from /usr/local/cpanel/scripts/updatenow (17431) at Wed Apr  3 08:42:51 2019
[2019-04-03 08:42:51 +0100]   Running version '11.78.0.18' of updatenow.
[2019-04-03 08:42:51 +0100]   Detected version '11.78.0.18' from version file.
[2019-04-03 08:44:01 +0100] E Tried to sync version /etc/cpanel/TIERS.json file but failed: httpupdate.cpanel.net did not have any working mirrors.  Please check your internet connection or dns server. at /usr/local/cpanel/Cpanel/HttpRequest.pm line 929.
[2019-04-03 08:44:01 +0100] ***** FATAL: The version for tier 'release' is not defined!
[2019-04-03 08:44:01 +0100]   The Administrator will be notified to review this output when this script completes
=> Log closed Wed Apr  3 08:44:01 2019
[2019-04-03 08:44:01 +0100]   17% complete
=> Log closed Wed Apr  3 08:44:01 2019

And this ultimately results in...

[2019-04-02 23:43:29 +0100] E [/usr/local/cpanel/scripts/autorepair] The “/usr/local/cpanel/scripts/autorepair autorepair” command (process 23298) reported error number 25 when it ended.
[2019-04-02 23:57:56 +0100] E [/usr/local/cpanel/scripts/manage_greylisting] The “/usr/local/cpanel/scripts/manage_greylisting --init --update_common_mail_providers” command (process 24273) reported error number 1 when it ended.

(NB: log snippets are from two separate runs of upcp, hence date stamp oddities. Disregard.)

From the server, I can manually wget http://httpupdate.cpanel.net/cpanelsync/TIERS.json from CLI without an issue...

# wget http://httpupdate.cpanel.net/cpanelsync/TIERS.json

--2019-04-03 09:11:30--  http://httpupdate.cpanel.net/cpanelsync/TIERS.json
Resolving httpupdate.cpanel.net... 72.29.88.74, 208.43.129.162, 70.87.220.252, ...
Connecting to httpupdate.cpanel.net|72.29.88.74|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: `TIERS.json'
    [ <=>  ] 3,415       --.-K/s   in 0s    

2019-04-03 09:11:30 (138 MB/s) - `TIERS.json' saved [3415]

And I can ping / nslookup / dig httpupdate also...

# nslookup httpupdate.cpanel.net && dig httpupdate.cpanel.net && ping httpupdate.cpanel.net

Server:        10.126.168.151
Address:    10.126.168.151#53

Non-authoritative answer:

Name:    httpupdate.cpanel.net
Address: 67.159.2.2

Name:    httpupdate.cpanel.net
Address: 216.14.113.158

Name:    httpupdate.cpanel.net
Address: 208.74.121.39

Name:    httpupdate.cpanel.net
Address: 70.87.220.252

Name:    httpupdate.cpanel.net
Address: 209.85.80.214

Name:    httpupdate.cpanel.net
Address: 206.130.99.76

Name:    httpupdate.cpanel.net
Address: 72.29.88.74

Name:    httpupdate.cpanel.net
Address: 208.74.121.41

Name:    httpupdate.cpanel.net
Address: 204.10.37.146

Name:    httpupdate.cpanel.net
Address: 122.201.72.171

Name:    httpupdate.cpanel.net
Address: 66.23.237.210

Name:    httpupdate.cpanel.net
Address: 208.109.109.239

Name:    httpupdate.cpanel.net
Address: 186.227.195.180

Name:    httpupdate.cpanel.net
Address: 208.43.129.162

Name:    httpupdate.cpanel.net
Address: 83.170.94.2

Name:    httpupdate.cpanel.net
Address: 67.227.128.74

Name:    httpupdate.cpanel.net
Address: 67.205.110.4

Name:    httpupdate.cpanel.net
Address: 103.252.152.1

Name:    httpupdate.cpanel.net
Address: 66.71.244.18

Name:    httpupdate.cpanel.net
Address: 67.222.0.10

Name:    httpupdate.cpanel.net
Address: 94.75.231.77

Name:    httpupdate.cpanel.net
Address: 159.253.142.50

Name:    httpupdate.cpanel.net
Address: 208.100.0.204

Name:    httpupdate.cpanel.net
Address: 208.43.108.66

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.1 <<>> httpupdate.cpanel.net

;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49148

;; flags: qr rd ra; QUERY: 1, ANSWER: 24, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;httpupdate.cpanel.net.        IN    A

;; ANSWER SECTION:
httpupdate.cpanel.net.    37    IN    A    216.14.113.158
httpupdate.cpanel.net.    37    IN    A    208.74.121.39
httpupdate.cpanel.net.    37    IN    A    70.87.220.252
httpupdate.cpanel.net.    37    IN    A    209.85.80.214
httpupdate.cpanel.net.    37    IN    A    206.130.99.76
httpupdate.cpanel.net.    37    IN    A    72.29.88.74
httpupdate.cpanel.net.    37    IN    A    208.74.121.41
httpupdate.cpanel.net.    37    IN    A    204.10.37.146
httpupdate.cpanel.net.    37    IN    A    122.201.72.171
httpupdate.cpanel.net.    37    IN    A    66.23.237.210
httpupdate.cpanel.net.    37    IN    A    208.109.109.239
httpupdate.cpanel.net.    37    IN    A    186.227.195.180
httpupdate.cpanel.net.    37    IN    A    208.43.129.162
httpupdate.cpanel.net.    37    IN    A    83.170.94.2
httpupdate.cpanel.net.    37    IN    A    67.227.128.74
httpupdate.cpanel.net.    37    IN    A    67.205.110.4
httpupdate.cpanel.net.    37    IN    A    103.252.152.1
httpupdate.cpanel.net.    37    IN    A    66.71.244.18
httpupdate.cpanel.net.    37    IN    A    67.222.0.10
httpupdate.cpanel.net.    37    IN    A    94.75.231.77
httpupdate.cpanel.net.    37    IN    A    159.253.142.50
httpupdate.cpanel.net.    37    IN    A    208.100.0.204
httpupdate.cpanel.net.    37    IN    A    208.43.108.66
httpupdate.cpanel.net.    37    IN    A    67.159.2.2

;; Query time: 1 msec
;; SERVER: 10.126.168.151#53(10.126.168.151)
;; WHEN: Wed Apr  3 09:13:10 2019
;; MSG SIZE  rcvd: 423

PING httpupdate.cpanel.net (208.74.121.39) 56(84) bytes of data.

64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=1 ttl=38 time=175 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=2 ttl=38 time=176 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=3 ttl=38 time=155 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=4 ttl=38 time=197 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=5 ttl=38 time=164 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=9 ttl=38 time=146 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=10 ttl=38 time=144 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=12 ttl=38 time=158 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=13 ttl=38 time=156 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=15 ttl=38 time=164 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=16 ttl=38 time=136 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=17 ttl=38 time=151 ms
64 bytes from httpupdate106.cpanel.net (208.74.121.39): icmp_seq=19 ttl=38 time=157 ms

Any further suggestions anyone can think of to aid diagnosis? I'm guessing the issue only occurs via the specific perl httprequest class within updatenow... SSL checks show https unavailable for all the httpupdate mirrors (- Removed -) so am assuming it's NOT using https and therefore this isn't a TLS related issue, and therefore I've done all my checks against http.

 

[cPanelLauren]

Hi @APatchworkBoy


Can you tell me about your networking configuration? Are you NAT routed? Have you made any changes recently?

What is the output of the following (remove/obfuscate your public IP or identifying information) :

cat /var/cpanel/cpnat

curl myip.cpanel.net/v1.0

 

[APatchworkBoy]

Hi Lauren - Yep, NAT config... recently switched to a Watchguard firewall, with sNAT / http-proxy / https-proxy rules in place... had some issues that resulted in us being delicensed as it stuck the webserver outbound on wrong IP which locked our license out the other day. Raised a ticket (cPanel Support ID #11821731) for it to be unlocked after the issue had been resolved at our end to shunt it out via the correct IP address...

#sudo cat /var/cpanel/cpnat
10.126.168.9 8x.x.x.52

#sudo curl myip.cpanel.net/v1.0
8x.x.x.52

All public IPs tally up to our licensed IP address etc...

 

[cPanelLauren]

Hi @APatchworkBoy

I can't help but assume that this is related as it seems like it's still a networking issue that you're unable to connect. Feel free to open a ticket though as I'm pretty limited to what I can check without access. I did look at the ticket and ran a quick nmap 80/443 over TCP both show as the only non-filtered ports which should be enough.

 

[APatchworkBoy]

Indeed - inclined to agree with you - unfortunately firewall is handled by a different team here... have opened 11847523

 

[cPanelLauren]

HI @APatchworkBoy

Great, I'm watching that ticket and I'll update here with any new information as it becomes available.

Thanks!

 

[APatchworkBoy]

Issue resolved!

So, cPanel's MirrorSearch.pm in doing what it does sends some requests/gets some responses without http content-types set. Our firewall / httpproxy was set to deny all http-responses without a content-type defined.

Looks like this stops the ping test working within MirrorSearch.pm so upsets autorepair / updatenow / upcp.

Our infrastructure team have now switched this to allow instead of deny and all has sprung back to life. Screenshot attached should anyone else land here with Watchguard kit who can't find where we're talking.

 

[cPanelLauren]

Hi @APatchworkBoy

That makes a lot of sense, I'm glad you were able to identify the source of the issue and thanks for letting us know how you resolved it!

Thanks!