Strange "not authoritative" for only few domains after nameserver IP change

Oh boy does this little dilemma have me a bit frazzled...

I have a small cluster of dedicated servers (using WHM DNS Clustering) and run my own DNS / name servers, and DNS is synced between all the servers. It has always worked very well even during migrations and nameserver IP changes in the past.

Recently I added a new server to the WHM DNS Cluster and, because I'm starting to phase out some old servers including the one that was in charge of "ns1" and "ns3" in my DNS nameservers list, I assigned "ns1" and "ns3" new IP addresses an re-registered / updated the nameserver registrations at my registrar, and proceeded with all of the usual steps when making this kind of change.

Now even though everything went well with the addition of the new box to the cluster and changing the IP's / updating registrations for ns1 and ns3 - for some strange reason a FEW of my customer's site's domain reports come back showing NS1 and NS3 as "not authoritative" for their sites / domains.

I'LL USE EXAMPLE NAMES AND IPS TO HELP EXPLAIN A BIT BETTER:

Let's say for the domain that I'm using for my registered name servers for the past 13 years is example.com

Now, I've always (well, for over 13 years now) used these 4 name servers for my DNS cluster and all of my customer domains that I host:

ns1.example.com (123.123.123.123)
ns2.example.com (456.456.456.456)
ns3.example.com (789.789.789.789)
ns4.example.com (010.010.010.010)

And so I'll I did was:

- Added a new server to the WHM DNS Cluster using the WHM > Clusters > DNS Cluster and also WHM > Clusters > Remote Access Key tools in WHM, I set all the server hostnames to DNS Role "Synchronize Changes" (status successful)

- Updated the registered nameservers ns1.example.com and ns3.example.com to 2 different new reserved IP addresses on the new server in the cluster

- Updated the appropriate options in /etc/named.conf and ran /scripts/restartsrv_named (successful)

- Updated WHM Basic cPanel & WHM Setup > Nameservers section by clicking Assign IP next to NS1 and NS3 to make sure it grabbed the new IP addresses properly, and clicked "Add A Entry for Nameserver" for NS1 and NS3. (all successful).

And much to my delight, within a few hours of propagation everything was working as expected with the new IPs assigned to NS1 and NS3.

The switch was rather quick and went as typical as the previous times that I've done this when phasing out old servers for new ones.

BUT HERE'S THE RUB - While 99% of my customer's domain reports come back nice and spiffy and everything is working great, a small number of customer domains are showing NS1 and NS3 as "not authoritative" for their domains in the domain reports on all the popular places like DNS check tool and intoDNS: checks DNS and mail servers health , thus failing the DNS tests and, wouldn't ya know it, one of those few customers has a Pingdom account and noticed it and opened a ticket with me.

THE STRANGE THING - Everything appears to be fine otherwise, and almost all other customer's DNS reports come back clean. Only a few user domains out of a few hundred report back NS1 and NS3 as not authoritative.

I've been banging my head against the wall for a few days over this, trying every possible troubleshooting procedure I can think of, to no avail.

ONE THING IN COMMON - I've noticed that the few domains that show NS1 and NS3 as not authoritative in their DNS reports are ones that have Dedicated IP Addresses. (At least the problem ones I've looked up so far).

BUT THE OTHER STRANGE THING - Now the twist is of course - there are plenty of other user domains that have Dedicated IP addresses assigned, with nice clean DNS reports. (There goes my thought that "oh ok, I've narrowed it down to only sites with dedicated IPs". NOT). :/

And so now I come to the cPanel forums hoping that someone here has either been through this and knows the solution, or has some ideas of things that I could check that might lead me to the solution for this little thorn in my side situation.

Many thanks for any replies! I'm getting a bit frazzled :-(

 

Thanks cPanelMichael. It's been 6 days since the change and I lowered all TTLs as well. Definitely easy longer than same change has taken before and definitely way past propagation time.

Again - 99.9% of all other domains and sites on the same servers do not have this issue. It's only a few and I would think that if it were a propagation issue then a lot more if not all Domains and sites would be affected.

I will definitely try your suggestion to delete and recreate designed for one of the sites to see if that makes any difference.

Edit - sorry for typos and bad grammar, trying to respond on the go from mobile device and I'm not very good without a keyboard

 

Hi cPanelMichael - I haven't gotten to this yet because I've been overwhelmed with urgent tickets and other tasks the past few days, but I'm hoping to get to it today.

I will first need to contemplate the best / safest method for deleting and re-creating the zone on one of the affected domains to see how it goes.

Your input on this and perhaps the actual steps that you would take if you were me would be quite welcome if you happen to have the time to respond to this request.

Thank you again for your assistance and advice!

 

Wow, that was fast, thank you!!

Following your steps did indeed resolve the problem!

One thing to note that might be a clue in all this and maybe helpful to anyone else who runs into this situation:

When I performed the step WHM > DNS Functions > Delete Zone for the domain that I tested this on, the following message (I've swapped my clustered server hostnames with the word example) displayed:

Zones Removed: example.com => deleted from exampleold1.
Zones Removed: example.com => deleted from exampleold2.
Zones Removed: example.com => deleted from exampleold3.
Unable to remove zone example.com from the Bind configuration (named.conf) on exmaplenew1.
The zone was possibly removed earlier on examplenew1.

So basically - WHM had no problem deleting the zone all of the older servers in the cluster, but it could not delete it from the new server in the cluster, which also happens to be the one handling the new IP addresses for NS1 and NS3 in my own DNS config.

That leads me to guess / speculate that maybe the trouble was initially started by the fact that MAYBE the new server is not syncing the DNS Zones properly for the few domains that were affected with this issue in the first place.

I hope I'm getting close to the real source of the issue and that maybe this info is useful to someone else someday.

However, obviously I could not have resolved this without the cPanel forum and cPanelMichael's help! THANK YOU SO MUCH! cPanelMichael - You guys should be allowed to have a tip jar or maybe Amazon Wish List on here so that we can give something back for your help in the forums.