Clustered DNS - Cannot Recreate Accounts Once Deleted

We are having a major issue here for the past few weeks, with our "clustered" DNS system. Firstly, the way our DNS servers work is that we have three central servers hosting the DNS entries for all our 5 servers. They are called bionic, vs12 and helicase, as you will see in the below output. Now, the problem is that recently (we upgraded to WHM 10.8.0 cPanel 10.8.2-R119 ), after we terminate account, we can't re-create the account with the same domain, as WHM gives an error that says "Sorry, a DNS entry for testacct12.net already exists, please delete it first (from all servers in the dns cluster)"

We checked /var/named as well as /usr/local/apache/sites as well as named.conf and all of them don't show up with the domain on ANY server. So we're at loss to explain how CPanel thinks that the entries are not deleted. Yes, we restarted apache/bind to make sure they refresh after any edits, but no luck.

Below is basically the way we follow to reproduce the DNS error:


How to Reproduce the Clustered DNS Error

PART I : Create a new test domain account via WHM
-----
1) Login to ANY server's WHM, bionic, benzene, primus or helicase
2) Go to "Create New Account"
3) Enter any domain as the domain, such as "testacct13.com", the username must not contain "test" itself, however (can be like "accttest") and enter a temp password and select any feature package for now. Uncheck the "Make owner" checkbox and then press Submit
4) Once the account is created successfully, go to Part II

PART II: Terminating the account

1) Click "Terminate an Account" option in WHM
2) Select the above account/domain you created and click Terminate
3) You will notice an error as follows:
................
Removing PostgreSQL databases and users....

Done
Removing User....Done
Removing Group....Done
Removing Named entry...testacct12.net -> deleted from bionic.

testacct12.net -> deleted from helicase.
Unable remove zone.
Unable to remove zone testacct12.net from the Bind configuration (named.conf).
The zone was possibly removed earlier.
testacct12.net -> deleted from vs12.
Done
......................

4) Now proceed to Part III

PART III: Recreate the account -- it will fail!

Follow the exact same steps as part I, and this time, the account creation will NOT work. You will see instead the following error message:

As you can see, the account is never added, and you can never re-add the domain. This functionality is NOT normal, you should be able to recreate the account. For some reason, the DNS says the domain exists, when any DNS entry doesn't show it does, as there are no traces of it in any file on any of the servers.

Any help on this issue is appreciated. Thanks!

 

anyone? Thanks

 

This does not seem to work still:

Code:

root@primus [~]# /scripts/killdns testdomain12.net
rndc: connection to remote host closed
This may indicate that the remote server is using an older version of the command protocol, this host is not authorized to connect, or the key is invalid.

testdomain12.net -> missing from named.conf on primusUnable to remove zone testdomain12.net from the Bind configuration (named.conf).
The zone was possibly removed earlier.
Unable to unlink /var/named/testdomain12.net.db, file does not exist.testdomain12.net -> deleted from primus.

testdomain12.net -> missing from named.conf on helicaseUnable to remove zone testdomain12.net from the Bind configuration (named.conf).
The zone was possibly removed earlier.
Unable to unlink /var/named/testdomain12.net.db, file does not exist.testdomain12.net -> deleted from helicase.

testdomain12.net -> missing from named.conf on bionicUnable to remove zone testdomain12.net from the Bind configuration (named.conf).
The zone was possibly removed earlier.
Unable to unlink /var/named/testdomain12.net.db, file does not exist.testdomain12.net -> deleted from bionic.
Unable remove zone.

Unable to remove zone testdomain12.net from the Bind configuration (named.conf).
The zone was possibly removed earlier.
Unable to unlink /var/named/testdomain12.net.db, file does not exist.testdomain12.net -> deleted from vs12.

Afterwards, going to server to recreate the account still produces the DNS exists error...