Clustered Nameservers and Problems

[silversurfer]

Dear all,

Currently one of our server clusters are setup as such:

- Server 1 to 4
- All 4 servers are set to syncronize with all the other 3 servers. ie. Server 1 syncronize to Server 2 to 4 and Server 4 will syncronize with Server 1 to 3 etc.

We are facing an issue where the DNS name.conf keeps getting corrupted. When this happens, Bind cannot be restarted and thus, new entries do not get loaded. Restarting will also fail till you correct the named.conf file to remove the duplicates.

For some unknown reason, Cpanel / Bind / etc seems to be inserted duplicate entries *sometimes* when new accounts are created in the named.conf. It can happen in 1 of the servers, a couple or none at all.

Is this an issue with our setup? Or do others have this issue. If you had, what fixes are there if any to fix this once and for all, instead of manually fixing the named.conf all the time?

[casey]

I also see this behavior *occasionally* and have to remove the duplicate entry. Have no idea what causes it.

[silversurfer]

The problem for us is that it occurs on a daily basis and on 2-3 servers (out of 4) daily.

My hypothesis is that the user when creating the account, instead of waiting for the operation to complete, pressed refresh on their browser. Or the user use the back button after creating the domain resulting in an attempt again to insert the dns record.

Either way, I guess how this has to be fixed is that the dns clustering has to:

1. Place a lock on the named.conf till editing is fully completed, file is saved etc.
2. Before inserting a new record in named.conf to check that no duplicate entries exist.

That way, this issue should not happen. Sometimes there are as many as 3 duplicate records in named.conf.

[silversurfer]

i created a bugzilla entry: http://bugzilla.cpanel.net/show_bug.cgi?id=1904

Feel free to subscribe to it.

[Marty]

I had the duplicate entry problem as well until I arrived at this configuration for clustering:

Server1 is the primary nameserver

Server2 is my secondary nameserver

I have 5 servers configured as follows:

Server1 <--> Server2 (i.e. changes on Server1 are transferred to Server2 and changes on Server2 are transferred to Server1)

Server1 <-- Server3

Server1 <-- Server4

Server1 <-- Server5

So changes on Server3, Server4, and Server5 are all transferred to Server1, but changes on Server1 are not transferred back to 3, 4, and 5. As changes from 3, 4,and 5 are transferred to 1, Server1 then transfers them to Server2.

This cluster has worked well. I get a duplicate once in a great while, but it is usually due to something involving a site transfer gone bad.

[silversurfer]

I run 4 nameservers so this wouldn't work for us.

[Marty]

If I were going to run 4 nameservers on 4 servers numbered 1 - 4, I would run it this way.

Server 1 <--> Server 2
Server 2 <--> Server 3
Server 3 <--> Server 4

Say that server 5 - 7 are boxes that host sites but do not host nameservers, I would set them up this way:

Server 1 <-- Server 5
Server 1 <-- Server 6
Server 1 <-- Server 7

Based on the pattern I used above, it should work, but it may be what you are already doing.

[silversurfer]

Sounds plausible. I give it a shot. Afterall there's nothing to lose since we are having named.conf corruption daily on multiple servers already.

[casey]

I had the duplicate entry problem as well until I arrived at this configuration for clustering:

Server1 is the primary nameserver

Server2 is my secondary nameserver

I have 5 servers configured as follows:

Server1 <--> Server2 (i.e. changes on Server1 are transferred to Server2 and changes on Server2 are transferred to Server1)

Server1 <-- Server3

Server1 <-- Server4

Server1 <-- Server5

So changes on Server3, Server4, and Server5 are all transferred to Server1, but changes on Server1 are not transferred back to 3, 4, and 5. As changes from 3, 4,and 5 are transferred to 1, Server1 then transfers them to Server2.

This cluster has worked well. I get a duplicate once in a great while, but it is usually due to something involving a site transfer gone bad.

I can't figure out which server "Standalone" and "Synchronize" is talking about in WHM. In the above scenario, in WHM on Server1, what do you select for Server3, Server4, and Server5 - Standalone or Synchronize?

[silversurfer]

It means

server 3 to 5 set to syncronize with 1 (but 1 isn't set to syncronize with 3 to 5)
server 2 set to syncronize with 1
server 1 set to syncronize with 2

[casey]

Aha! Thanks. I had them all synchronising with each other, which I guess is pretty unnecessary.

[Marty]

Aha! Thanks. I had them all synchronising with each other, which I guess is pretty unnecessary.

I had it that way at first, but when I changed it so that everything syncronized back to Server 1 and Server 1 only syncronized with 2, and all other servers were standalone with 1, my duplications went away.

Definistions for symbols in previous post:

1 <--> 2 = 1 is added to 2 as synchronize, and 2 is added to one as syncronize

1 <-- 2 = 1 is added to 2 as syncronize, and 2 is added to one as standalone.

[wtl]

I'm fighting this same issue! Daily at least one server has duplicates in named.conf, sometimes up to three servers are affected.

I like the scenario suggested, but I wonder if anyone has tried more of a round robin approach, where, given 4 servers for example, server 1 updates 2, 2 updates 3, 3 updates 4 and 4 updates 1.

I just wonder if anyone has taken that approach, and does it help with duplicates? We have several sets of reseller nameservers spread across 6 servers, and I'm not sure whether I should try the idea posted above, or this circular approach.

Of course, with this method, inserting new servers requires a little more configuration, as you'll have to break the last link (last server updating first) and fit the new server in.

[silversurfer]

Yes. Since I used Marty's approach, not a single corruption issue. Been over a week.

[Marty]

wtl,

The problem with the scenario you suggest is this.

If you an account to server 1, that actual addition of the account adds the zone and the conf entry, then when server 4 gets updated, it updates server 1 again which I think causes the duplicate.

If you use the scenario I outlined above, no server ever gets updated more than once and thus the dupes are eliminated just as silversurfer has seen.

I am glad it worked for you. It bugged the heck out of me when I first started using clustering.