The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

cPanel DNS-Only Large Deployment

Discussion in 'Bind / DNS / Nameserver Issues' started by dwright, Oct 4, 2011.

  1. dwright

    dwright Registered
    PartnerNOC

    Joined:
    Oct 4, 2011
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    Hello All,

    We are currently using Rackspace Cloud + cPanel DNS-Only to run nearly 100,000 zones. BIND seems to be running slow, but the system load is fairly low and there is plenty of available memory. I understand this is a lot of zones so i'm open to adding more hardware, etc.

    Here are my goals ...

    #1 - Keep the same name servers. (NS1 / NS2)
    #2 - Maintain tight cPanel integration
    #3 - Scale to 1,000,000+ zones.

    How would this model look? Does anyone have experience creating a load distribution system?

    Creating new name servers for new zones is an option, but it breaks a lot of back-end processes (moving sites between servers).

    Any thoughts / experience on scaling BIG with cpanel DNS-Only?

    Dustin
     
  2. VeZoZ

    VeZoZ Well-Known Member

    Joined:
    Dec 14, 2002
    Messages:
    248
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    DataCenter Provider
    We've scaled beyond the 100,000 mark on a cluster and we're convinced that it is probably not wise to grow much larger than that. We noted that on our systems a large amount of adding or removing of zones slows down the resolution speeds big time. This is on a system with dual 5620's and 12gb of memory where there is plenty of cpu and memory. I think this is a problem of using bind rather than a problem with cpanel.

    We've just ended up adding new clusters to make sure we don't allow a single cluster to get that many zones. I'm curious if something like MyDNS-NG will be better for large amounts of zones. It's coming to 11.32 on cPanel portion and I'm wondering if it's part of DNSOnly as well.


    Sorry no solutions to actually scale it up. We couldn't find anything to help with the bind slowness and went the multi cluster route.
     
  3. dwright

    dwright Registered
    PartnerNOC

    Joined:
    Oct 4, 2011
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    Thank you for the insight.

    We're experiencing slow resolution right now @96,000 zones. My reservation with setting up new name servers is that we'll be unable to transparently migrate sites between servers. We frequently "upsell" customers to less dense servers, etc. Adding the next 100,000 customers will require extra hardware and will prevent us from "backfilling" less populated servers without icky name server changes.

    Anyone have information on MySQL driven DNS-Only Cpanel?

    Any other ideas?

    Dustin
     
  4. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    I doubt anything MySQL driven would be faster than serving flat files such as BIND uses. My recommendation is the same as VeZoZ to get another set of nameservers.
     
  5. SoftDux

    SoftDux Well-Known Member

    Joined:
    May 27, 2006
    Messages:
    983
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Johannesburg, South Africa
    cPanel Access Level:
    Root Administrator
    Have you tried running the DNS-only cluster on servers with SSD drives?
     
  6. dwright

    dwright Registered
    PartnerNOC

    Joined:
    Oct 4, 2011
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    Currently we're running on Rackspace Cloud. We've not tried to run the system on a dedicated server yet.

    I've got a tech lined up later this week who's done large deployments. I will post follow-up if we make progress.
     
  7. dwright

    dwright Registered
    PartnerNOC

    Joined:
    Oct 4, 2011
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    We also see the problem whenever the dns server reloads. I think the server would be fine if not for adding more every few minutes...
     
  8. VeZoZ

    VeZoZ Well-Known Member

    Joined:
    Dec 14, 2002
    Messages:
    248
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    DataCenter Provider
    This is the exact problem we run into. cPanel does not do any queue system at all for the DNS cluster to give it time to catch up. So if we have 5 machines each add a dns zone at the exact same time it runs ten total reloads. It runs one for external and one for internal for each domain. With a bunch of them our DNS system will not respond to other requests. I thought the whole idea of reload was so you did not have to restart bind and miss dns requests. Unfortunately reload seems to cause the same problem it just doesn't last as long.

    I really need to dig deeper into this but bind 9 had multi threaded support added. You'd think because of this bind would not have a problem with one request causing blocking for others. It however does as our reload which uses 100% CPU to do block all other requests until it completes.


    I wish I had more time to dig deep into mydns-ng or powerdns for that matter. I know the old mydns claimed the ability to do real time updates to dns via mysql. Was ideal for systems with many dns zones. So if mydns-ng takes the same sort of approach then thought maybe this was a solution. We can throw more hardware to serve DNS requests the problem is we can't throw more hardware at reloads as it's not a server performance problem to begin with.
     
  9. VeZoZ

    VeZoZ Well-Known Member

    Joined:
    Dec 14, 2002
    Messages:
    248
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    DataCenter Provider
    Since my last post we've found one bug and cPanel confirmed with a cPanel function sending the wrong commands to bind. It was sending a full 'rndc reload' rather than just a reload for the specific domain. It would not surprise me if there were more like this that could explain why things get really bad. We found it by building a wrapper around rndc to see what sort of commands were being issued.

    I haven't had an opportunity to go through every cPanel function and see what commands it's issuing to rndc. Though if you're logging commands and notice an odd amount of reloads it would be a good idea to ticket cPanel in case it's a bug.
     
Loading...

Share This Page