The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Clustered CPanel with failover **completed**

Discussion in 'Bind / DNS / Nameserver Issues' started by mr2jzgte, Jan 26, 2005.

  1. mr2jzgte

    mr2jzgte Well-Known Member

    Joined:
    Jun 18, 2003
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Florida
    **Note this is for a 99.999% high availabilty project. Certian variables such as upstream connectivity can not be helped. The only other option here is to have multiple incoming ips connections routed via bgp to handle the multihomed address such as what we are using.)

    Ok.
    Brief Overview.

    We were at a stage where we had 3000 clients on one aging machine. My boss approached me and gave me a project to undertake.

    Multiple Clustered CPanel Server with expandability.

    After 2 months of solid work we've finally had complete success with this project. I'll provide some details here for anyones thats interested and to prove that it works..

    Network Diagram - best that i can do at the moment.
    [​IMG]

    Componants required

    2 x mid level machines - lots of memory + dual gige network cards (LVS front loaders)
    2-10 x mid-high level machines + dual gige cards (these are the actual cpanel machines)
    2 x low -mid level machines - FileStorage Servers. again dual gige cards.
    2 x GIG E Switches.

    In our case i used the following
    2 x P3 1.3GHz Xeons with 1gb memory - 3 x 36gb scsi disks in a mirrored array + hotswap
    2 x dual PIV 2.4Ghz Xeon Machines with 3 x 36gb scsi disks in a mirrored array + hotswap
    2 x Dual PIV 2.4ghz Xeons with 2 x 80 Sata Mirrored Array (system drive) + 4 x 200gb Sata drives in a mirrored striped array ( 380 gb usuable space)

    The network flow is data comes into the system via a multihomed address which carries the working subnet (a /28 expandable in this case)

    This is all based on Redhat Enterprise 3.0 with a 2.4 kernel.
    The 2 front loaders are identical config with exception of the ip addresses.
    1st machine handles all the connections and return traffic. Should this machine go down then the 2nd lvs machine will assume all ip's and routing information.
    These machines run IPVS, heartbeat, ldirector and thats about it + a few custom scripts for the network interfaces. (using ripd to announce routes to all machines in case one switch goes down)
    IPVS handles all the port forwarding and load balancing.
    Ldirector checks services on the cpanel machines and according to the setup will drop port forwards to any dead services and route them through to the live machine.
    Should both cpanel machines be down then ldirector modifies the IPVS table so that requests are handled by the lvs machine (i..e http webpage with system maint etc, local mail server which holds all mail in a spool ready to deliver when the servers come back up. Likewise for dns - it runs off the same config zones as the cpanel servers via a nfs share.

    Backend - 2 x fileservers. Again similar setup to above. One primary server which rsync's data to a 2nd server. If the first server goes down then data loss will be less than 10 mins. These servers also are the MYSQL cluster management services (4.1.9) Relatively dumb machines.. all they do is serve files to the cpanel machines.

    Cpanel machines.. This is where things got tricky.

    Started with a base install of Redhat Ent 3.0 with current patchset and 2.4.20-27 kernel.

    Mounts are as follows
    /home to nfs:/nfs/home
    /mounts/var to nfs:/nfs/var
    /mounts/etc to nfs:/nfs/etc
    /mounts/usr to nfs:/nfs/usr
    hard mounts with intr (this enables apps to stall and then start running again if the nfs servers go down)
    etc

    which enabled the use of symlinks (by god if i never see or have to use another symlink i'll be all too happy)

    I then proceeded to install Cpanel 10r6 on both machines.
    After this was done i started the hard bit. Modifying and symlinking.
    There were very few changes made to the cpanel program (zero if i recall correctly)
    Only major changes were converting the system to work with mysql 4.1.9 (which has a new password hashing method which caused all sorts of dramas for me till i figured out the legacy tables.)
    Main problems here were the eximstats app.
    I set it up so both machines use the same rsa keys - for https/ssl certs to work correctly.

    cpanel machines have aliased lo:0-lo:10 with the /28 subnet also all data returns via the lvs machines this setup is known as LVS-NAT

    the all machines have 10.0.x.x addresses with exception of the 2 lvs machines with have aliased interfaces for the /28 and multihomed addresses

    Thats about all for now that i can think of. If anyone has specific questions I'll see what i can do about answering them, I'm considering doing up a step by step document for this process. I wil be charging for this though due to the ammount of time and effort spent on this project.

    This setup is now live and hosting clients - No problems so far...

    Feel free to add comments / flames / disbelief etc..

    Things to do
    ** Setup monitoring software which emails admins about machine status.
    ** Work out why quotas are not working (they are working except a zero value is being returned for used space on clients account (not a real big concern with the space that we have)
    ** Finish off the snmp monitoring programs
    ** Complete the app to handle failback ftp/pop3/etcx requests
     
  2. myusername

    myusername Well-Known Member
    PartnerNOC

    Joined:
    Mar 6, 2003
    Messages:
    691
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    chown -R us.*yourbase*
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    We need this for our setup, will you do this for others?

    As for your quota issue, make sure that your files are owned by the right user, I had a similar issue with rsync changing ownership to root or other, make sure you have the right flags turned on to preserve the user from server a to server b.
     
  3. mr2jzgte

    mr2jzgte Well-Known Member

    Joined:
    Jun 18, 2003
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Florida
    Trust me i've looked at everything in regards to the quota issue. Just a matter of time before i sort out what it is..

    Doing the setup for others - no problems. either they supply their own machines to me and for a fee i'll set them up... or when i get the time to do up the documentation a fee for that + email/phone support to get everything up and running.
     
  4. myusername

    myusername Well-Known Member
    PartnerNOC

    Joined:
    Mar 6, 2003
    Messages:
    691
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    chown -R us.*yourbase*
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    I'd say start getting the docs togather as anyone who pays for your service (at least those who do not have ten zillion dollars) are going to want proper docs on it so they can reproduce it and or fix it if there is an issue.

    An rpm or web based interface woud be even that much better, but I'll take just docs. i'll bookmark this post and let you know.
     
  5. mr2jzgte

    mr2jzgte Well-Known Member

    Joined:
    Jun 18, 2003
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Florida
    The only reason I would be charging for the service is the sheer ammount of hours and research time and numerous configs that we tried in order to get one that works perfectly every time...

    Originally spent a good solid 6 weeks building the first semi working config, only to blow it away and start from scratch. The current config took about 200 hours to build with an extra 50 hours or so spent testing it.

    My documentations kinda crappy and likewise for my rpm building skills...

    But when i have a free moment I'll start a document with screenshots/commands which are used to build the system. What problems you have to address etc and some basic troubleshooting guides..

    If cpanel are willing to give me an educational licence or some sorts then I'll set up the machines on my home network and give you guys access to clustered array so you can see the working system.

    Anyone interested in this at all?
    It all comes down to if anyones actually wants the documentation or wants to build their own system If no one wants it then its hardly going to be worth my time doing it all again.
     
  6. myusername

    myusername Well-Known Member
    PartnerNOC

    Joined:
    Mar 6, 2003
    Messages:
    691
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    chown -R us.*yourbase*
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    I am interested in the docs on it. PM me for your rate.

    Thank you.
     
Loading...

Share This Page