Success - Complete Clustered and Redundant CPanel with load Balancing and Fail Over

[meganet]

has anyone determined the usability of the new cluster features of CentoOS 5?

[4u123]

I'm not too bothered about full cluster failover - it seems a very expensive and complicated thing to acheive...

However, I'd very much like to be able to seperate the storage so that every servers /home and /backup partitions are contained on a couple of shared external raid arrays. This way there is more resiliance built into the system with regards to the customers data and storage can be increased whenever required.

We once had a server fail and both the main drive and backup drive got fried - that coudnt happen with this solution.

I suppose that a large raid array would be sufficient for most needs - using ATAOE you could realistically connect ALL your web servers to one array for the home partition and mysql usage. This would also be very simple to configure on each server without any major changes.

I am making the presumption that with ATAOE you can configure an array as if it were a local volume - is that possible ?

If so....

If a server fails, you have a fresh cpanel server already on the network and all you have to do is mount the respective backup partition and run a full restore - this can be done via whm without a visit to the DC or paying for remote hands. Ok, there is still some disruption to the customers site but in the absence of an expensive and complicated clustering system, this would be the next best thing imho.

If you got this far..... Heres the genius part......

The customers home directorys would not have been affected by the server crash because they are held on the external array - only the internal cpanel information about the domains / accounts etc woudl have been lost....

So it would be VERY simple to just modify the restore script to NOT include a restore of the site files or databases - as they will already exist in exactly the same place they were before. All you would need to do is restore the cpanel data - which would take about 10seconds for each account -- meaning a full server restore would take only a matter of minutes!!

Any thoughts ?

[freedman]

I'm not too bothered about full cluster failover - it seems a very expensive and complicated thing to acheive...

[...]

I suppose that a large raid array would be sufficient for most needs - using ATAOE you could realistically connect ALL your web servers to one array for the home partition and mysql usage. This would also be very simple to configure on each server without any major changes.

I am making the presumption that with ATAOE you can configure an array as if it were a local volume - is that possible ?

[...]

So it would be VERY simple to just modify the restore script to NOT include a restore of the site files or databases - as they will already exist in exactly the same place they were before. All you would need to do is restore the cpanel data - which would take about 10seconds for each account -- meaning a full server restore would take only a matter of minutes!!

Any thoughts ?

you could use iSCSI or ATAOE I suppose. And yes, the OS would see these as "local" drives. However, external RAID arrays are not inexpensive and wouldn't save money over a shared storage solution using GFS or something similar.

There's also no need for a "full restore".. just store your cpanel info (/var/cpanel) on your external disk as well, then the only problem you might have is the IP addresses if those aren't available on the new server and these can be changed via whm or via a script .. there are other concerns...
if your IP addresses change, you have some nasty DNS issues.

Also, ANY external storage will have added latency so you're server performance will degrade slightly. any Over Ethernet disk is going to introduce another element of potential failure. YOu could mitigate this by having a small local private network for the disk, but you still are adding a bunch of network overhead on top of the scsi or ata protocols.

At that point, you might as well just NFS mount them (most external raid arrays which support iSCSI or ATAOE also support NFS), and then you can have them mounted LIVE on multiple machines and simply use round robin DNS to have multiple LIVE servers which is a very inexpensive "cluster" solution which doesn't require any manual intervention in the event of a server failure.
--one server fails the users browser just goes to the other one.. then you can take all the time you need getting the other server up and running.

the real problem is.. Cpanel is not multi-server friendly, so you have some extra work to do to syncronize apache, exim, mailman,etc...

[4u123]

you could use iSCSI or ATAOE I suppose. And yes, the OS would see these as "local" drives. However, external RAID arrays are not inexpensive and wouldn't save money over a shared storage solution using GFS or something similar.

I agree, there is little cost difference on the hardware but using GFS, the expense comes in trying to cluster cpanel servers which are not cluster friendly - which is why I am looking at this slightly differently.

There's also no need for a "full restore".. just store your cpanel info (/var/cpanel) on your external disk as well, then the only problem you might have is the IP addresses if those aren't available on the new server and these can be changed via whm or via a script .. there are other concerns...
if your IP addresses change, you have some nasty DNS issues.

You must be missing my point. Storing the /var/cpanel info would not be appropriate - its in a way "server specific". If you do that you start to complicate things too much. The backup data can be used to re-create the accounts back onto any server without worrying about IP addresses. I'm not proposing any kind of clustering solution, just a shared storage solution that allows for a full server recovery in less than ten minutes without the need for any complicated file system sharing or whatever you call it. Its clear by reading this forum that real clustering and fault tolerance of cpanel servers is very complicated and difficult to acheive. I'm only suggesting an alternative.

Also, ANY external storage will have added latency so you're server performance will degrade slightly. any Over Ethernet disk is going to introduce another element of potential failure. You could mitigate this by having a small local private network for the disk, but you still are adding a bunch of network overhead on top of the scsi or ata protocols.

Yes, I agree. I think this issue would be apparent on any system that is using NAS but the technology is very sophisticated these days. I think smaller arrays shared with fewer servers would help. I'm afraid I dont know anything about NFS, I'm looking at this from a very inexperienced point of view when it comes to these technologies, but that doesnt matter - ignore the fact that I mentioned networked storage - imagine the same idea with a smaller, physically attached array shared between 5 or 10 servers.

At that point, you might as well just NFS mount them (most external raid arrays which support iSCSI or ATAOE also support NFS), and then you can have them mounted LIVE on multiple machines and simply use round robin DNS to have multiple LIVE servers which is a very inexpensive "cluster" solution which doesn't require any manual intervention in the event of a server failure.
--one server fails the users browser just goes to the other one.. then you can take all the time you need getting the other server up and running.

The real problem is.. Cpanel is not multi-server friendly, so you have some extra work to do to syncronize apache, exim, mailman,etc...

Well thats why I suggested what I did - you seem to come full circle back to the problem faced by anyone wanting to do what you describe. It is difficult because cpanel isnt easily "clusterable". Thats why I suggested only providing shared storage for the /home and /backup partitions. If you seperate those from the operating system and cpanel installation you are not basically clustering anything - thats kind of my point.

What I'm suggesting is that if you use a shared storage system between a group of servers and selectively only share the mysql home and backup directories you are providing a more robust and fault tolerant storage solution than the usual setup which, if required, can very greatly speed up the disaster recovery process in the event of server failure.

To me, being able to complete a full restore of a server backup in only a few minutes with no complications and nothing really to configure is a good solution, whilst not providing a fully redundant and fault tolerant solution, its the next best thing.

[freedman]

You must be missing my point. Storing the /var/cpanel info would not be appropriate - its in a way "server specific". If you do that you start to complicate things too much. The backup data can be used to re-create the accounts back onto any server without worrying about IP addresses.
[...]
It is difficult because cpanel isnt easily "clusterable". Thats why I suggested only providing shared storage for the /home and /backup partitions. If you seperate those from the operating system and cpanel installation you are not basically clustering anything - thats kind of my point.

What I'm suggesting is that if you use a shared storage system between a group of servers and selectively only share the mysql home and backup directories you are providing a more robust and fault tolerant storage solution than the usual setup which, if required, can very greatly speed up the disaster recovery process in the event of server failure.

To me, being able to complete a full restore of a server backup in only a few minutes with no complications and nothing really to configure is a good solution, whilst not providing a fully redundant and fault tolerant solution, its the next best thing.

I fully understood your point.. you are missing mine.
/var/cpanel can be replicated (and should), while it contains "server specific" information, cpanel is smart enough to recognize that the IP address specified isn't available on the system and will use one that is.

my suggestions offer you a solution which does not require doing a restore via cpanel since all the data is there. the apache.conf needs to be rebuilt and other than that, you're ready to go... if you also maintain a copy of httpd.conf somewhere, then you can just "replace" the old IP's with the new IP's. and boom.. you're live.

your method wont do a "full restore" in "only a few minutes".. because when you do a "restore" through whm, it re-copies over everything in /home from /backup which is quite time consuming and unnecessary.

The way I run my servers are in "pairs". some of these pairs only have one cpanel license, some have 2.

Both are live and I'm serving the IP addresses for the domains in DNS round-robin.

The hard part is adding the entries to httpd.conf on the other server when a new account is added via cpanel, and manually adding the other servers IP to the named entrie for the domain. (one day I'll write a script to do this).

on the pairs where I only have 1 cpanel license, I just direct DOMAIN.COM/cpanel to cpanelhostIP:2086/

this way, all the cpanel stuff happens on one server (and if that server is down, their site is up, email works, but they can't access the control panel temporarily).
in the case where both servers have cpanel licenses, then this isn't a problem.

hope that clarifies.

in this scenario, which works wonderfully for me, the complicated part is filesystem synchronization. using GFS or NAS or NFS based storage would solve that problem. I currently use unison to sync the filesystems but there is latency. and I'm looking into Csync2 which I think might improve performance and reduce latency in replication.

The nice thing about csync2 or unison based solution is that you dont need any kind of specialized hardware or networking--you just use local disk.

[big]

with the changes in cPanel 11, its almost impossible to keep track with things that cpanel guys changes to keep them in sync...

[rejected]

It is possible using all the API hooks and mounting network shares correctly .

[CoolMike]

Would it be possible to use DRDB and Xen? With DRDB we mirroring a partition on two different server in the same datacenter. With Xen we run cpanel on this partition. I have not tried this yet, but could this work?

Then we don't need to worry about IP and licensing, because only one server is active and accessible.

Michael

[freedman]

Would it be possible to use DRDB and Xen? With DRDB we mirroring a partition on two different server in the same datacenter. With Xen we run cpanel on this partition. I have not tried this yet, but could this work?

Then we don't need to worry about IP and licensing, because only one server is active and accessible.

Michael

I assume you mean DRBD
this seems doable. As for the licensing, I'm not sure of the details, but I dont believe the licenses "float" if it's IP based licensing, and you move the IP to the failover server then it should be fine. if it's cpu based, then you'll have a problem. I *think* it's ip based so this should be just fine.

DRBD, as I understand it, sucks up a ton of bandwidth and can be a bit slow, but I dont have personal experience with it so I can't say for sure.

The main goal, in my opinion, should be not just to have hot-standby but to have multiple LIVE loadbalanced servers.

Ideally, for licensing $ savings, you direct all /cpanel requests to the server with the cpanel license running on it. for all other request, they get balanced across your (in this case 2) machines.

Some other things to consider are how DRBD works when there's a failure (network or hardware). how does the resync happen, what bandwidth/cpu/etc does this take up, etc...

Largely it depends on the volume manager you're using to do the mirroring, but make sure you consider all these issues.

I dont think there's a golden solution out there that will work for every scenario.

I've recently implemented gluster.
it's still very young but it's maturing fast and it is a good 98% solution.

I have running now, pairs of servers which run gluster doing live mirrors to eachother.

for now we're doing simple dns round robin for load balancing.

the solution isn't integrated with cpanel completely and we have to manually update DNS to add the other IP's and manually update httpd.conf on the other machine when accounts are added, but for us this isn't so much an issue.

I'm hoping to have time before the end of the year to flush out the system and will report back here when I have something more useful to share.

[cass]

Interesting ....

I'm needing something like this.... for maybe easier.

I need to do a round robin dns, for only ONE domain, but is one big one, like 8Gb data, and lot of requests.
It already have a separate mysql server.
now I've a second server, that will use only for HTTPD requests.

I thought in rsync with a cron for the user data.... (wich not changes a lot...)
And mysql should not be a problem as it's separate...

So the problem I think is basically the php, sessions, and this stuff.
What if one client got ip of server "a" for the first request (say they logued in the app), and then they got ip of server "b" ... they lost the login right? because the session is not there....

Someone that have done this, please explain me how to do this...
if with nfs, or gluster or whatever... and what partitions I need to do this way, and why... and if there's any problem if one server is rebooted, or down or something like this (mostly with the shared partition).
I know if one server goes down, 50% of the requests will probably get 404, but this is another story... client dont needs failover, mostly just balance.

[freedman]

So the problem I think is basically the php, sessions, and this stuff.
What if one client got ip of server "a" for the first request (say they logued in the app), and then they got ip of server "b" ... they lost the login right? because the session is not there....

There are a couple ways to solve this.
1) use mysql for your session handler - search for session_mysql

2) create a separate folder for your sessions and make sure it's mounted on both servers:
serverA# mkdir /sharedtmp
serverA# chmod 1777 /sharedtmp
serverA# exportfs /sharedtmp
serverB# mkdir /sharedtmp
serverB# mount serverA:/sharedtmp /sharedtmp
<php.ini>
session.save_path = "/sharedtmp"

(I believe nfs features are by default disabled on cpanel installations so you'll have to get those going if it doesn't work automatically)

3) make sure /home is a live filesystem, either an NFS mount on both machines, or some sort of clustered filesystem.


in your case, probably option 2 is the easiest, option 1 should do just fine.
if you're certain data will only change on one server, then rsync is fine, otherwise you need something that handles bi-directional syncing.
I use "unison" for this.

[wzd]

The original poster of this thread has gone AWOL a couple of years ago...

It seems to me that from a cost point of view (hardware redundancy wise) one should consider going VPS or other Grid Solutions..

We recently got a VPS server which was on 'trial' to serve as the full scale client server but last week it decided to corrupt (read only access on the hdd) and we had to do a recovery (almost 48hours lost)

---
seems to me unless you're spending a great deal of money you are stuck with problems...

[freedman]

[QUOTE=wzd;457165]The original poster of this thread has gone AWOL a couple of years ago...

It seems to me that from a cost point of view (hardware redundancy wise) one should consider going VPS or other Grid Solutions..

We recently got a VPS server which was on 'trial' to serve as the full scale client server but last week it decided to corrupt (read only access on the hdd) and we had to do a recovery (almost 48hours lost) QUOTE]

I'm not sure what you're saying. in one sentence you're suggesting a VPS and the next describing a horrible problem that occurred with one.

I have cpanel in a dual server environment, for the same amount of disk, bandwith, and CPU a VPS would cost me 2-3x what I'm paying now. It would also introduce a single point of failure and would limit the cpu and memory to whatever the largest instance could be on a computer.

I've looked into various grid solutions and they're all basically useless with cpanel. Cpanel is very resistant to a multi-computer infrastructure. All a grid does is offer a fast failover/failure recovery solution, which adds a little availability, but you dont get any of the other nice features of the grid like dynamically adding web instances in the case of a sudden traffic spike, or spanning a database instance across multiple computers.

I think once cpanel gets on board with grids it will be more useful, but what we really need for them for this to work is to have a central whm which can control accounts on multiple machines (these machines can be a grid, a vps, or a pile of stand-alone servers). I'd like to be able to let one account grow to use up more resources and when it does "migrate" other's off of it and onto different instances. Basically the way the grids allow you to do this with individual vps's/appliances/nodes.

For now, I'm using mysql in a master-master replication scenario and using a clustering filesystem for the homedirectories and for php's session files.
then I use "unison" to sync the relevant system files.

the only current annoyance with my setup is that i have to manually replicate http.conf changes to the 2nd server and dont really deal with cases where people change their user passwords (the ones in /etc/password, not the ones in /home/.../etc which are on the clustered filesystem.

Using round robin DNS, in my setup, if one server goes down 1/2 the people wont notice, the other half will hopefully be using smart browsers which timeout and go for the next IP in the pool.

This setup behind a load balancer would eliminate some of those issues, but it's still better than being down for 48 hours, or even for 2 mintues while a grid node comes online