Complete Server Redundant and Failover ?

[Drake]

Hi Cpanel Staff & Members,

Suggestion. Maybe I'm suggesting it in the wrong place, but anyhow here it is.

I've always been very concerned about backups and downtime. I have been reading this bulletin board for years, and one thing I always look for is Server Redundancy and Failover. There are many, many good workarounds that members have instituted.

However, I would really like to see a Complete Server Clustering (not just DNS) Redundancy and failover solution incorporated in WHManager by the CPanel staff.

By complete, I even mean the Operating System as well.

It wouldn't even mind if if required the use of mechanically identical servers and an additional CPanel license for the second, third, server, etc. Even make it as an additional option that requires purchase of lifetime licenses. Because when you're serious about having your customers' sites & e-mail up and available, it's well worth it.

Redundancy, Failover, Backup, and Load Balancing.

So, CPanel guys , what do you say?
Thanks,

Drake P.

 

[gupi]

Drake, you may have a look at thegridlayer.com grid hosting.
It is a different approach and gives you both redundancy AND scalability.

( thegridlayer.com uses 3tera.com solution)

 

[Drake]

Hi, thanks for the reply.

However it seems like they're in the server hosting business themselves.

It would be counterproductive for me to move my boxes from my building to someone elses.

I have plenty of IP's and enough bandwidth in my own building, and a colleague of mine 50 miles away has the same scenerio. Both of us have server rooms, host linux & windows boxes of our own and other peoples too. And many of our servers, between me and him are running CPanel.

I, nor my colleague, don't claim to own a world-class datacenter, like some hosting companies would lead you to believe (though they often just rent a few spaces in a rack somewhere), but we each have a few nice pipes out to the internet. My colleague and me enjoy total control over our server rooms and are right there, 50 or 100 feet away from the boxes.


Now, if we could deploy this redundant / failover scenaro between my building and my colleague's building, that would be something worth investigating further.

Thre still remains the issue of creating fail-over for small quantities of IP numbers that are assigned to each CPanel box, as well as keeping the mirrored box in sync, and CPanel licensing for the mirrored set of boxes. (my mirrors at his building and his mirrors at my building) But if you took the geographical location failover out of the equation, say for instance, having redundant CPanel box on the same premises, that would be an easier issue to deal with. Since 2 interfaces can't bind to the same IP address, any CPanel mirrored standby box would need its own "main IP", for which we'd have to purchase a cpanel license.

However, I truely appreciated your reply.

Thanks,
Drake P.

 

[gupi]

Hello back, Drake

what I wanted to highlight was the solution, not the provider.
Maybe if you take a deeper look at 3tera's website, you will end up by implementing their solution yourself.

Yours,

 

[Drake]

Hey,
Thanks Gupi,
I appreciate that.

~Drake

 

[cPanelDavidG]

Hi Cpanel Staff & Members,

Suggestion. Maybe I'm suggesting it in the wrong place, but anyhow here it is.

I've always been very concerned about backups and downtime. I have been reading this bulletin board for years, and one thing I always look for is Server Redundancy and Failover. There are many, many good workarounds that members have instituted.

However, I would really like to see a Complete Server Clustering (not just DNS) Redundancy and failover solution incorporated in WHManager by the CPanel staff.

By complete, I even mean the Operating System as well.

It wouldn't even mind if if required the use of mechanically identical servers and an additional CPanel license for the second, third, server, etc. Even make it as an additional option that requires purchase of lifetime licenses. Because when you're serious about having your customers' sites & e-mail up and available, it's well worth it.

Redundancy, Failover, Backup, and Load Balancing.

So, CPanel guys , what do you say?
Thanks,

Drake P.

We have combined a variety of feature requests relating to clustering into Bugzilla 1066: http://bugzilla.cpanel.net/show_bug.cgi?id=1066

Here you can follow the long-term ongoing conversation about this functionality.

 

[sehh]

I've been looking for such a solution myself.

I came up with the following results which work with cPanel:


1) Multiple similar servers. One acts as a primary server and a second (or more) server(s) act as secondary. Once the main server is down, the secondary server becomes primary and continues from were the dead server was.

-Software Requirements: rsync (or similar) for file synchronization, MySQL Mirroring for database synchronization.
-Advantages: fast fail-over change, secondary servers can be of lower specifications, cheaper than other solutions.
-Disadvantages: requires a "3rd party" to do the switch, either a dedicated router or by manually changing the IP addresses (via script, whatever).


2) Multiple identical servers. Since cPanel doesn't support real clustering, this can be simulated with very expensive firewall/router devices which do load-balancing across multiple servers. Each system is identical with each other but share the same storage space. Example of such a system: 6 servers: 2 web servers load balanced via dedicated firewall/router, 2 SQL clustered database servers load balancing requests or just mirroring data, 2 storage servers or similar system (SAN, etc).

-Software Requirements: Not supported by cPanel, since it doesn't expect that "things" are spread out on so many different systems. Requires a MySQL version which has proper support (>= 4.1)
-Advantages: Probably the best solution which covers fail over, load balancing and fail safe requirements for critical operations.
-Disadvantages: Very expensive, hard to maintain with cPanel, extra hardware requirements (servers have secondary ethernet cards since they need to be interconnected).


I believe most people will go for option (1) which is cheap (can even be done with a two VPS accounts!) and doesn't break cPanel.

 

[finly]

1) Multiple similar servers. One acts as a primary server and a second (or more) server(s) act as secondary. Once the main server is down, the secondary server becomes primary and continues from were the dead server was.

-Software Requirements: rsync (or similar) for file synchronization, MySQL Mirroring for database synchronization.
-Advantages: fast fail-over change, secondary servers can be of lower specifications, cheaper than other solutions.
-Disadvantages: requires a "3rd party" to do the switch, either a dedicated router or by manually changing the IP addresses (via script, whatever).

Hello,

Could you please provide some tutorials how to accomplish this?

Thanks,

Arifin FinLy

 

[sehh]

unfortunately, i have none.

the easy stuff is to just read the documentation for "rsync" then read the documentation for MySQL mirroring (dev.mysql.com), but the rest i haven't found anything that can help.

 

[finly]

Hello Sehh,

Thank you for your reply. I will follow your suggestion to read the documentation.

Btw, if main server has whm/cpanel, should the slave server has whm/cpanel too?

Thanks

 

[sehh]

yes, it must be a complete mirror of the primary server, so you'll clone cPanel/WHM as well.

i don't know what happens with licenses, but in theory you shouldn't need an extra one if your secondary machine is inactive while the primary system is serving pages.

if you keep the same IP addresses when you do the fail over switch then the backup system will be running with the same license because its got the same IP address as the licensed cPanel/WHM system.

if you give it separate IPs when it comes online (DNS will take care of the rest) then you probably need a second license.

 

[finly]

Thanks Sehh for your answer. But can we keep the same ip if the slave server in different datacenter?

 

[sehh]

No you can't. These fail over systems works best (faster fail over switch) when done within the same data center.

If you want to use different IP's on different data centers, then you'd need two separate cPanel/WHM licenses and both systems would need to share/use a DNS server which is going to handle the switch of the IP addresses.

 

[tiggerific]

Issues with active mirrors

I've recently been setting up failover systems attempting to have mirrors of specific cPanel accounts available as "live" standbys. I must admit that cPanel doesn't seem to make it easy to mirror single account and keep it totally in sync with another. When just relying on rsync, you may be missing mail accounts, cron settings, new databases a user has created, and other things where the data itself doesn't sit nicely inside the public_html directory.

Anyway, I've found a way around it and it's effective for what I need. I simply rsync my daily backups over to my mirror every couple of days and restore the full account which ensures it has EVERYTHING a cPanel account requires. Every half hour or so between these full restores, I'm rsyncing the database and public_html.

This all works fine and the failover from DNSMadeEasy kicks in nicely if it needs to.

But, I have one problem and I'm not sure what the best thing to do is... when you have an account sitting on a cPanel box somewhere, even though it's not "live" as far as the domain name and DNS goes, it's still "active". Meaning that any cron jobs for that account are still run if they are stored there. I really don't want my standby server to be running the user cronjobs unless it "goes live". When there are accounts there that do things such as recurring invoices, sending emails, downloading emails, etc. I don't want those jobs to run both on the live server and the standby server.

The only idea I've come up with at the moment is to simply delete any user crons from /var/spool/cron after I've restored the account. Does anyone have any other ideas? I can't completely disable cron on that box because I use it to do the mirroring in the first place.

Any input appreciated.

 

[sehh]

First of all, my solution described above is about TOTAL mirroring of the entire server.

In other words, you won't deal with the mess you've described above, since the entire server is mirrored (with the exception of a few minor stuff that differentiate the two servers, like IP addresses, etc).

Your solution might work ok for small accounts, but i think you'll have problems if the accounts are really big (50GB of data, etc).

About your cronjob problem, your only option is to temporarily move the user cron files in a temporary directory and keep them there until the server becomes live.

 

[tiggerific]

They are only small accounts and this particular setup is only for specific accounts. The entire server is already mirrored on another in the same data center which automatic failover. The extra step here is to get some protection from any data center issues that may arise and have them in a separate DC for those that absolutely must have 100% uptime. My first few attempts at a reasonable solution didn't involve backups and restores, but figured it was the best way (for these small accounts) to get a complete sync of anything they needed.

And surely this situation arises with any "hot spare" that is in a different DC? It would only be if you are reusing the same IP's that the spare might sit there totally inactive right?

The other thing is that I'm of course hoping and expecting that any downtime of the main server would be minimal and it would revert back to the master in a very short amount of time. And it would probably be ok to never run the crons on that spare, as long as the website was functioning and any changes are replicated back when the master becomes available again.

To add on to your discussion, you said that to have a spare in a different data center would require 2 cPanel licenses. As long as that spare was running a cPanel license with the mirrored accounts on it - cron jobs would be running on both servers at the same time would they not? Even if the DNS didn't "flip the switch".

 

[sehh]

Cron jobs should never be alive in a backup server, you either disable cron or move them to a temporary directory.

If you disable the cron while the server is not live then you use a "push" method to initiate the mirroring scripts from the main server.

 

[tiggerific]

Cron jobs should never be alive in a backup server, you either disable cron or move them to a temporary directory.

That's what I thought. It just seems to be something that isn't mentioned in any of the threads/articles related to backups, hot spares and failover systems. Just wanted to make sure that I was handling it in the same or similar way to others.

 

[fniederm]

tiggerific: how is your setup done with dnsmadeeasy? does it automatically sync with cpanels dns? if so how did you get that to work?

thanks in advance!