Consolidating multiple servers into one during upgrade/migration

Ok, so I figured I'd ask the forum on this one, in case anybody has run into this.

We've got some old CPanel shared/reseller hosting servers. They're nearing the end of their useful lives (CentOS 5.11 going EOL and SSL support issues, etc.), and we're doing a forklift upgrade to CentOS 7.1, new hardware, etc.

The population on these servers has dropped since their original deployments, and the new hardware we've spec'd is many multiples more powerful (much due to SSD's vs HDD's). We're looking at a 3:1 consolidation, and still having oodles of space/processor/etc. left over. We don't really want to drop VM's into play, as it'd be nice to reduce management overhead.

Typically when doing moves like this in the past, we would deploy the new hardware, install OS/CPanel/perform basic setup, then rsync all of the content over (using the old cpanel backup script as a reference for what to fetch). We'd rsync a number of times in advance, then on the day it came to actually move, we'd shut down services on the old (namely MySQL and mail), and do one final sync to catch everything before shifting IP's/routing/etc. as necessary. We had several big moves with < 5 minutes of downtime from this method.

In this case, since we want to do a 3:1 consolidation, that's not going to work. The problem is, we don't plan to migrate to new IP's (which seems to be what WHM centers around last I looked), as a slew of customers have custom NS's, setup, and there will literally be a good number of IP's per server once the dust settles (from the SSL, private shared IP's, nameservers, etc.)

What have you folks found works best for this sort of stuff? I realize we can use the pkgacct and restorepkg scripts to pull accounts over [and even set IP's], etc. (but then we'd have some work cut out to ensure every account restored, or got migrated back to their proper IP's [IE: what happens with addon/parked/sub domains in that scenario?], etc.). We've already scanned for conflicts in terms of identical users across the machines we plan to collapse together, etc.

The least amount of downtime possible is preferable, though I'd be keep to do 3 servers a night (aka 1 finished server per night), and hopefully be as clean as possible.... Wouldn't we all?

 

Hard to say, that depends entirely on what the customers setup I suppose, though I recall it was difficult (if not impossible) to assign different IP's to subdomains/addon domains/parked domains VIA CPanel awhile back (as we had a customer trying to setup SSL on an addon domain, and there was no practicable manner to separate it off to a dedicated IP).

My concern is we don't intend to do an IP migration here (the new server will take over the IP's from the old server), and IIRC, the transfer tool cannot directly accommodate that (IE: it'll renumber into new IP space arbitrarily on transfer), correct?

I realize we can do a mass "change multiple sites IP addresses" after the migration, but if there's an easier way/cut down on steps/errors/etc., it's preferable obviously.

 

I had run across this, and it was high up on my list of potentially good candidates for the move (going by the restorepkg tool VIA CLI instead of WHM).

I can operate the primary interface for the new server on a temporary IP, then add the actual/proper IP's onto the server as additional IP's, having the server in a VLAN that simply cannot route the additional IP's (so it can add accounts, but the IP's will not have conflicts, etc. until it's swung over).

The problem is, all of the servers are currently on the same VLAN (they share a /24 to allow IP's to allow swapping accounts/IP's among 'em, etc.). This means with a 3:1 consolidation, I can do this for the first source server (to shuffle everything over, and bring it fully online), but after this, source servers #2-3 need to be down before the replacement server starts restoring the accounts (as the replacement server will need to be in the same VLAN for the IP's from the first source server to be usable/changes to be effective/validation to occur, which will create a conflict).

This also means since I'd be doing several hundred accounts VIA the CLI as fast as possible, it's entirely likely we'd miss any errors/etc. (as it'd be a simple bash script with a few hundred lines for the accounts/IP's/etc.). This could be caught by reviewing the logs (in real time or after the moves)... But that's a slippery slope in most scenario's, when dealing with massive logs, etc.

The other potential is to write out the entire script for all 3 source servers, gather all of the backups from all of the servers, and blast out the restorepkg action blind/all at once (IIRC: the cpmove files are identical to the regular CPanel backup files). I assume doing multiple restores in parallel (IE: to expedite) is a horrible idea and should be avoided? And/or the restore process creates a lock file as it goes to prevent this sort of shenanigans?

Obviously I want there to be minimum time between when the backup was made, and when it was restored, to ensure DB entries, emails, etc. are not duplicated/lost/abandoned on the old servers/all the fun you get when doing moves like this.

 

I haven't used that tool in many many years. The last time we ran it (again, *many* years back, nearly 8-10 I believe), there was substantial leave-behind of old junk on the IP's which were no longer in use/bound to the system/etc.

I assume it's improved since then?

How is this different than the "change multiple sites IP addresses" tool? I remember the "IP Migration Wizard" was setup to stage the move, old/new IP's in parallel/etc. however that's not a desirable feature in this instance.