How to upgrade server from CentOS 5 to 6 without much downtime and pain?

[syno]

I know year ago Bluehost upgraded their centos5 servers to centos6, and they announced that work is done in 2 hours (per server). Does anybody know how they did it? I doubt this (as is usually recommended) is migration to other server/hw via pkgacct/restorepkg, or multiple accounts copy, because moving so much users/gigabytes (per server) could take much more hours.

I just want on same hardware server to have centos6 after (most painless way preferrable) upgrade anybody could suggest. Main requirement is that cpanel does not change dns records of all clients in crazy way (i.e. ruin) and most important is that IP addresses of clients do not change and process does not take whole day (few hours is ok). If php/apache/etc change version this is ok too.

I tried mass move clients from older centos server to other centos6 server but it had many problems which I want to avoid next time. That is what I did before:

1. useradd all client accounts to the new server.
2. rsync each client homedir from old server to the new. I thought this will be much faster that pkgacct zip/copy/unzip every client homedir.
3. pkgacct --skiphomedir for each client account on old server.
4. ip addr del for each ip in the old server ip pool.
5. Add old IPs to the new server.
6. scp cpmove-*.tar.gz to the new server.
7. Then I attempt to restorepkg --force --ip=x.x.x.x cpmove-$client.tar.gz on the new server.
This is failed because restorepkg didn't liked --ip=x.x.x.x
8. restorepkg --force cpmove-$client.tar.gz for each $client username.
This of course assigned wrong IPs to clients, but I planned to reassign them later. This somehow led to mess in dns records.
9. Disable DNS clustering on old server.
10. Change Multiple Sites’ IP Addresses onnew server to proper (old) IP.
There I forced to play with main shared IP address to finally set correct IPs for these clients, otherwise it didn't work.
Then I found that some domains are empty (as of no client's php/html), it seems that restorepkg cleared subdomain directories. To fix that I rsynced homedirs again.
And tons of other mess, like fixing dns records (it somehow did not updated them correctly so many clients still got wrong IPs), and fixing soa records.

This was very painful, plus this time I need to stay on the same hardware.

So, how is recommended (or someone succeeded) to upgrade to centos6 w/o change client IPs and too much downtime?

 

[dalem]

if you have 2 servers its pretty painless (setting the main shared IP to the old IP but not having it on the system)
move all accounts skipping the home directory then rsyncing the home directory twice stopping mysql and rsync it so all of the data is as fresh as possible.

then set add the new old ip's to the new server only about 5-10 minutes of downtime
not sure why your using pkgacct from the command line just use
copy Multiple Accounts/Packages From Another Server

 

[cPanelMichael]

Hello

Have you considered using the transfer options in Web Host Manager? It packages accounts the same way that /scripts/pkgacct does, but it makes it easier to manage than completing it all manually. You could move the IP addresses to the destination server after the transfer completes to avoid changing the name servers.

Thank you.

 

[syno]

Thanks dalem and cPanelMichael for answers.

cPanelMichael, you probably meant 'Copy Multiple Accounts/Packages From Another Server', but this is much harder to use than scripted pkgacct/restorepkg, because it requires a lot of attentive mouse clicking. First I should plan transfer in waves, becasue of difference or shared IPs (many shared IPs is also possible becasue older server could have users transferred from even older server). First I need to transfer accounts from one IP (maybe hundred(s)), then for second IP, etc. In multiple copy I need carefully first click hundreds of packages from the list, then hundreds correct account names from thsi wave, for each account name I need to click also 'Dedicated IP' or 'Skip Homedir' when needed. So with just 100 users migration I should click like 300 times and each username should be carefully checked from the list (so I don't click user from other IP). This is very hard, requires a lot of attention, and error prone. That's why I prefer to use pkgacct/restorepkg.

What I would like intead is option to transfer IP from one server to another, with all linked to it users automatically, and/or ability to really force IP in restorepkg. And I like to really be sure it never touch DNS. Maybe cPanel could add more transfer friendly options into restorepkg? (Like --force IP really force it no matter what, and subdomain creation to not clean its directory, and protect dns from changing old records).

Anyway, I (preferrably) need to transfer users to upgraded system on the same hardware (after os upgrade). And I really want to replace all these older rhel/centoses with centos 6. Would be good if cPanel devise checked intructions/plan for painless transfer like this (w/o touching DNS and IPs). I think many people will want to migrate from centos 5 as it became 'final'. But I have even older, centos4 (these even with unsupported version of cPanel, sigh). So long wanted to upgrade all this, but can't...

I also don't want to migrate (i.e. copy in usual way) users from server A to server B, and then back to A after A's os is upgraded, because people don't like these movements (downtimes and mess with dns updates, which not always even possible if dns is external). Ideally all this migration should be done in one night per ~200 users server (I dont even believe in 2 hours like Bluehost could promise).

ps. Why need to 'protect dns from changing old records' -- some domains could have changed its IP (migrated to other hosting, but dns left here), when I copy(ed) account cPanel assign(ed) new IP to these domains which is very wrong. Then I search manually which IPs is changed wrongly and revert them manually, mess.

 

[cPanelMichael]

I suggest opening a feature request for these improvements via:

Submit A Feature Request

Thank you.

 

[syno]

cPanelMichael, thanks, I posted feature request, but I don't see it appearing in the feature requests list. So I post its text here:

Please provide account migration features which can be applicable for upgrading server from old centos to centos 6. This request is based on forum thread http://forums.cpanel.net/f49/how-upgrade-server-centos-5-6-w-o-much-downtime-pain-377781.html

Possible features that maybe useful in that (these maybe not optimal as I am not expert in all details of inner cpanel workings):

Top level IP migration from one server to another. Which could migrate all related accounts automatically. (Preferably with smallest downtime as possible. And because IP will be same DNS does not need to be changed.)
And/Or documentation for performing upgrade like that with existing methods.

Low level:

restorepkg without changing DNS records for client domains (that is source of many errors which is painful on mass migration).
restorepkg force IP to really force it (now it can deny force ip by some reasons). This is need when new server have one shared IP but migrated users have another shared IP.
restorepkg to not clean subdomains if user directory is already filled with content and cpmove is packaged with --skiphomedir. Btw, 'Account copy' have some options to optimize transfer but they are almost not explained and very puzzling how to actually do them properly in real situation of mass migration. I want to pre-copy /home with rsync first (before shutting down client sites) and then faster rsync just before restorepkg. But now restorepkg will clean subdomain directories. This forces me to do rsync again after restorepkg and I'm not sure that user's homedir on new system could be rsynced like this from old system without bad side effects.
and detailed documentation how to speed-up pkgacct/restorepkg cycle as much as possible to minimize clients sites downtime on mass migration. And how to keep IP and DNS untouched.
Overall I conclude that cpanel is not designed for mass migration except for simplest cases of account copy and multiple account copy to new server with obligatory IP and DNS changes. (Or it's not documented). What I need to do is migration of users without IP and DNS changes, because changing DNS is very problematic. (There is many custom DNS configurations and cpanel is crude in changing it to what it 'thinks' right, plus changing DNS is downtime simply because of ttl issues, and cpanel have default ttl of many hours. Also not always is possible to change DNS at all, there is always several clients like this on any server).

If scripted mass migration (w/o IP & DNS change) will be possible, then it will be possible to just migrate all users to other server in one night, then upgrade first server to centos6 by reinstall (which is only supported method of upgrading to centos 6) next day, and migrate them back other night. Without this, now, upgrading older centos (4,5) systems to centos 6 is almost impossible or very long, error prone, and tedious process.

Maybe someone can re-post it or support it by voting up (if it appears).

- - - Updated - - -

cPanelMichael, btw link to feature request system on your page is not working.
You gave link to page http://forums.cpanel.net/cpfeatures.php which links to http://go.cpanel.net/iwant which does not work.
Correct link seems to be http://features.cpanel.net/

 

[cPanelMichael]

The following URL is loading well for me:

cPanel Feature Requests

However, you are welcome to use cPanel Feature Requests as well. Note that feature requests can take up to 48 hours to be approved.

Thank you.

 

[syno]

It seems that feature request is not accepted...

 

[cPanelMichael]

It seems that feature request is not accepted...

Did you get a specific rejection message or have you simply not seen it's approval as of yet? What did you name the feature request?

Thank you.

 

[syno]

Did you get a specific rejection message or have you simply not seen it's approval as of yet? What did you name the feature request?

Thank you.

No, I didn't get any message. I don't rememebr exact request title (except it had word centos in it) and I forgot to save it either. (As I didnt expect there will be problems with it.) But its text is fully quoted in #6.

 

[cPanelMichael]

No, I didn't get any message. I don't rememebr exact request title (except it had word centos in it) and I forgot to save it either. (As I didnt expect there will be problems with it.) But its text is fully quoted in #6.

You may want to try submitting a new feature request for just a single feature. It looks like you had several features in the post you referenced, so limiting it to just one may help. Then, once it's approved, you can add comments on additional modifications that may go along with it.

Thank you.

 

[Infopro]

No, I didn't get any message. I don't rememebr exact request title (except it had word centos in it) and I forgot to save it either. (As I didnt expect there will be problems with it.) But its text is fully quoted in #6.

It was not rejected, it was being held in moderation. You can find the new Feature Request posted here:
Migration features for painless upgrade from older Centos to Centos 6. | cPanel Feature Requests

 

[syno]

cPanelMichael, Infopro, Thanks! I see that feature request is appeared.

If anyone else want to support it - please vote or comment!