huge site migration - urgent help needed

[hozyali]

Hello,

I have a dedicated server with ThePlanet and a VPS from HostV.
I have a huge video website of 48GB data hosted on my VPS, due to some reasons, I have to move this huge website account to the dedicated server on ThePlanet.

The problem is, the website is so heavy and I can't figure out how to move it.
When I try to initiate the account migration from the dedicated Server via WHM, it keeps running and running and then gives an error of remote connection.

Can someone please provide me full details on how to migrate this huge website to the new dedicated server?

Thanks very much.

 

[webignition]

The most significant problem is the volume of data. Move the data first and then perform a regular transfer on the reduced-size account.

On the new dedicated server, create a hosting account for the 48GB of video data under a subdomain of the main domain. So if your site runs from example.com, create a subdomain such as video.example.com.

Add an A record as necessary, on the VPS, pointing the subdomain to the IP of the dedicated server. Wait for these changes to propagate.

Copy all the video data over to the new host and update all the video links on your site to get the video data served from the new host.

You can now safely delete all the video data from the VPS host.

You will now have a significantly smaller hosting account which should transfer as normal.

You will now also have your large volume of video data being served from a separate subdomain which offers the future option of hosting the video data separately from the main site, allowing you to improve performance by serving up the video data from either a separate dedicated video host or through a high performance content delivery network.

 

[cPanelDon]

As a precaution I would make use of a temporary page or notice on the hosted site to indicate a period of maintenance is underway or planned; this should help to alleviate some concern from site visitors during the time frame of the planned administrative maintenance tasks.

To accomplish the server migration I would consider transferring the account via WHM but with the "skip homedir" option enabled so that the account is packaged without the home directory contents; this will allow the largest portion of data to be transferred separately. Given the amount of data involved I believe using "rsync" may be a desired and efficient option; rsync can be used to transfer the home directory contents after having performed the initial transfer via WHM.

The account-packaging and transfer option "skip homedir" is available in WHM via the following menu paths:
WHM: Main >> Transfers >> Copy an Account from Another Server
WHM: Main >> Transfers >> Copy Multiple Accounts/Packages from Another Server

For "rsync" usage information I recommend referencing its manual "man" page and help documentation using the following commands via SSH access:

# man rsync
# rsync --help

 

[hozyali]

As a precaution I would make use of a temporary page or notice on the hosted site to indicate a period of maintenance is underway or planned; this should help to alleviate some concern from site visitors during the time frame of the planned administrative maintenance tasks.

To accomplish the server migration I would consider transferring the account via WHM but with the "skip homedir" option enabled so that the account is packaged without the home directory contents; this will allow the largest portion of data to be transferred separately. Given the amount of data involved I believe using "rsync" may be a desired and efficient option; rsync can be used to transfer the home directory contents after having performed the initial transfer via WHM.

The account-packaging and transfer option "skip homedir" is available in WHM via the following menu paths:
WHM: Main >> Transfers >> Copy an Account from Another Server
WHM: Main >> Transfers >> Copy Multiple Accounts/Packages from Another Server

For "rsync" usage information I recommend referencing its manual "man" page and help documentation using the following commands via SSH access:

# man rsync
# rsync --help

Thanks for your details. I think this would be a good way to work with this.
Regarding rsync, I will have to find out more information and commands on it as I never used it.

Do you know any site to give me example commands or you may be able to write example command for me to use it for my site?

thanks for your help.

 

[madaboutlinux]

for example:

/usr/bin/rsync -vrplogDtH -e "ssh -p 22" /home/username root@DestinationIP:/home/

You need to execute this command on the source server i.e. the VPS.

 

[hozyali]

and I also could not find the skip homdir option. can you explain plz

 

[madaboutlinux]

and I also could not find the skip homdir option. can you explain plz

Refer the following

http://forums.cpanel.net/612449-post2.html

It is useful while migrating such big sites.

 

[hozyali]

Refer the following

http://forums.cpanel.net/612449-post2.html

It is useful while migrating such big sites.

dont think I can do this way. first, I still do not understand how it is working.
second, I can't put the site down for too long, the procedure seems to take more time.

 

[natong]

0. Do not update new DNS.
1. Move *.mpg (and any big files) to somewhere out of user home dir.
2. Quicky transfer account to the new server by WHM.
3. Move *.mpg back to it own place.
4. Manual transfer *.mpg (and any big files) to the new server by ftp or rsync.
5. When finish everything just update new DNS.

I think downtime is < 5 mins. on the process step 1 - 3.

 

[cPanelDon]

and I also could not find the skip homdir option. can you explain plz

What is the full cPanel version number? This may be obtained using the following command via root SSH access:

# /usr/local/cpanel/cpanel -V

 

[hozyali]

well, here is the current situation now.

my existing VPS support people helped and they made 2 tar.gz files of the most heavy folders in my site as they have the videos.

Now, the one tar file is of 15gb and the second is 35Gb,

What I currently need is the best way to download this 35Gb file on the new dedi. server without any interruptions. I want the command to run even if the SSH is closed from my own PC.

Any suggestions?

 

[madaboutlinux]

The solution I gave you above won't take your website offline for too long. Since you will be creating an empty home directory for the account, the /scripts/pkgacct will complete in a couple of minutes. Once done, you can remove the empty home directory of the user and rename the original one back. The site will be online immediately.

Once you restore the backup on the new server, you can execute the rsync command to copy the users home directory and the website won't be offline when the rsync will be in process.

By doing a tar and zip, I would say, you are going through a long route instead of a simple solution.

 

[anojpala]

You can transfer the two tar files using the rsync command as described earlier.

rsync -avz -e "ssh -p SSH-PORT" TAR-FILE root@REMOTE-SERVER-IP:/home/

replace
SSH-PORT wiht your ssh port in the destination server
TAR-FILE is the your movie tar.gz files. You should be in that folder containing the tar file.
REMOTE-SERVER-IP is the destination server ip

But the best option is to do the following steps
1) Package account of that huge account without it home directory (as thats the one taking this much of space)
You can do this using the following command
/scripts/pkgacct --skiphomedir USERNAME
It will take backup of all the account files except its home directory.

2) Restore the account in the remote server using
/scripts/restorepkg USERNAME

3) Then rsync this user's home directory completely using rsync
rsync -avz -e "ssh -p SSH-PORT" /home/USER/ root@REMOTE-SERVER-IP:/home/USER/

Use screen (command) to run rsync in the background, so even if you are logged out of your shell. It will be running.

 

[hozyali]

thanks for your help.

the command you gave worked perfectly fine. but I have no idea how to create an empty homedir or move the existing one etc.

everyone is saying about package account without homedir, but I dont have any idea how to do that excluding homedir.

I did not find any option in WHM > transfer account.

The solution I gave you above won't take your website offline for too long. Since you will be creating an empty home directory for the account, the /scripts/pkgacct will complete in a couple of minutes. Once done, you can remove the empty home directory of the user and rename the original one back. The site will be online immediately.

Once you restore the backup on the new server, you can execute the rsync command to copy the users home directory and the website won't be offline when the rsync will be in process.

By doing a tar and zip, I would say, you are going through a long route instead of a simple solution.

 

[hozyali]

your solution makes sense a bit. thanks for that.

now the problem is, if I rsync the entire homedir, it will take ages again as the homedir is about 48GB.

You can transfer the two tar files using the rsync command as described earlier.

rsync -avz -e "ssh -p SSH-PORT" TAR-FILE root@REMOTE-SERVER-IP:/home/

replace
SSH-PORT wiht your ssh port in the destination server
TAR-FILE is the your movie tar.gz files. You should be in that folder containing the tar file.
REMOTE-SERVER-IP is the destination server ip

But the best option is to do the following steps
1) Package account of that huge account without it home directory (as thats the one taking this much of space)
You can do this using the following command
/scripts/pkgacct --skiphomedir USERNAME
It will take backup of all the account files except its home directory.

2) Restore the account in the remote server using
/scripts/restorepkg USERNAME

3) Then rsync this user's home directory completely using rsync
rsync -avz -e "ssh -p SSH-PORT" /home/USER/ root@REMOTE-SERVER-IP:/home/USER/

Use screen (command) to run rsync in the background, so even if you are logged out of your shell. It will be running.

 

[anojpala]

With that third step rsync will be fast, wont have to worry about untarring it in the remote server , but it will have to transfer more data that tar files.

But if you dont want that then you can do the following procedure.
Follow the first two steps provided earlier.
Then rsync the tar files to the remote server. Untar it there.
Now rsync the home directory without public_html folder and the tar files (if its there in the user home directory ) .
You can use the following option with rsync.
--exclude=PATTERN exclude files matching PATTERN

 

[cPanelDon]

everyone is saying about package account without homedir, but I dont have any idea how to do that excluding homedir.

I did not find any option in WHM > transfer account.

Please refer to my earlier post; knowing the version of cPanel will help greatly in being able to assist:

What is the full cPanel version number? This may be obtained using the following command via root SSH access:

# /usr/local/cpanel/cpanel -V

Please use the aforementioned command to obtain the full cPanel version number. This will include both the version and build information.

 

[hozyali]

I really appreciate all of your trying to help me solve my problem.

Here is what I am up to so far.

my vps host (source server) made two huge files for 2 internal directories which had videos. and now they are also transferred to the new dedi. server also via rsync.

I have also moved the user account completely by skipping homedir and restored on the new server.

so now I do not have the data in my public_html directory as I skipped it.

I also have 2 huge tar files there too but I have not extracted it yet as I am not sure if I should or no.

my question is now this.

If I use the rsync command on the destination server to get public_html folder from the source server, how long do you guys think it will take to execute completely knowing the source folder has approx 48GB of data?

is it doable or not? will appreciate if the exact rsync command provided to run on destination server and allowed to execute in background also so I can logout the ssh.

Thanks again everyone

 

[Shrex]

Cant we use wget command...

 

[cPanelDon]

Cant we use wget command...

You may use any tool to transfer files, such as compressed backup archives, from one location to another; however, using "wget" may not be a safe and desirable option for such a large amount of data (i.e., 48 GB) as stated in the opening post of this thread.

We would welcome it if you would like to elaborate on how you are suggesting the "wget" utility be used under the aforementioned circumstances; providing us with greater details will be very much appreciated and should help to ensure a better understanding.