The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

huge site migration - urgent help needed

Discussion in 'General Discussion' started by hozyali, Jan 27, 2010.

  1. hozyali

    hozyali Well-Known Member

    Joined:
    Jan 24, 2007
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    Hello,

    I have a dedicated server with ThePlanet and a VPS from HostV.
    I have a huge video website of 48GB data hosted on my VPS, due to some reasons, I have to move this huge website account to the dedicated server on ThePlanet.

    The problem is, the website is so heavy and I can't figure out how to move it.
    When I try to initiate the account migration from the dedicated Server via WHM, it keeps running and running and then gives an error of remote connection.

    Can someone please provide me full details on how to migrate this huge website to the new dedicated server?

    Thanks very much.
     
  2. webignition

    webignition Well-Known Member

    Joined:
    Jan 22, 2005
    Messages:
    1,880
    Likes Received:
    0
    Trophy Points:
    36
    The most significant problem is the volume of data. Move the data first and then perform a regular transfer on the reduced-size account.

    On the new dedicated server, create a hosting account for the 48GB of video data under a subdomain of the main domain. So if your site runs from example.com, create a subdomain such as video.example.com.

    Add an A record as necessary, on the VPS, pointing the subdomain to the IP of the dedicated server. Wait for these changes to propagate.

    Copy all the video data over to the new host and update all the video links on your site to get the video data served from the new host.

    You can now safely delete all the video data from the VPS host.

    You will now have a significantly smaller hosting account which should transfer as normal.

    You will now also have your large volume of video data being served from a separate subdomain which offers the future option of hosting the video data separately from the main site, allowing you to improve performance by serving up the video data from either a separate dedicated video host or through a high performance content delivery network.
     
  3. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    As a precaution I would make use of a temporary page or notice on the hosted site to indicate a period of maintenance is underway or planned; this should help to alleviate some concern from site visitors during the time frame of the planned administrative maintenance tasks.

    To accomplish the server migration I would consider transferring the account via WHM but with the "skip homedir" option enabled so that the account is packaged without the home directory contents; this will allow the largest portion of data to be transferred separately. Given the amount of data involved I believe using "rsync" may be a desired and efficient option; rsync can be used to transfer the home directory contents after having performed the initial transfer via WHM.

    The account-packaging and transfer option "skip homedir" is available in WHM via the following menu paths:
    WHM: Main >> Transfers >> Copy an Account from Another Server
    WHM: Main >> Transfers >> Copy Multiple Accounts/Packages from Another Server

    For "rsync" usage information I recommend referencing its manual "man" page and help documentation using the following commands via SSH access:
    Code:
    # man rsync
    # rsync --help
     
  4. hozyali

    hozyali Well-Known Member

    Joined:
    Jan 24, 2007
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6

    Thanks for your details. I think this would be a good way to work with this.
    Regarding rsync, I will have to find out more information and commands on it as I never used it.

    Do you know any site to give me example commands or you may be able to write example command for me to use it for my site?

    thanks for your help.
     
  5. madaboutlinux

    madaboutlinux Well-Known Member

    Joined:
    Jan 24, 2005
    Messages:
    1,052
    Likes Received:
    2
    Trophy Points:
    38
    Location:
    Earth
    for example:

    You need to execute this command on the source server i.e. the VPS.
     
  6. hozyali

    hozyali Well-Known Member

    Joined:
    Jan 24, 2007
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
  7. madaboutlinux

    madaboutlinux Well-Known Member

    Joined:
    Jan 24, 2005
    Messages:
    1,052
    Likes Received:
    2
    Trophy Points:
    38
    Location:
    Earth
  8. hozyali

    hozyali Well-Known Member

    Joined:
    Jan 24, 2007
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    dont think I can do this way. first, I still do not understand how it is working.
    second, I can't put the site down for too long, the procedure seems to take more time.
     
  9. natong

    natong Well-Known Member

    Joined:
    May 17, 2008
    Messages:
    85
    Likes Received:
    0
    Trophy Points:
    6
    0. Do not update new DNS.
    1. Move *.mpg (and any big files) to somewhere out of user home dir.
    2. Quicky transfer account to the new server by WHM.
    3. Move *.mpg back to it own place.
    4. Manual transfer *.mpg (and any big files) to the new server by ftp or rsync.
    5. When finish everything just update new DNS.

    I think downtime is < 5 mins. on the process step 1 - 3.
     
  10. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    What is the full cPanel version number? This may be obtained using the following command via root SSH access:
    Code:
    # /usr/local/cpanel/cpanel -V
     
  11. hozyali

    hozyali Well-Known Member

    Joined:
    Jan 24, 2007
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    well, here is the current situation now.

    my existing VPS support people helped and they made 2 tar.gz files of the most heavy folders in my site as they have the videos.

    Now, the one tar file is of 15gb and the second is 35Gb,

    What I currently need is the best way to download this 35Gb file on the new dedi. server without any interruptions. I want the command to run even if the SSH is closed from my own PC.

    Any suggestions?
     
  12. madaboutlinux

    madaboutlinux Well-Known Member

    Joined:
    Jan 24, 2005
    Messages:
    1,052
    Likes Received:
    2
    Trophy Points:
    38
    Location:
    Earth
    The solution I gave you above won't take your website offline for too long. Since you will be creating an empty home directory for the account, the /scripts/pkgacct will complete in a couple of minutes. Once done, you can remove the empty home directory of the user and rename the original one back. The site will be online immediately.

    Once you restore the backup on the new server, you can execute the rsync command to copy the users home directory and the website won't be offline when the rsync will be in process.

    By doing a tar and zip, I would say, you are going through a long route instead of a simple solution.
     
  13. anojpala

    anojpala Member

    Joined:
    Nov 29, 2005
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    You can transfer the two tar files using the rsync command as described earlier.

    rsync -avz -e "ssh -p SSH-PORT" TAR-FILE root@REMOTE-SERVER-IP:/home/

    replace
    SSH-PORT wiht your ssh port in the destination server
    TAR-FILE is the your movie tar.gz files. You should be in that folder containing the tar file.
    REMOTE-SERVER-IP is the destination server ip

    But the best option is to do the following steps
    1) Package account of that huge account without it home directory (as thats the one taking this much of space)
    You can do this using the following command
    /scripts/pkgacct --skiphomedir USERNAME
    It will take backup of all the account files except its home directory.

    2) Restore the account in the remote server using
    /scripts/restorepkg USERNAME

    3) Then rsync this user's home directory completely using rsync
    rsync -avz -e "ssh -p SSH-PORT" /home/USER/ root@REMOTE-SERVER-IP:/home/USER/

    Use screen (command) to run rsync in the background, so even if you are logged out of your shell. It will be running.
     
  14. hozyali

    hozyali Well-Known Member

    Joined:
    Jan 24, 2007
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    thanks for your help.

    the command you gave worked perfectly fine. but I have no idea how to create an empty homedir or move the existing one etc.

    everyone is saying about package account without homedir, but I dont have any idea how to do that excluding homedir.

    I did not find any option in WHM > transfer account.

     
  15. hozyali

    hozyali Well-Known Member

    Joined:
    Jan 24, 2007
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    your solution makes sense a bit. thanks for that.

    now the problem is, if I rsync the entire homedir, it will take ages again as the homedir is about 48GB.



     
  16. anojpala

    anojpala Member

    Joined:
    Nov 29, 2005
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    With that third step rsync will be fast, wont have to worry about untarring it in the remote server , but it will have to transfer more data that tar files.

    But if you dont want that then you can do the following procedure.
    Follow the first two steps provided earlier.
    Then rsync the tar files to the remote server. Untar it there.
    Now rsync the home directory without public_html folder and the tar files (if its there in the user home directory ) .
    You can use the following option with rsync.
    --exclude=PATTERN exclude files matching PATTERN
     
    #16 anojpala, Jan 29, 2010
    Last edited: Jan 29, 2010
  17. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Please refer to my earlier post; knowing the version of cPanel will help greatly in being able to assist:
    Please use the aforementioned command to obtain the full cPanel version number. This will include both the version and build information.
     
  18. hozyali

    hozyali Well-Known Member

    Joined:
    Jan 24, 2007
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    I really appreciate all of your trying to help me solve my problem.

    Here is what I am up to so far.

    my vps host (source server) made two huge files for 2 internal directories which had videos. and now they are also transferred to the new dedi. server also via rsync.

    I have also moved the user account completely by skipping homedir and restored on the new server.

    so now I do not have the data in my public_html directory as I skipped it.

    I also have 2 huge tar files there too but I have not extracted it yet as I am not sure if I should or no.

    my question is now this.

    If I use the rsync command on the destination server to get public_html folder from the source server, how long do you guys think it will take to execute completely knowing the source folder has approx 48GB of data?

    is it doable or not? will appreciate if the exact rsync command provided to run on destination server and allowed to execute in background also so I can logout the ssh.

    Thanks again everyone
     
  19. Shrex

    Shrex Member

    Joined:
    Apr 28, 2010
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Mangalore
    Cant we use wget command...
     
  20. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    You may use any tool to transfer files, such as compressed backup archives, from one location to another; however, using "wget" may not be a safe and desirable option for such a large amount of data (i.e., 48 GB) as stated in the opening post of this thread.

    We would welcome it if you would like to elaborate on how you are suggesting the "wget" utility be used under the aforementioned circumstances; providing us with greater details will be very much appreciated and should help to ensure a better understanding.
     
Loading...

Share This Page