The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Streaming homedir in copy process doesn't work

Discussion in 'General Discussion' started by MediaServe, Apr 21, 2009.

  1. MediaServe

    MediaServe Well-Known Member
    PartnerNOC

    Joined:
    Apr 9, 2004
    Messages:
    126
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Nashville, TN USA
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    cPanel 11.24.4-C35075 - WHM 11.24.2 - X 3.9

    The account copy feature is really frustrating me!

    We've started putting upgraded servers online and moving sites to them, but the copy process doesn't seem to work larger sites (which we tend to have a lot of.)

    For example, this morning I attempted to move a 15GB site to a new server.

    The tail end of the homedir streaming process of the copy reports this:

    Code:
    ..11254784550..
    Done
    xferstream tag passed!
    The homedir has 15GB but has only transfered 11GB, and yet it is reporting success! What is up with that?

    A du -sh on the old server's homedir shows 15GB, on the new server it's 11GB. I have to tar the whole homedir across after the copy to make sure everything is there.

    The problem is, I really need to do some bulk copies, but I can't with the process being this unreliable. Anyone else noticing this?

    Also, SSL certificates and keys are never copied properly - they always have to be manually corrected afterwards.
     
  2. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    Please submit a support ticket so our technical analysts can assist you with this. Also mention the SSL certificate issue so we can take a look at what is causing that. You can contact our technical analysts at: http://tickets.cPanel.net/submit
     
  3. wwwcad

    wwwcad Member

    Joined:
    Oct 4, 2006
    Messages:
    17
    Likes Received:
    0
    Trophy Points:
    1
    Same problem here

    Today i had copied a lot of accounts with the whm multicopy script, but for several accounts, like 20/150, it report sucess with missing data from the old homedir. Sometimes the public_html had missing files or the email accounts were missing in the new cpanel.

    These transfer scripts seems to have problems with the old step: Streaming Homedir from source server.... It should compare at least the size of the home directory before reporting success.

    Regards
     
  4. Vision

    Vision Active Member

    Joined:
    Mar 18, 2002
    Messages:
    35
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Same problem here with multiple copy. 10/40 accounts were missing data from public_html and no email accounts created in cpanel.
     
  5. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    I recommend anyone experiencing these issues to submit a support ticket so we can determine the causes of this situation.
     
  6. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,461
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    Please verify that all directories and files in the home directory on the source server either:

    1) are owned by user:user; or

    2) have permissions 0755 for directories and 0644 for files

    The streaming process is performed by the account user name ( not root ). Hence any file or directory the user cannot access or read will be skipped by the streaming process.
     
  7. MediaServe

    MediaServe Well-Known Member
    PartnerNOC

    Joined:
    Apr 9, 2004
    Messages:
    126
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Nashville, TN USA
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    This is not the problem in our case - all ownership/permissions are good.

    I'll try to open a ticket soon, as we really need this fixed so we can start migrating sites to new servers.
     
  8. MediaServe

    MediaServe Well-Known Member
    PartnerNOC

    Joined:
    Apr 9, 2004
    Messages:
    126
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Nashville, TN USA
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    I never did open a ticket on this, but probably will soon. For anyone who comes across this thread, I thought I'd post some current observations.

    This problem seems to happen for sites that are over about 5GB. Sites under 5GB don't exhibit the problem. For sites over 5GB, the process seems to stop at the same relative spot close to 5GB - like there is some sort of 5GB limitation to the file streaming.

    I've started moving sites from one data center to another, and I started with the big sites, 10GB-20GB in size. None of them are completely transferring, and I started noticing with the last couple that they seem to only transfer about 5GB of files. Oddly, the process says the streaming is complete - i.e. no errors, just not everything is transferred.

    I just did a 4.7GB site to see if it worked properly, and it did. I'm going to do a couple more transfers of >5GB sites and will post exactly where they stop in the streaming process, then open a ticket for further investigation.
     
  9. dave_g

    dave_g Registered

    Joined:
    Jan 10, 2010
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Hello,

    I used cpanel for many years, now I have big accounts too (5-10Go) of different customers. What is the best way to transfer this kind of account when we have to change of server?


    Dave G
     
  10. MediaServe

    MediaServe Well-Known Member
    PartnerNOC

    Joined:
    Apr 9, 2004
    Messages:
    126
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Nashville, TN USA
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Here are the results of a couple tests:

    First site actual size (per du -s --bytes in home directory) is 8984700792.

    Last few lines of the streaming process:

    ..5436783600..
    ..5439568838..
    ...........
    ..5442354075..
    ...........
    ..5445139313..
    Done
    xferstream tag passed!

    Second site actual size is 8037912962

    Last few lines of the streaming process:

    ..4941011325..
    ..4943796563..
    ..4946581800..
    ...........
    ..4949367038..
    Done
    xferstream tag passed!

    When I used rsync to finish transferring files for these sites, the files which had been missed were files with proper ownership and permissions, so no reason for them not to transfer.
     
  11. cPanelCory

    cPanelCory Developer - cPanel Security Team
    Staff Member

    Joined:
    Jan 18, 2008
    Messages:
    69
    Likes Received:
    5
    Trophy Points:
    8
    Location:
    Houston
    cPanel Access Level:
    Root Administrator
    Can I get the version and build numbers of both servers?

    Thanks!
     
  12. MediaServe

    MediaServe Well-Known Member
    PartnerNOC

    Joined:
    Apr 9, 2004
    Messages:
    126
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Nashville, TN USA
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Both servers are: cPanel 11.25.0-C42399 - WHM 11.25.0 - X 3.9

    I am meaning to take the content from one of these large sites and create a dummy site that can be used for testing, and then provide access information in a ticket.

    Would this be useful, or can you perhaps duplicate this in house?
     
  13. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Thank you for the version information. So that we may examine the issue more in-depth, obtaining more detail to reproduce the symptoms, and investigate the surrounding circumstances please submit a support request/ticket referencing this thread; when available, please PM me your new ticket ID number so I may follow-up internally. For reference, the links in my forums signature may be used to initiate a ticket submission.
     
  14. cPanelCory

    cPanelCory Developer - cPanel Security Team
    Staff Member

    Joined:
    Jan 18, 2008
    Messages:
    69
    Likes Received:
    5
    Trophy Points:
    8
    Location:
    Houston
    cPanel Access Level:
    Root Administrator
    When you do start a ticket with access to your servers where this is occurring please put ATTN QA: 5GB Transfer Limit in the subject.

    I could not reproduce this with the builds you mentioned on a 10G account:

    ..11088030488..
    ..11090815725..
    ..11093600963..
    ..11096386200..
    ..11099171438..
    ..11101956675..
    ..11104741913..
    ..11107527150..
    ..11110312388..
    ...........

    ..11113097625..
    ..11115882863..
    ..11118668100..
    Done
    xferstream tag passed!


    The servers you are seeing this on will give up invaluable information as to what is causing this issue on your servers and we can go from there =)

    Thanks!
     
  15. urano007

    urano007 Member

    Joined:
    Feb 11, 2007
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    I have the same problem, but my site is 4 GB.
    How I can create a ticket?
     
  16. cPanelJared

    cPanelJared Technical Analyst
    Staff Member

    Joined:
    Feb 25, 2010
    Messages:
    1,842
    Likes Received:
    18
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
  17. urano007

    urano007 Member

    Joined:
    Feb 11, 2007
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    The tiicket is 1031677.

    Thank you.
     
Loading...

Share This Page