Streaming homedir in copy process doesn't work

MediaServe

Well-Known Member
PartnerNOC
Apr 9, 2004
140
4
168
Dallas, Texas
cPanel Access Level
DataCenter Provider
cPanel 11.24.4-C35075 - WHM 11.24.2 - X 3.9

The account copy feature is really frustrating me!

We've started putting upgraded servers online and moving sites to them, but the copy process doesn't seem to work larger sites (which we tend to have a lot of.)

For example, this morning I attempted to move a 15GB site to a new server.

The tail end of the homedir streaming process of the copy reports this:

Code:
..11254784550..
Done
xferstream tag passed!
The homedir has 15GB but has only transfered 11GB, and yet it is reporting success! What is up with that?

A du -sh on the old server's homedir shows 15GB, on the new server it's 11GB. I have to tar the whole homedir across after the copy to make sure everything is there.

The problem is, I really need to do some bulk copies, but I can't with the process being this unreliable. Anyone else noticing this?

Also, SSL certificates and keys are never copied properly - they always have to be manually corrected afterwards.
 

cPanelDavidG

Technical Product Specialist
Nov 29, 2006
11,212
13
313
Houston, TX
cPanel Access Level
Root Administrator
cPanel 11.24.4-C35075 - WHM 11.24.2 - X 3.9

The account copy feature is really frustrating me!

We've started putting upgraded servers online and moving sites to them, but the copy process doesn't seem to work larger sites (which we tend to have a lot of.)

For example, this morning I attempted to move a 15GB site to a new server.

The tail end of the homedir streaming process of the copy reports this:

Code:
..11254784550..
Done
xferstream tag passed!
The homedir has 15GB but has only transfered 11GB, and yet it is reporting success! What is up with that?

A du -sh on the old server's homedir shows 15GB, on the new server it's 11GB. I have to tar the whole homedir across after the copy to make sure everything is there.

The problem is, I really need to do some bulk copies, but I can't with the process being this unreliable. Anyone else noticing this?

Also, SSL certificates and keys are never copied properly - they always have to be manually corrected afterwards.
Please submit a support ticket so our technical analysts can assist you with this. Also mention the SSL certificate issue so we can take a look at what is causing that. You can contact our technical analysts at: http://tickets.cPanel.net/submit
 

wwwcad

Member
PartnerNOC
Oct 4, 2006
22
6
153
cPanel Access Level
DataCenter Provider
Same problem here

Today i had copied a lot of accounts with the whm multicopy script, but for several accounts, like 20/150, it report sucess with missing data from the old homedir. Sometimes the public_html had missing files or the email accounts were missing in the new cpanel.

These transfer scripts seems to have problems with the old step: Streaming Homedir from source server.... It should compare at least the size of the home directory before reporting success.

Regards
 

Vision

Active Member
Mar 18, 2002
35
0
306
cPanel Access Level
Root Administrator
Same problem here with multiple copy. 10/40 accounts were missing data from public_html and no email accounts created in cpanel.
 

cPanelKenneth

cPanel Development
Staff member
Apr 7, 2006
4,607
79
458
cPanel Access Level
Root Administrator
Please verify that all directories and files in the home directory on the source server either:

1) are owned by user:user; or

2) have permissions 0755 for directories and 0644 for files

The streaming process is performed by the account user name ( not root ). Hence any file or directory the user cannot access or read will be skipped by the streaming process.
 

MediaServe

Well-Known Member
PartnerNOC
Apr 9, 2004
140
4
168
Dallas, Texas
cPanel Access Level
DataCenter Provider
Please verify that all directories and files in the home directory on the source server either:

1) are owned by user:user; or

2) have permissions 0755 for directories and 0644 for files

The streaming process is performed by the account user name ( not root ). Hence any file or directory the user cannot access or read will be skipped by the streaming process.
This is not the problem in our case - all ownership/permissions are good.

I'll try to open a ticket soon, as we really need this fixed so we can start migrating sites to new servers.
 

MediaServe

Well-Known Member
PartnerNOC
Apr 9, 2004
140
4
168
Dallas, Texas
cPanel Access Level
DataCenter Provider
I never did open a ticket on this, but probably will soon. For anyone who comes across this thread, I thought I'd post some current observations.

This problem seems to happen for sites that are over about 5GB. Sites under 5GB don't exhibit the problem. For sites over 5GB, the process seems to stop at the same relative spot close to 5GB - like there is some sort of 5GB limitation to the file streaming.

I've started moving sites from one data center to another, and I started with the big sites, 10GB-20GB in size. None of them are completely transferring, and I started noticing with the last couple that they seem to only transfer about 5GB of files. Oddly, the process says the streaming is complete - i.e. no errors, just not everything is transferred.

I just did a 4.7GB site to see if it worked properly, and it did. I'm going to do a couple more transfers of >5GB sites and will post exactly where they stop in the streaming process, then open a ticket for further investigation.
 

dave_g

Registered
Jan 10, 2010
2
0
51
Hello,

I used cpanel for many years, now I have big accounts too (5-10Go) of different customers. What is the best way to transfer this kind of account when we have to change of server?


Dave G
 

MediaServe

Well-Known Member
PartnerNOC
Apr 9, 2004
140
4
168
Dallas, Texas
cPanel Access Level
DataCenter Provider
Here are the results of a couple tests:

First site actual size (per du -s --bytes in home directory) is 8984700792.

Last few lines of the streaming process:

..5436783600..
..5439568838..
...........
..5442354075..
...........
..5445139313..
Done
xferstream tag passed!

Second site actual size is 8037912962

Last few lines of the streaming process:

..4941011325..
..4943796563..
..4946581800..
...........
..4949367038..
Done
xferstream tag passed!

When I used rsync to finish transferring files for these sites, the files which had been missed were files with proper ownership and permissions, so no reason for them not to transfer.
 

cPanelCory

Release Manager - EasyApache
Staff member
Jan 18, 2008
71
7
133
Houston
cPanel Access Level
Root Administrator
Can I get the version and build numbers of both servers?

Thanks!
 

MediaServe

Well-Known Member
PartnerNOC
Apr 9, 2004
140
4
168
Dallas, Texas
cPanel Access Level
DataCenter Provider
Can I get the version and build numbers of both servers?

Thanks!
Both servers are: cPanel 11.25.0-C42399 - WHM 11.25.0 - X 3.9

I am meaning to take the content from one of these large sites and create a dummy site that can be used for testing, and then provide access information in a ticket.

Would this be useful, or can you perhaps duplicate this in house?
 

cPanelDon

cPanel Quality Assurance Analyst
Staff member
Nov 5, 2008
2,544
12
268
Houston, Texas, U.S.A.
cPanel Access Level
DataCenter Provider
Twitter
Both servers are: cPanel 11.25.0-C42399 - WHM 11.25.0 - X 3.9

I am meaning to take the content from one of these large sites and create a dummy site that can be used for testing, and then provide access information in a ticket.

Would this be useful, or can you perhaps duplicate this in house?
Thank you for the version information. So that we may examine the issue more in-depth, obtaining more detail to reproduce the symptoms, and investigate the surrounding circumstances please submit a support request/ticket referencing this thread; when available, please PM me your new ticket ID number so I may follow-up internally. For reference, the links in my forums signature may be used to initiate a ticket submission.
 

cPanelCory

Release Manager - EasyApache
Staff member
Jan 18, 2008
71
7
133
Houston
cPanel Access Level
Root Administrator
When you do start a ticket with access to your servers where this is occurring please put ATTN QA: 5GB Transfer Limit in the subject.

I could not reproduce this with the builds you mentioned on a 10G account:

..11088030488..
..11090815725..
..11093600963..
..11096386200..
..11099171438..
..11101956675..
..11104741913..
..11107527150..
..11110312388..
...........

..11113097625..
..11115882863..
..11118668100..
Done
xferstream tag passed!


The servers you are seeing this on will give up invaluable information as to what is causing this issue on your servers and we can go from there =)

Thanks!