Multi-threaded "Copy multiple accts/pkgs from another server" [Case 57564]

[Jay M]

Hey guys,

Not sure if this has been asked before, but is there a way the "Copy multiple accounts/packages from another server" tool can be improved for major server migrations.

I'm talking where there's a couple of hundred accounts you wish to shift to a new server, but you want to have multiple windows open to speed it up (say 3-4 concurrent transfers). It's pretty tedious checking alot of tick boxes for each window. So two potential solutions:

1) Create an auto select where you can specify it to automatically tick all accounts with say usernames A through to E?
2) Be able to specify to the transfer tool that you want it to multi-thread the transfer by migrating 3-4 accounts at any one time? (not sure how this would work)

More just a time saving (and RSI prevention) feature request, but honestly not sure how useful this would be to the majority.

Cheers

[Brian]

We are always looking for and investigating ways to speed up account transfers (and in the same vein, account backups).

Some examples of items under consideration/investigation via current Feature Requests are:

pigz
MyDumper

I'd suggest posting your support for them accordingly, as it sounds like some examples of items you'd like to see that would assist with this request.

Regarding item #1, I wouldn't have any ETA for you, but improvements to transfer process that make selecting batches/specific accounts easier is something we're indeed looking at (Case 52038).

Regarding item #2, I'm unsure as to how this would provide a benefit. Pkgacct is largely I/O bound, spawning multiple transfers simultaneously would (for the most part) slow all transfers down equally and result in a net zero benefit. Some cpu heavy aspects of pkgacct itself could potentially be improved for sure (pigz for threaded gzip and mydumper for threaded MySQL backup/restore are two promising examples). But I am not sure I see the value in simultaneous pkgacct. Having previously been a part of the Migration Team for cPanel, I can attest to the fact that the last thing I wanted when performing entire server to server transfers was multiple pkgaccts running at one time

I certainly welcome you to post any and all ideas in the feature request forums (like you are) so that everything can at least be put forth for consideration/review in addition to community feedback.

[cPanelDavidG]

I met with our Product Manager yesterday about this. While we agree with Brian's assessment that inherently the solution proposed will not adequate resolve the problem that it attempts to solve, we do want to tackle this issue.

We think we have devised a better solution for this problem (a need to speed up transfers): multi-threading. To elaborate, currently the transfer process runs effectively as a single thread doing a single thing at a time, never doing things in parallel. For example, a portion of the transfer process may look like this (looking at it abstractly):

- Compress an account
- Transfer the account
- Compress another account

During the transfer, the source server is effectively sitting idle. This obviates how multi-threading can speed up transfers. Instead of that server sitting idle, it can be busy compressing the next account WHILE it is transferring the previous account. Combined with pigz (or even if this comes before our pigz implementation), we could see a very noticeable boost in speed by doing things this way.

Let me know what you (the community) think of this proposal.

[LaceHost-Ishan]

- Compress an account
- Transfer the account
- Compress another account (while transferring old account)

This does sound good.

[Jay M]

Hi Brian and David,

Your proposal sounds quite good, and I'm all for supporting that sort of implementation of increased efficiency in the process.

The only reason we've opted for multiple windows in the past was because as you highlighted the source server can often sit idle for sometime (especially on really large account restores). So improvement in the efficiency of the backup process is probably the end goal, as opposed to simply working around the shortfalls of the current setup.

Thanks for the efforts

[bhanuprasad1981]

i support mulithreaded file transfer & pigz too i had similar situation where an account of 25gb data took me nearly 24 hours to transfer from US to EU location while multithread can easily cut down time a lot maybe you can give us an option to select multithread copy while doing transfers , i had dual nehlam cpu lying underused on both sides
which can be effectively used to speedup transfer process

[Kent Brockman]

- Compress an account
- Transfer the account
- Compress another account (while transferring old account)

Sweet: +1 to this.

BUT, if you are compressing one account that weighs 2 GB while transferring the previous account which weighs 8 GB, there will be some time when the source server becomes again, uselessly idle. Also, if you have to transfer just one account of 25 GB, the process should manage to cut the compressed file into smaller pieces and do a multithreaded transfer.

[cPanelDavidG]

Sweet: +1 to this.

BUT, if you are compressing one account that weighs 2 GB while transferring the previous account which weighs 8 GB, there will be some time when the source server becomes again, uselessly idle. Also, if you have to transfer just one account of 25 GB, the process should manage to cut the compressed file into smaller pieces and do a multithreaded transfer.

Very good point. I have made a note of that in our internal case about that since you do obviate a deficiency in the proposed algorithm.

[Kent Brockman]

Very good point. I have made a note of that in our internal case about that since you do obviate a deficiency in the proposed algorithm.

ok, the problem is that you guys basically have to program a kind of download manager in order to satisfy this feature Nice goal, huh?

[cPanelDavidG]

ok, the problem is that you guys basically have to program a kind of download manager in order to satisfy this feature Nice goal, huh?

Hehe, I can't guarantee it will be in the initial implementation, but it's certainly a nice goal to keep in mind when designing this modification to the transfer system.