Transfer tool slow

[refueledinc]

I'm using the Transfer Tool to move cpanel accounts from one dedicated server to another dedicated server and I'm seeing extremely slow migration times.

Source server: 64GB ram, 16 core, SSD
Destination server: 64GB ram, 16 core, NVMe raid10

Both servers have nearly idle resource usage before and during the transfer.

Both servers are in the same datacenter and both have a 1Gbps connection.

When transferring a 23GB account, it took almost 2 hours and transfer output shows 51.01MB per 15 seconds being processed. Watching the server resource usage during transfer shows neither server is under any type of heavy load - both are close to idle.

htop shows disk I/O usage less than 1% of drive capabilities, so it doesn't seem to be a disk I/O issue.

wget of a 400MB file between servers only takes a few seconds, so it seems to not be a network issue.

Why is the Transfer Tool only processing 51MB per 15 seconds?

How can I improve this and use the full resources of both servers? The next account to transfer is 250GB, so 51MB every 15 seconds is not acceptable.

 

[cPRex]

Hey there! There isn't anything on the cPanel side things that controls the transfer speed of data on the server, but since you've already tested with wget I'm really not sure. If you can create a ticket with our team we could always perform a test transfer to see if we can catch anything happening, but I don't have any good ideas off the top of my head as usually this is a network problem, which you've already ruled out.

 

[ffeingol]

Do the accounts have large amounts of mail? Do you use mdir format? The transfer system uses rsync and rsync'ing a large number of small files (i.e. mdir mail files) can take a long time. The same thing happens on ecommerce sites (Magento for example) that leave 100's of 1000's of little session files in a folder.

 

[cPRex]

That's a good point too, @ffeingol - if there are many tiny files, like email, it will extend the time a transfer takes significantly, even though they physically don't take up a lot of space.

 

[refueledinc]

Yes, the account in question was largely emails.

The next account to transfer is 250GB and is largely emails and an old magento site that is no longer public, but needs to remain running for historical data access.

Is it possible for me to use the Transfer Tool for account setup and DB setup and then rsync the entire home directory myself? Would that be faster?

The source server is an SSD and the destination server is NVMe, so they should be able to handle writing small files better than an HDD.

Are you saying this is a limitation of rsync? I ran rsync on the mail directory after the Transfer Tool completed and it was reasonably fast. Only took a few minutes even though the bulk of the disk space usage is email on this account. That's why I'm having such a hard time understanding why the Transfer Tool would need nearly 2 hours, but rsyncing the mail directory myself took so much less time.

 

[cPRex]

Is it possible for me to use the Transfer Tool for account setup and DB setup and then rsync the entire home directory myself? Would that be faster?

This is exactly what I'd recommend. There is a 'home directory' check box in the transfer tool that you can uncheck, so the account will be created, configuration files and databases get moved, but then you can manually rsync /home/username to completely the transfer outside of the Transfer Tool.

Transfer Tool takes longer because we move the files in batches, so they get added to the file list, compressed, and then moved and uncompressed, but that often doesn't work well for small files.

 

[refueledinc]

Thank you. I might give that a try then. There is no way to change the batch size or just have the tool rsync the files without batch and compression? Seems like the tool was batching in 51MB sizes which seems like a very small size to me.

 

[cPRex]

No, there isn't a way to change that behavior.