Backing up to Backblaze B2 is slow

[dataforge]

I have WHM/cpanel, 86.02 and i have backblaze b2 setup as a remote backup destination. I backup about 120gb each backup run. I have just recently started using B2 as the destination, and it is taking a long time to upload my data. I previously had been using S3 as the destination and it was extremely fast. I do not know how to tell the exact speed, but since switching to b2, i receive notifications from whm that my disk usage is unusually high. this is because the backups of each account stays on my server until they are transferred to b2. As a file is transferred to the destination it is removed from my server. with s3 it never had a chance to trigger the alert, but with b2 it is.
For instance, my backup finished about 5am this morning, and it is almost 4pm and it is still transferring the backups to b2.

I posted this here Uploading to B2 slow and the general consensus is there is a 'threads' option that needs to be increased. How do I increase the threads used for the transport? How do I know how many threads I can increase to?

 

[cPanelLauren]

The only modifications you can make to the cPanel backup transport are:

I/O priority level at which nightly backups are run (Minimum: 0; Maximum: 7)
You must compile ionice with your kernel to use this option. ionice manages I/O scheduling for processes on your system. Specifying a lower value for this option gives the process a higher priority. This option can accept integer values between 0 and 7.

Extra CPUs for server load
The load average above the number of CPUs at which cpuwatch, cpanellogd, backups, and CPU stats consider the system to be in a critical load state. For example, a server with 4 physical CPUs and a value of 2 in this field will be considered “critical” in these cases once the load reaches 6.

 

[dataforge]

Can you guys add support for multi threading uploading then?

Multithreading uploads
If your tool is uploading many files, the best performance will be achieved by multithreading the upload and launching multiple threads simultaneously.

• When initializing upload threads, call b2_get_upload_url or b2_get_upload_part_url for each thread. The returned upload URL / upload authorization token should be used in this thread to upload files until one of the error conditions described above occurs. When that happens, simply call b2_get_upload_url or b2_get_upload_part_url again and resume the uploads.
• When uploading files larger than 200MB, use the large files support to break up the files into parts and upload the parts in parallel.

See also Large Files

 

[cPanelLauren]

Thanks for that, note that I am indicating you couldn't configure not that it doesn't exist. From reading the internal story for the implementation of the Backblaze transporter, I know we break the files down into 5GB chunks and we're using the Backblaze API to connect with them, I am though unclear if this is being performed in parallel though I would assume it is, I'll confirm this with the team tomorrow though.

 

[jandafields]

... I'll confirm this with the team tomorrow though.

Any update on this?

 

[cPanelLauren]

Hello,

I discussed this in-depth with the PO and one of the developers on the team that built this. We don't use multi-threading for this when we built the module. They did note that it could be possible but it would have to be implemented into a sprint. They are going to track it on their board but did ask me to have you open a Feature Request (you can do this by clicking the link in my signature).

I'll set a reminder to follow up on it as soon as you provide me the link to the feature request

Thanks!

 

[phil99]

Not the OP, but I have created a feature request here (currently on moderation):

Edit: link removed as the feature request was indeed declined as per Lauren's post below.

 

[cPanelLauren]

Awesome, thanks @phil99 - I talked to our Community team today and they let me know they may close that Feature Request due to an internal improvement case being opened CPANEL-31619 - Parallelize upload of B2 File Parts as a result of this thread as well.