SOLVED [CPANEL-25247] Transporting backups to a Backblaze destination fails for large accounts

[nitro-it]

Hi Guys,

I setup BackBlaze as an additional destination for WHM backups which was quick and easy to do. My first backup job ran overnight which was the same as usual as far as running time, however I did get a transport error on a larger account without no clear explanation.

The error simply stated "Unable to send “/backup/2019-02-22/accounts/websitename.tar.gz” to destination “My Backblaze”

When I logged into BackBlaze my other accounts were backed up successfully, however I noticed that any compressed account backup over 5GB was being split into 5 or 10GB parts. One 6.5GB account was split into a 5 and a 1.5GB, which worked fine, but the bigger account which is around 12.5GB was like this:

websitename.tar.gz._b2_manifest (3) 171 bytes
websitename.tar.gz_b2_part_1 (3) 10GB
websitename.tar.gz_b2_part_2 (2) 0 bytes

So at the point of splitting the backup file into multiple parts the job has died and not been able to transport the remaining 2.5GB.

Is this a WHM/cPanel issue not being able to handle what BackBlaze is doing as far as splitting the files is concerned??

 

[nitro-it]

Hi Guys,

I setup BackBlaze as an additional destination for WHM backups which was quick and easy to do. My first backup job ran overnight which was the same as usual as far as running time, however I did get a transport error on a larger account without no clear explanation.

The error simply stated "Unable to send “/backup/2019-02-22/accounts/websitename.tar.gz” to destination “My Backblaze”

When I logged into BackBlaze my other accounts were backed up successfully, however I noticed that any compressed account backup over 5GB was being split into 5 or 10GB parts. One 6.5GB account was split into a 5 and a 1.5GB, which worked fine, but the bigger account which is around 12.5GB was like this:

websitename.tar.gz._b2_manifest (3) 171 bytes
websitename.tar.gz_b2_part_1 (3) 10GB
websitename.tar.gz_b2_part_2 (2) 0 bytes

So at the point of splitting the backup file into multiple parts the job has died and not been able to transport the remaining 2.5GB.

Is this a WHM/cPanel issue not being able to handle what BackBlaze is doing as far as splitting the files is concerned??

It appears that there are other large files that I didn't originally notice. When I view my bucket on BackBlaze, there is a link to view large unfinished files, however all it does is list them...doesn't tell me why they didn't upload or why they're unfinished...it says I can delete them if I browse files, but they dont appear when browsing files.

 

[nitro-it]

For anyone interested, I reached out to BackBlaze support and was given this response:

"Our service only supports individual files up to 5GB. Any file past that size must be split into parts under that size using our large file API calls. 'https://www.backblaze.com/b2/docs/large_files.html'

This is not done automatically on our end though, the integration you are using would have to manually handle the splitting of the large files, then submit the index to reassemble the files on our end. You would need to contact the developers for the integration you're using to see if they can provide insight as to why their usage of that API call would produce errors for you."

So maybe this destination is not setup well in WHM? I'm reverting to AWS until this is resolved.

 

[brianc]

Try increasing the timeout to 300 in your backup settings. I was getting the same problem. I do notice that backups being transported to Backblaze are slowwwwww.

 

[Mark_H]

I have the same problem, even with it set to a timeout of 300. There is definately something not quite right with it as it takes a crazy long time to complete compared to running an rclone on the command line to do the same upload of files to Backblaze (which also gives no such errors).

 

[cPanelMichael]

Hello,

Can anyone facing this problem open a support ticket so we can take a closer look at the affected system? We have a couple of internal cases (ref: CPANEL-25310, CPANEL-25247) open that relate to the transportation of large backup archives to the BackBlaze B2 destination type, but we'll need to review the affected system to determine if it's related to either of those cases. Please post the ticket number here once it's opened and I'll link this thread to it.

Thank you.

 

[Mark_H]

Mine is 11533407

 

[cPanelMichael]

Hello,

Thank you both for submitting tickets. I'll monitor each ticket and update this thread with the outcomes.

@brianc, I removed your previous posts since that patch did not succeed as a workaround.

 

[nitro-it]

Any update on this one?

 

[cPanelMichael]

Hello Everyone,

Internal case CPANEL-25247 is open to address an issue where transporting backup archives to the BackBlaze destination fails when the backup archive is large (5GB appears to be the trigger but the actual size is not yet confirmed). Here's an example of what you'll see in the backup transport logs found under the /usr/local/cpanel/logs/cpbackup_transporter/ directory when this hapens:

warn [cpbackup_transporter] Upload attempt failed: Can't "continue" outside a when block at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 801.
at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 30.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x24b8510), "Upload attempt failed: Can't \"continue\" outside a when block "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 636

A workaround is not available at this time. I'll update this thread as soon as new information on the status of this case is available.

Thank you.

 

[cPanelMichael]

Hello Everyone,

We've determined the cause of the problem and will publish a fix as part of internal case CPANEL-25247. I don't have a specific time frame to offer at this time, but it's planned for publication before cPanel & WHM version 78 reaches the STABLE release tier.

Thank you.

 

[brianc]

Will you let us know when the fix has been applied?

 

[cPanelMichael]

New Will you let us know when the fix has been applied?

Yes, I'll update this thread once CPANEL-25247 is published.

Thank you.

 

[Peter Horvath]

As I see there is no solution until yet. My ticket ID is
11634087. If you would may have any suggestion....

 

[cPanelMichael]

Hi Peter,

"Pending Publication" indicates we've reproduced the issue internally and we are planning to publish a fix in an upcoming version. A support ticket is not required when a thread is tagged with this prefix. I'll update this thread as soon as CPANEL-25247 is published.

Thank you.

 

[cPanelMichael]

Hello Everyone,

This is fixed in cPanel & WHM version 78.0.17:

Fixed case CPANEL-25247: Fix erroneous use of continue instead of next in backblaze transport.

Please let us know if you encounter any additional issues after upgrading to this version.

Thank you.

 

[cPanelMichael]

Hello Everyone,

A separate thread was opened for anyone seeing the "Use of uninitialized value $args{"api_method"} in string ne at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 259" error message in the backup transport logs after updating to cPanel & WHM version 78.0.17:

In Progress - [CPANEL-25310] BackBlaze transport error - Use of uninitialized value $args{"api_method"}

Thank you.

 

[Denis Gomes Franco]

Joining in here to drop my 2 cents on this. Switched from S3 to Backblaze two days ago in order to reduce my data storage bill and I'm having the issues described by the OP.

First it was a PITA to set up the account, BAD CREDENTIALS errors abound until I found out about what should I really put in these fields. Took me a good half an hour but okay, Backblaze finally validated.

But when it came to backup time large accounts failed. These files were split and only the first part was being uploaded. CPU spiked to 100% and only half my accounts were backed up. Sites also went down a couple of times while the CPU was at 100%. Seems like this is something related to Backblaze itself or the transport on Cpanel, because when using Amazon S3 everything sails smoothly and large accounts don't need splitting up. So I'm switching back to S3 for now.

Backblaze has great pricing but this makes it impossible to use. Also, here's a PSA: you cannot download files larger than 100 MB via Backblaze's web interface, you need to use the command line. Sure, that's not a big deal for advanced sysadmins but I am not one of them :D S3 does not complain about downloading large files via the web GUI, which makes it easier for me to just WGET the file back to the server whenever I need it.

EDIT: Forgot to mention that I'm on 78.0.21.

 

[Denis Gomes Franco]

And by the way, not related to Cpanel but curious nonetheless: I take care of some other Wordpress websites that aren't hosted on my server and thus I use UpdraftPlus to back them up. These are large sites and they also failed when backed up to Backblaze (whereas backups just worked fine on S3). I'll contact Backblaze support but I'm quite sure they will point it right back at Cpanel & Updraft.

 

[cPanelMichael]

But when it came to backup time large accounts failed. These files were split and only the first part was being uploaded. CPU spiked to 100% and only half my accounts were backed up. Sites also went down a couple of times while the CPU was at 100%. Seems like this is something related to Backblaze itself or the transport on Cpanel, because when using Amazon S3 everything sails smoothly and large accounts don't need splitting up. So I'm switching back to S3 for now.

Hello @Denis Gomes Franco,

It looks like this relates to case CPANEL-26694:

In Progress - [CPANEL-26694] Backblaze backup transport fails without error

I'll update the above thread as soon as the solution is published. Let me know if you have any questions.

Thank you.