SOLVED [CPANEL-25247] Transporting backups to a Backblaze destination fails for large accounts

nitro-it

Member
Feb 22, 2019
5
0
1
Australia
cPanel Access Level
Root Administrator
Hi Guys,

I setup BackBlaze as an additional destination for WHM backups which was quick and easy to do. My first backup job ran overnight which was the same as usual as far as running time, however I did get a transport error on a larger account without no clear explanation.

The error simply stated "Unable to send “/backup/2019-02-22/accounts/websitename.tar.gz” to destination “My Backblaze”

When I logged into BackBlaze my other accounts were backed up successfully, however I noticed that any compressed account backup over 5GB was being split into 5 or 10GB parts. One 6.5GB account was split into a 5 and a 1.5GB, which worked fine, but the bigger account which is around 12.5GB was like this:

websitename.tar.gz._b2_manifest (3) 171 bytes
websitename.tar.gz_b2_part_1 (3) 10GB
websitename.tar.gz_b2_part_2 (2) 0 bytes

So at the point of splitting the backup file into multiple parts the job has died and not been able to transport the remaining 2.5GB.

Is this a WHM/cPanel issue not being able to handle what BackBlaze is doing as far as splitting the files is concerned??
 

nitro-it

Member
Feb 22, 2019
5
0
1
Australia
cPanel Access Level
Root Administrator
Hi Guys,

I setup BackBlaze as an additional destination for WHM backups which was quick and easy to do. My first backup job ran overnight which was the same as usual as far as running time, however I did get a transport error on a larger account without no clear explanation.

The error simply stated "Unable to send “/backup/2019-02-22/accounts/websitename.tar.gz” to destination “My Backblaze”

When I logged into BackBlaze my other accounts were backed up successfully, however I noticed that any compressed account backup over 5GB was being split into 5 or 10GB parts. One 6.5GB account was split into a 5 and a 1.5GB, which worked fine, but the bigger account which is around 12.5GB was like this:

websitename.tar.gz._b2_manifest (3) 171 bytes
websitename.tar.gz_b2_part_1 (3) 10GB
websitename.tar.gz_b2_part_2 (2) 0 bytes

So at the point of splitting the backup file into multiple parts the job has died and not been able to transport the remaining 2.5GB.

Is this a WHM/cPanel issue not being able to handle what BackBlaze is doing as far as splitting the files is concerned??
It appears that there are other large files that I didn't originally notice. When I view my bucket on BackBlaze, there is a link to view large unfinished files, however all it does is list them...doesn't tell me why they didn't upload or why they're unfinished...it says I can delete them if I browse files, but they dont appear when browsing files.
 

nitro-it

Member
Feb 22, 2019
5
0
1
Australia
cPanel Access Level
Root Administrator
For anyone interested, I reached out to BackBlaze support and was given this response:
Code:
"Our service only supports individual files up to 5GB. Any file past that size must be split into parts under that size using our large file API calls. 'https://www.backblaze.com/b2/docs/large_files.html'

This is not done automatically on our end though, the integration you are using would have to manually handle the splitting of the large files, then submit the index to reassemble the files on our end. You would need to contact the developers for the integration you're using to see if they can provide insight as to why their usage of that API call would produce errors for you."
So maybe this destination is not setup well in WHM? I'm reverting to AWS until this is resolved.
 
Last edited by a moderator:

brianc

Well-Known Member
May 16, 2003
178
4
168
Try increasing the timeout to 300 in your backup settings. I was getting the same problem. I do notice that backups being transported to Backblaze are slowwwwww.
 

Mark_H

Member
Nov 9, 2010
7
1
53
I have the same problem, even with it set to a timeout of 300. There is definately something not quite right with it as it takes a crazy long time to complete compared to running an rclone on the command line to do the same upload of files to Backblaze (which also gives no such errors).
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,910
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Hello,

Can anyone facing this problem open a support ticket so we can take a closer look at the affected system? We have a couple of internal cases (ref: CPANEL-25310, CPANEL-25247) open that relate to the transportation of large backup archives to the BackBlaze B2 destination type, but we'll need to review the affected system to determine if it's related to either of those cases. Please post the ticket number here once it's opened and I'll link this thread to it.

Thank you.
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,910
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Hello,

Thank you both for submitting tickets. I'll monitor each ticket and update this thread with the outcomes.

@brianc, I removed your previous posts since that patch did not succeed as a workaround.
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,910
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Hello Everyone,

Internal case CPANEL-25247 is open to address an issue where transporting backup archives to the BackBlaze destination fails when the backup archive is large (5GB appears to be the trigger but the actual size is not yet confirmed). Here's an example of what you'll see in the backup transport logs found under the /usr/local/cpanel/logs/cpbackup_transporter/ directory when this hapens:

Code:
warn [cpbackup_transporter] Upload attempt failed: Can't "continue" outside a when block at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 801.
at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 30.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x24b8510), "Upload attempt failed: Can't \"continue\" outside a when block "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 636
A workaround is not available at this time. I'll update this thread as soon as new information on the status of this case is available.

Thank you.
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,910
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Hello Everyone,

We've determined the cause of the problem and will publish a fix as part of internal case CPANEL-25247. I don't have a specific time frame to offer at this time, but it's planned for publication before cPanel & WHM version 78 reaches the STABLE release tier.

Thank you.
 
  • Like
Reactions: brianc

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,910
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Hi Peter,

"Pending Publication" indicates we've reproduced the issue internally and we are planning to publish a fix in an upcoming version. A support ticket is not required when a thread is tagged with this prefix. I'll update this thread as soon as CPANEL-25247 is published.

Thank you.
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,910
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Hello Everyone,

This is fixed in cPanel & WHM version 78.0.17:

Fixed case CPANEL-25247: Fix erroneous use of continue instead of next in backblaze transport.

Please let us know if you encounter any additional issues after upgrading to this version.

Thank you.
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,910
2,233
363
cPanel Access Level
DataCenter Provider
Twitter

Denis Gomes Franco

Well-Known Member
Sep 3, 2018
45
6
8
Tupã, São Paulo, Brazil
cPanel Access Level
Root Administrator
Joining in here to drop my 2 cents on this. Switched from S3 to Backblaze two days ago in order to reduce my data storage bill and I'm having the issues described by the OP.

First it was a PITA to set up the account, BAD CREDENTIALS errors abound until I found out about what should I really put in these fields. Took me a good half an hour but okay, Backblaze finally validated.

But when it came to backup time large accounts failed. These files were split and only the first part was being uploaded. CPU spiked to 100% and only half my accounts were backed up. Sites also went down a couple of times while the CPU was at 100%. Seems like this is something related to Backblaze itself or the transport on Cpanel, because when using Amazon S3 everything sails smoothly and large accounts don't need splitting up. So I'm switching back to S3 for now.

Backblaze has great pricing but this makes it impossible to use. Also, here's a PSA: you cannot download files larger than 100 MB via Backblaze's web interface, you need to use the command line. Sure, that's not a big deal for advanced sysadmins but I am not one of them :D S3 does not complain about downloading large files via the web GUI, which makes it easier for me to just WGET the file back to the server whenever I need it.

EDIT: Forgot to mention that I'm on 78.0.21.
 
Last edited:

Denis Gomes Franco

Well-Known Member
Sep 3, 2018
45
6
8
Tupã, São Paulo, Brazil
cPanel Access Level
Root Administrator
And by the way, not related to Cpanel but curious nonetheless: I take care of some other Wordpress websites that aren't hosted on my server and thus I use UpdraftPlus to back them up. These are large sites and they also failed when backed up to Backblaze (whereas backups just worked fine on S3). I'll contact Backblaze support but I'm quite sure they will point it right back at Cpanel & Updraft.
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,910
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
But when it came to backup time large accounts failed. These files were split and only the first part was being uploaded. CPU spiked to 100% and only half my accounts were backed up. Sites also went down a couple of times while the CPU was at 100%. Seems like this is something related to Backblaze itself or the transport on Cpanel, because when using Amazon S3 everything sails smoothly and large accounts don't need splitting up. So I'm switching back to S3 for now.
Hello @Denis Gomes Franco,

It looks like this relates to case CPANEL-26694:

In Progress - [CPANEL-26694] Backblaze backup transport fails without error

I'll update the above thread as soon as the solution is published. Let me know if you have any questions.

Thank you.