SOLVED [CPANEL-27679] Backblaze B2 - large file not reassembled

phil99

Active Member
Jun 10, 2018
34
9
8
UK
cPanel Access Level
Root Administrator
Posting this as a new thread, as it appears to be different than [CPANEL-25247], where not all parts were uploaded.

We have been testing backups to Backblaze B2. Last night for the first time, we uploaded a file larger than 5GB (the tar.gz size was about 11GB).

According to the transport log, the file was successfully uploaded to Backblaze. However, when looking at the file on Backblaze, it has been split into 3 parts. There are two files of 5GB plus one of about 1GB. There is also a related manifest file:

account.tar.gz._b2_manifest 93bytes
account.tar.gz._b2_part_1 5GB
account.tar.gz._b2_part_2 5GB
account.tar.gz._b2_part_3 1.2GB

The Backblaze B2 API imposes a limit of 5GB on the size of the file that can be uploaded. Larger files have to be split into parts for upload. However, once all parts are uploaded they need to be transformed into a single file with a separate API call. It would appear that all parts were successfully uploaded, but the reassembly didn't take place.

There were no errors in the transport log to indicate this has not been done, it reports a successful transfer.
 

phil99

Active Member
Jun 10, 2018
34
9
8
UK
cPanel Access Level
Root Administrator
We let this account backup run again last night. This time other errors occurred as well as the parts not being reassembled.

The first attempt to upload the account.tar.gz failed with:

Code:
[2019-05-10 02:48:16 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2019-05-10/accounts/account.tar.gz to backup_cpanel/2019-05-10/accounts/account.tar.gz for destination:  Backblaze B2
[2019-05-10 05:09:06 +0100] warn [cpbackup_transporter]
Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 3yu366) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “POST” request to “https://api002.backblazeb2.com/b2api/v1/b2_finish_large_file” because of an error: CODE :400: MSG :400 Bad Request: {
  "code": "bad_request",
  "message": "Part number 951 has not been uploaded",
  "status": 400
}
There are then various other messages in the transport log from perl modules.

A second upload attempt is then started and appears to have succeeded, though the parts were not reassembled. Also, this meant that on Backblaze:

- there were two copies of the manifest
- 2 copies of part 1 (each 5GB)
- 2 copies of part 2 (one 5GB, on an unfinsihed file of 0 bytes)
- 1 copy of part 3 (1.2GB)

Hope that is helpful!
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hello @phil99,

Can you verify if you have reproduced this behavior after updating to cPanel & WHM version 78.0.23?

Thank you.
 

phil99

Active Member
Jun 10, 2018
34
9
8
UK
cPanel Access Level
Root Administrator
Hi Michael

Yes the upgrade to 78.0.23 occurred just after 23:00 on 8th May, so both of these runs (9th and 10th May) were on that version .

Thanks.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hello @phil99,

While it may seem as though the backup/transport succeeded, it's possible there was an error during the backup process that wasn't fully recorded in the logs. In particular, case CPANEL-26914 is open to address an issue where large file transports fail when the authorization token expires due to the filepart counter incrementing inappropriately. Can you open a support ticket so we can take a closer look at your system and confirm if it's related to this particular case?

Thanks!
 

phil99

Active Member
Jun 10, 2018
34
9
8
UK
cPanel Access Level
Root Administrator
Hi

The conclusion of the support ticket was also that the error message in the second run was similar to that seen in CPANEL-26914, and we would have to wait for the fix to that to be released to see if it fixes the error we saw. There was no mention of the the fact that the file was not reassembled. I wonder if the developers have been consulted as to whether this is the same issue?

Thanks.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hello @phil99,

Thank you again for opening the support ticket. Your forum threads and tickets have helped us identify multiple issues with the BackBlaze B2 backup destination.

Edit: Jul-16-2019:

CPANEL-27679 will address this issue through the addition of a separate upload pathway for files larger than 5GB.


Thank you.
 
Last edited:

phil99

Active Member
Jun 10, 2018
34
9
8
UK
cPanel Access Level
Root Administrator
Hi

Last night, for the first time since upgrading to WHM 80.0.11, we again tested backing up the same large account as reported in the original post.

Exactly the same thing has happened. The file has been split into 3 parts (two of 5GB and one of 1.8GB), and whilst all parts appear to have been successfully uploaded, the parts have not been reassembled into a single file on Backblaze. Again there are no errors in the log file, a successful transfer has been reported.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Exactly the same thing has happened. The file has been split into 3 parts (two of 5GB and one of 1.8GB), and whilst all parts appear to have been successfully uploaded, the parts have not been reassembled into a single file on Backblaze. Again there are no errors in the log file, a successful transfer has been reported.
Hello @phil99,

Can you report this as a separate defect using our Defects link? Let me know the ticket number generated for the report and I'll add some notes to the ticket explaining the issue.

Thank you.
 

phil99

Active Member
Jun 10, 2018
34
9
8
UK
cPanel Access Level
Root Administrator
Hi Michael

Request ID is: 12453139

Some further information (which I have expanded on in the ticket with some log entries) is that the transport of the split file seems to take an excessively long time. An 11.8GB file took over 2 and a half hours to transport. Another account which was a 4.8GB file (and so presumably wasn't split) took just under 11 minutes in the same run.

Let us know if you need any more information.

Regards, Phil
 
  • Like
Reactions: cPanelMichael

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hi Phil,

Thanks, I've added a note to the ticket explaining the issue. I'll monitor the ticket and report the outcome here once it's closed.

Thank you.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hello,

To update, case CPANEL-27679 was opened to address an issue where split backup files are not reassembled on the Backblaze destination. I'll update this thread with more information on the status of this case as it becomes available.

Thank you.
 

cyberh3x

Member
Oct 5, 2013
6
2
3
cPanel Access Level
Root Administrator
Hello,

Is there any progress on this? We would like to use B2 as we can save a lot of money in storage costs by using B2 vs other cloud platforms.

Thanks
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Is there any progress on this? We would like to use B2 as we can save a lot of money in storage costs by using B2 vs other cloud platforms.
We don't have a firm time frame to share on the publication of a fix at this time, however I've shared your feedback to the internal case and will update this thread as soon as new information is available.

Thank you for your patience.
 

phil99

Active Member
Jun 10, 2018
34
9
8
UK
cPanel Access Level
Root Administrator
Just to add that we have been using the backup to Backblaze successfully for our smaller accounts. The tar.gz file must be less than 5GB (5 billion bytes). Once they get larger than this, they need to be split, which is when the problems occur.
 
  • Like
Reactions: cPanelMichael

nhadsall

Member
Nov 7, 2015
7
0
1
San Antonio, TX
cPanel Access Level
Root Administrator
Backups are such an essential part of our job as sysadmins. I don't understand how something so essential can be overlooked. Apparently my backups have not been working for weeks, and I'm just now seeing this. There was no error message to let me know. IT was lucky that I went to restore a backup in a non emergency situation. Had there been a panicked client in need of help, this would have been a massive headache.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,222
463
Hello @nhadsall,

Can you confirm if you are experiencing issues with the BackBlaze B2 backup destination for all accounts, or is the issue isolated to accounts over 5GB (the issue noted in this thread)?

The priority of this case was increased based on the feedback from this thread, but I don't have a specific time frame to offer on a solution at this time. I'll provide more updates as they become available.

Thank you.
 

cyberh3x

Member
Oct 5, 2013
6
2
3
cPanel Access Level
Root Administrator
Hello @nhadsall,

Can you confirm if you are experiencing issues with the BackBlaze B2 backup destination for all accounts, or is the issue isolated to accounts over 5GB (the issue noted in this thread)?

The priority of this case was increased based on the feedback from this thread, but I don't have a specific time frame to offer on a solution at this time. I'll provide more updates as they become available.

Thank you.

Hello,

Yes this is for all accounts over 5GB. All smaller accounts work fine. About 80% of our clients are over 5gb.

This is a pretty big important issue since it does impact mission critical operations.

Thanks