Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

In Progress [CPANEL-27679] Backblaze B2 - large file not reassembled

Discussion in 'Data Protection' started by phil99, May 9, 2019.

  1. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    23
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Posting this as a new thread, as it appears to be different than [CPANEL-25247], where not all parts were uploaded.

    We have been testing backups to Backblaze B2. Last night for the first time, we uploaded a file larger than 5GB (the tar.gz size was about 11GB).

    According to the transport log, the file was successfully uploaded to Backblaze. However, when looking at the file on Backblaze, it has been split into 3 parts. There are two files of 5GB plus one of about 1GB. There is also a related manifest file:

    account.tar.gz._b2_manifest 93bytes
    account.tar.gz._b2_part_1 5GB
    account.tar.gz._b2_part_2 5GB
    account.tar.gz._b2_part_3 1.2GB

    The Backblaze B2 API imposes a limit of 5GB on the size of the file that can be uploaded. Larger files have to be split into parts for upload. However, once all parts are uploaded they need to be transformed into a single file with a separate API call. It would appear that all parts were successfully uploaded, but the reassembly didn't take place.

    There were no errors in the transport log to indicate this has not been done, it reports a successful transfer.
     
  2. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    23
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    We let this account backup run again last night. This time other errors occurred as well as the parts not being reassembled.

    The first attempt to upload the account.tar.gz failed with:

    Code:
    [2019-05-10 02:48:16 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2019-05-10/accounts/account.tar.gz to backup_cpanel/2019-05-10/accounts/account.tar.gz for destination:  Backblaze B2
    [2019-05-10 05:09:06 +0100] warn [cpbackup_transporter]
    Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 3yu366) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “POST” request to “https://api002.backblazeb2.com/b2api/v1/b2_finish_large_file” because of an error: CODE :400: MSG :400 Bad Request: {
      "code": "bad_request",
      "message": "Part number 951 has not been uploaded",
      "status": 400
    }
    There are then various other messages in the transport log from perl modules.

    A second upload attempt is then started and appears to have succeeded, though the parts were not reassembled. Also, this meant that on Backblaze:

    - there were two copies of the manifest
    - 2 copies of part 1 (each 5GB)
    - 2 copies of part 2 (one 5GB, on an unfinsihed file of 0 bytes)
    - 1 copy of part 3 (1.2GB)

    Hope that is helpful!
     
  3. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,273
    Likes Received:
    2,154
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @phil99,

    Can you verify if you have reproduced this behavior after updating to cPanel & WHM version 78.0.23?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    23
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi Michael

    Yes the upgrade to 78.0.23 occurred just after 23:00 on 8th May, so both of these runs (9th and 10th May) were on that version .

    Thanks.
     
  5. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,273
    Likes Received:
    2,154
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @phil99,

    While it may seem as though the backup/transport succeeded, it's possible there was an error during the backup process that wasn't fully recorded in the logs. In particular, case CPANEL-26914 is open to address an issue where large file transports fail when the authorization token expires due to the filepart counter incrementing inappropriately. Can you open a support ticket so we can take a closer look at your system and confirm if it's related to this particular case?

    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    23
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi Michael

    Support request ID: 12263991

    Thanks.
     
  7. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    23
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi

    The conclusion of the support ticket was also that the error message in the second run was similar to that seen in CPANEL-26914, and we would have to wait for the fix to that to be released to see if it fixes the error we saw. There was no mention of the the fact that the file was not reassembled. I wonder if the developers have been consulted as to whether this is the same issue?

    Thanks.
     
  8. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,273
    Likes Received:
    2,154
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @phil99,

    Thank you again for opening the support ticket. Your forum threads and tickets have helped us identify multiple issues with the BackBlaze B2 backup destination.

    Yes, this should happen automatically once case CPANEL-26914 is published. Here are the sections of /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm that pertain to reassembly:

    Code:
    We want to make sure we upload a large file in chunks rather than as a whole because that will conserve memory, we do not want to load a 4GB file into memory to upload it. 
    
    To implement this, we took advantage of the backblaze large file upload protocol
    
      b2_start_large_file
      b2_get_upload_part_url, and use that url to upload the chunks one at a time
      b2_finish_large_file
    And;

    Code:
    sub _get_finish_large_file {
        my ( $self, $large_file_info, $sha_ar ) = @_;
    
        my $content_hr = {
            'fileId'        => $large_file_info->{'fileId'},,
            'partSha1Array' => $sha_ar,
        };
    
        my $content = Cpanel::JSON::Dump($content_hr);
    
        my $response = $self->_do_request( 'api_method' => 'b2_finish_large_file', 'payload' => $content );
        my $response_data = Cpanel::JSON::Load( $response->content );
    
        return $response_data;
    }
    Backblaze B2 API Reference: b2_finish_large_file

    Let me know if this information helps.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    23
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi Michael

    That's great. Looking forward to the fix being published so we can test again.

    Regards, Phil
     
    cPanelMichael likes this.
  10. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    23
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi

    Last night, for the first time since upgrading to WHM 80.0.11, we again tested backing up the same large account as reported in the original post.

    Exactly the same thing has happened. The file has been split into 3 parts (two of 5GB and one of 1.8GB), and whilst all parts appear to have been successfully uploaded, the parts have not been reassembled into a single file on Backblaze. Again there are no errors in the log file, a successful transfer has been reported.
     
  11. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,273
    Likes Received:
    2,154
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @phil99,

    Can you report this as a separate defect using our Defects link? Let me know the ticket number generated for the report and I'll add some notes to the ticket explaining the issue.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    23
    Likes Received:
    5
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi Michael

    Request ID is: 12453139

    Some further information (which I have expanded on in the ticket with some log entries) is that the transport of the split file seems to take an excessively long time. An 11.8GB file took over 2 and a half hours to transport. Another account which was a 4.8GB file (and so presumably wasn't split) took just under 11 minutes in the same run.

    Let us know if you need any more information.

    Regards, Phil
     
    cPanelMichael likes this.
  13. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,273
    Likes Received:
    2,154
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hi Phil,

    Thanks, I've added a note to the ticket explaining the issue. I'll monitor the ticket and report the outcome here once it's closed.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,273
    Likes Received:
    2,154
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    To update, case CPANEL-27679 was opened to address an issue where split backup files are not reassembled on the Backblaze destination. I'll update this thread with more information on the status of this case as it becomes available.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice