Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

In Progress [CPANEL-27679] Backblaze B2 - large file not reassembled

Discussion in 'Data Protection' started by phil99, May 9, 2019.

  1. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Posting this as a new thread, as it appears to be different than [CPANEL-25247], where not all parts were uploaded.

    We have been testing backups to Backblaze B2. Last night for the first time, we uploaded a file larger than 5GB (the tar.gz size was about 11GB).

    According to the transport log, the file was successfully uploaded to Backblaze. However, when looking at the file on Backblaze, it has been split into 3 parts. There are two files of 5GB plus one of about 1GB. There is also a related manifest file:

    account.tar.gz._b2_manifest 93bytes
    account.tar.gz._b2_part_1 5GB
    account.tar.gz._b2_part_2 5GB
    account.tar.gz._b2_part_3 1.2GB

    The Backblaze B2 API imposes a limit of 5GB on the size of the file that can be uploaded. Larger files have to be split into parts for upload. However, once all parts are uploaded they need to be transformed into a single file with a separate API call. It would appear that all parts were successfully uploaded, but the reassembly didn't take place.

    There were no errors in the transport log to indicate this has not been done, it reports a successful transfer.
     
  2. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    We let this account backup run again last night. This time other errors occurred as well as the parts not being reassembled.

    The first attempt to upload the account.tar.gz failed with:

    Code:
    [2019-05-10 02:48:16 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2019-05-10/accounts/account.tar.gz to backup_cpanel/2019-05-10/accounts/account.tar.gz for destination:  Backblaze B2
    [2019-05-10 05:09:06 +0100] warn [cpbackup_transporter]
    Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 3yu366) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “POST” request to “https://api002.backblazeb2.com/b2api/v1/b2_finish_large_file” because of an error: CODE :400: MSG :400 Bad Request: {
      "code": "bad_request",
      "message": "Part number 951 has not been uploaded",
      "status": 400
    }
    There are then various other messages in the transport log from perl modules.

    A second upload attempt is then started and appears to have succeeded, though the parts were not reassembled. Also, this meant that on Backblaze:

    - there were two copies of the manifest
    - 2 copies of part 1 (each 5GB)
    - 2 copies of part 2 (one 5GB, on an unfinsihed file of 0 bytes)
    - 1 copy of part 3 (1.2GB)

    Hope that is helpful!
     
  3. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @phil99,

    Can you verify if you have reproduced this behavior after updating to cPanel & WHM version 78.0.23?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi Michael

    Yes the upgrade to 78.0.23 occurred just after 23:00 on 8th May, so both of these runs (9th and 10th May) were on that version .

    Thanks.
     
  5. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @phil99,

    While it may seem as though the backup/transport succeeded, it's possible there was an error during the backup process that wasn't fully recorded in the logs. In particular, case CPANEL-26914 is open to address an issue where large file transports fail when the authorization token expires due to the filepart counter incrementing inappropriately. Can you open a support ticket so we can take a closer look at your system and confirm if it's related to this particular case?

    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi Michael

    Support request ID: 12263991

    Thanks.
     
  7. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi

    The conclusion of the support ticket was also that the error message in the second run was similar to that seen in CPANEL-26914, and we would have to wait for the fix to that to be released to see if it fixes the error we saw. There was no mention of the the fact that the file was not reassembled. I wonder if the developers have been consulted as to whether this is the same issue?

    Thanks.
     
  8. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @phil99,

    Thank you again for opening the support ticket. Your forum threads and tickets have helped us identify multiple issues with the BackBlaze B2 backup destination.

    Edit: Jul-16-2019:

    CPANEL-27679 will address this issue through the addition of a separate upload pathway for files larger than 5GB.


    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #8 cPanelMichael, May 15, 2019
    Last edited: Jul 16, 2019 at 11:27 AM
  9. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi Michael

    That's great. Looking forward to the fix being published so we can test again.

    Regards, Phil
     
    cPanelMichael likes this.
  10. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi

    Last night, for the first time since upgrading to WHM 80.0.11, we again tested backing up the same large account as reported in the original post.

    Exactly the same thing has happened. The file has been split into 3 parts (two of 5GB and one of 1.8GB), and whilst all parts appear to have been successfully uploaded, the parts have not been reassembled into a single file on Backblaze. Again there are no errors in the log file, a successful transfer has been reported.
     
  11. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @phil99,

    Can you report this as a separate defect using our Defects link? Let me know the ticket number generated for the report and I'll add some notes to the ticket explaining the issue.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Hi Michael

    Request ID is: 12453139

    Some further information (which I have expanded on in the ticket with some log entries) is that the transport of the split file seems to take an excessively long time. An 11.8GB file took over 2 and a half hours to transport. Another account which was a 4.8GB file (and so presumably wasn't split) took just under 11 minutes in the same run.

    Let us know if you need any more information.

    Regards, Phil
     
    cPanelMichael likes this.
  13. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hi Phil,

    Thanks, I've added a note to the ticket explaining the issue. I'll monitor the ticket and report the outcome here once it's closed.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    To update, case CPANEL-27679 was opened to address an issue where split backup files are not reassembled on the Backblaze destination. I'll update this thread with more information on the status of this case as it becomes available.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  15. cyberh3x

    cyberh3x Registered

    Joined:
    Oct 5, 2013
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Hello,

    Is there any progress on this? We would like to use B2 as we can save a lot of money in storage costs by using B2 vs other cloud platforms.

    Thanks
     
  16. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    We don't have a firm time frame to share on the publication of a fix at this time, however I've shared your feedback to the internal case and will update this thread as soon as new information is available.

    Thank you for your patience.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  17. phil99

    phil99 Member

    Joined:
    Jun 10, 2018
    Messages:
    24
    Likes Received:
    6
    Trophy Points:
    3
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Just to add that we have been using the backup to Backblaze successfully for our smaller accounts. The tar.gz file must be less than 5GB (5 billion bytes). Once they get larger than this, they need to be split, which is when the problems occur.
     
    cPanelMichael likes this.
  18. nhadsall

    nhadsall Member

    Joined:
    Nov 7, 2015
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    San Antonio, TX
    cPanel Access Level:
    Root Administrator
    Backups are such an essential part of our job as sysadmins. I don't understand how something so essential can be overlooked. Apparently my backups have not been working for weeks, and I'm just now seeing this. There was no error message to let me know. IT was lucky that I went to restore a backup in a non emergency situation. Had there been a panicked client in need of help, this would have been a massive headache.
     
  19. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @nhadsall,

    Can you confirm if you are experiencing issues with the BackBlaze B2 backup destination for all accounts, or is the issue isolated to accounts over 5GB (the issue noted in this thread)?

    The priority of this case was increased based on the feedback from this thread, but I don't have a specific time frame to offer on a solution at this time. I'll provide more updates as they become available.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice