SOLVED [CPANEL-26694] Backblaze backup transport fails without error

[brianc]

Continued from: SOLVED - [CPANEL-25310] BackBlaze transport error - Use of uninitialized value $args{"api_method"}

Well I updated the server to 78.0.20 and backup job ran but not the transport job. The log said cpbackup was queuing the files for transport but nothing actually happened.

 

[cPanelMichael]

Well I updated the server to 78.0.20 and backup job ran but not the transport job. The log said cpbackup was queuing the files for transport but nothing actually happened.

Hi @brianc,

Can you submit a support ticket for the affected system and let me know the ticket number? I'd like to have one of our Technical Analysts investigate to make sure this isn't a defect associated with the changes in CPANEL-25310.

Thank you.

 

[phil99]

Our server was upgraded last night to 78.0.20 before the backup to Backblaze ran. The backup ran successfully with all files transported, but I note that in the transport log, the api error still occurred:

[2019-04-04 02:01:41 +0100] info [cpbackup_transporter] Use of uninitialized value $args{"api_method"} in string ne at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 260.

 

[brianc]

Hi @brianc,

Can you submit a support ticket for the affected system and let me know the ticket number? I'd like to have one of our Technical Analysts investigate to make sure this isn't a defect associated with the changes in CPANEL-25310.

Thank you.

Done. Your Support Request ID is: 11857299. Perhaps I can get a free expedited request on this ticket? :^)

 

[cPanelMichael]

Hello Brian,

Thanks for opening the ticket and helping us track down a new issue.

CPANEL-26694 is now open to address an issue where the transfer of backups to a Backblaze destination fails without warning as a result of excessive memory usage. I'll monitor this case and update this thread with more information as it becomes available.

Thank you.

Our server was upgraded last night to 78.0.20 before the backup to Backblaze ran. The backup ran successfully with all files transported, but I note that in the transport log, the api error still occurred:

[2019-04-04 02:01:41 +0100] info [cpbackup_transporter] Use of uninitialized value $args{"api_method"} in string ne at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 260.

Hi @phil99,

This looks to be unrelated to case CPANEL-26694. Can you open a support ticket so we can rule out any additional defects? You can post the ticket number here and I'll link this thread to it.

Thank you.

 

[phil99]

Hello Brian,

Thanks for opening the ticket and helping us track down a new issue.

CPANEL-26694 is now open to address an issue where the transfer of backups to a Backblaze destination fails without warning as a result of excessive memory usage. I'll monitor this case and update this thread with more information as it becomes available.

Thank you.



Hi @phil99,

This looks to be unrelated to case CPANEL-26694. Can you open a support ticket so we can rule out any additional defects? You can post the ticket number here and I'll link this thread to it.

Thank you.

Hi Michael

Support request ID: 11869311

There were a number of the same errors in the transport again last night.

 

[brianc]

So I ran the backup on another server that was running 78.0.20 and it had some different issues:

1. No transporter log was created. This was similar behavior in the ticket (11857299) I opened for another server.
2. However most of the accounts were backed up and transported to Backblaze. No difference in change here however from previous bugs.
3. The one large account on the server still did not transport correctly so your recent fix (applied in 78.0.20) did not fix the original issue that was reported. It shows up at backblaze in parts and are not merged into one file as it is intended.

Edit:

4. I just realized a NEW issue appeared. Not all of the backup files were transported. So it seems the transported job just ended abruptly with no error message. This is similar behavior reported in ticket 11857299.

 

[cPanelMichael]

Support request ID: 11869311

There were a number of the same errors in the transport again last night.

Hello @phil99,

Thanks for submitting the ticket. The next cPanel & WHM version 78 build will address the "Use of uninitialized value $args{"api_method"}" warning that appears in the backup transport logs (despite a successful transport to BackBlaze). I'll update this thread once the next build is published.

1. No transporter log was created. This was similar behavior in the ticket (11857299) I opened for another server.
2. However most of the accounts were backed up and transported to Backblaze. No difference in change here however from previous bugs.
3. The one large account on the server still did not transport correctly so your recent fix (applied in 78.0.20) did not fix the original issue that was reported. It shows up at backblaze in parts and are not merged into one file as it is intended.

Edit:

4. I just realized a NEW issue appeared. Not all of the backup files were transported. So it seems the transported job just ended abruptly with no error message. This is similar behavior reported in ticket 11857299.

Hello @brianc,

CPANEL-26694 encompasses all of these issues (the transport to BackBlaze abruptly ends without logging a warning/error due to a lack of available memory). I'll continue to monitor this case and update this thread with more information as it becomes available.

Thank you.

 

[cPanelMichael]

@phil99, cPanel & WHM version 78.0.21 includes a fix for the issue you reported:
Fixed case CPANEL-25310: Fix uninitialized variable warning in Backblaze transport.

@brianc, It looks like we've narrowed down the cause of case CPANEL-26694 to part of the script that deals with the transport of backup files under 5GB. I'll continue to monitor the case and provide more information here as it becomes available. If you're interested in testing this, see if you can reproduce the issue when backing up accounts over 5GB.

Thank you.

 

[dataforge]

Following. I Have 78.0.21 and setup the b2 backup, but it fails also. files aren't finished transferring. Some do transfer, but not all. I've now gone back go aws with no problems there.

Please note anyone that has this issue you should log into the b2 control panel and browse your directories for Unfinished Large Files as this error is dropping a lot of these, and they count towards your usage.
You can also use this tool to do a delete
How Can I Easily Delete All Files in a Bucket?

 

[phil99]

Just to add that we saw this for the first time last night (WHM 78.0.21). A partial transport log was created, but it just ends suddenly, and I don't think there is anything useful in it.

 

[cPanelMichael]

Hello Everyone,

To update, the following cases are open to address issues with the BackBlaze B2 backup destination:

CPANEL-26694: Fix OOM error on file uploads smaller than 5GB in BackBlaze B2 transport.
CPANEL-26914: Large file uploads to Backblaze B2 destinations fail when auth token expires due.

CPANEL-26694 is fixed in cPanel & WHM version 80 (the development build is on the EDGE release tier), but it has not yet been published to cPanel & WHM version 78. I'll update this thread as soon as it's published.

CPANEL-26914 is still under investigation at this time. I'll continue to monitor this case and update this thread with more information as it becomes available.

Thank you.

 

[cPanelMichael]

Hello Everyone,

CPANEL-26694 was published to cPanel & WHM version 78.0.23 today:

Fixed case CPANEL-26694: Fix OOM error on file uploads smaller than 5GB in BackBlaze B2 transport.

Version 78.0.23 is published to the CURRENT release tier only at this time. I'll update this thread again once it's published to the RELEASE tier.

Thank you.

Update 05/08/2019: Version 78.0.23 is now published to the RELEASE tier.