SOLVED CPANEL-36050 - Backblaze B2: Backup transport errors

phil99 · Feb 26, 2021

We have occasional problems when uploading to Backblaze B2, with backups failing to upload.

There are errors in cpbackup_transporter.log like the following example (actual account name has been replaced with "accountx"):

Code:

[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Uploading account backup /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz (from 2021-02-23/accounts/account1.tar.gz)
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Attempting to upload /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz for destination:  Backblaze B2
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz for destination:  Backblaze B2
[2021-02-23 05:03:19 +0000] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 4rn8t3) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/530/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
    Cpanel::Transport::Files::Backblaze::_error(Cpanel::Transport::Files::Backblaze=HASH(0x37befe8), "HTTP::Network", HASH(0x31660e8)) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 272

[Lots more perl stuff snipped]

The backup software then immediately tries twice more:

Code:

[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Upload attempt #2 starting for /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz for destination:  Backblaze B2
[2021-02-23 05:03:19 +0000] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 9ed6ug) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)
...
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Upload attempt #3 starting for /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz for destination:  Backblaze B2
[2021-02-23 05:03:19 +0000] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 24zw5x) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

After failing for a third time, it gives up and immediately moves onto the next account:

Code:

[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2021-02-23/accounts/account2.tar.gz to backup_cpanel/2021-02-23/accounts/account2.tar.gz for destination:  Backblaze B2
[2021-02-23 05:03:19 +0000] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID amkjxg) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

This carries on with a number of accounts failing to upload until there is a successful upload attempt 3 seconds later:

Code:

[2021-02-23 05:03:22 +0000] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2021-02-23/accounts/account3.tar.gz to backup_cpanel/2021-02-23/accounts/account3.tar.gz for destination:  Backblaze B2
[2021-02-23 05:03:25 +0000] info [cpbackup_transporter] Successful transfer of /backup_cpanel/2021-02-23/accounts/account3.tar.gz to backup_cpanel/2021-02-23/accounts/account3.tar.gz for destination Backblaze B2

I'm assuming the "error: CODE :500" is the error code received from Backblaze. If so, then according to the Backblaze blog post below [1], an error code of 500 is to be expected. If the application receives an error 500 code, it should immediately try again after getting a new url. However, if it gets another 500 code, it should wait a few seconds before trying again. It would appear that the WHM backup software is not doing this. This has resulted in the failure to upload a number of accounts.

[1] Why You Are Getting a B2 503 or 500 Server Error and What to Do Next

cPRex · Feb 26, 2021

Hey there! Thanks for bringing this to my attention. I've created case CPANEL-36050 to have our developer explore the possibility of introducing a delay in the Backblaze upload settings when an error is encountered. I'm following along with that case and I'll make sure to post an update here as soon as I hear something.

phil99 · Feb 26, 2021

Thanks Rex.

cPRex · Feb 26, 2021

Sure thing!

cPRex · Apr 7, 2021

Update - we have a fix for this that will be included in version 96 when this gets released :D

phil99 · Apr 7, 2021

Great, thanks.

PhxChris · Jun 6, 2021

Sorry, but still seeing this error in version v.96.0.9. Should be pausing for a few seconds after failure #2, but that's not happening.

First few backups upload with no problem, and then B2 get's busy and the script does not pause.

Here's an excerpt from my logs/transport_errors.txt

[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "copy" operation on the "Backblaze" destination ID "[[hidden]]".
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Validating destination path weekly/2021-06-06/accounts/
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Making Path weekly/2021-06-06/accounts/
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Path creation successful
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading account backup /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz (from weekly/2021-06-06/accounts/[[hidden]].tar.gz)
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Attempting to upload /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID npjcr9) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID peybg9) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID cquwju) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "copy" operation on the "Backblaze" destination ID "[[hidden]]".
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Validating destination path 2021-06-06/accounts/
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Making Path 2021-06-06/accounts/
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Path creation successful
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading account backup /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz (from 2021-06-06/accounts/[[hidden]].tar.gz)
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Attempting to upload /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID xnpbj9) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 7pygtp) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID w73j6d) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "copy" operation on the "Backblaze" destination ID "[[hidden]]".
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Validating destination path weekly/2021-06-06/accounts/
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Making Path weekly/2021-06-06/accounts/
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Path creation successful
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading account backup /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz (from weekly/2021-06-06/accounts/[[hidden]].tar.gz)
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Attempting to upload /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID r9aur6) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID pmqg37) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID nxv2q7) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "copy_backup_metadata" operation on the "Backblaze" destination ID "[[hidden]]".
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading backup metadata file /backup/2021-06-06/accounts/.master.meta to Backblaze
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Validating destination path 2021-06-06/accounts/
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Making Path 2021-06-06/accounts/
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Path creation successful
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading backup metadata /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta (from 2021-06-06/accounts/.master.meta)
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Attempting to upload /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta for destination: Backblaze
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID zxfwpf) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID qudqvq) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta for destination: Backblaze
[2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID g2882n) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)

Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
[[DUMP]]
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object
[2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "prune" operation on the "Backblaze" destination ID "[[hidden]]".

phil99 · Jun 7, 2021

OP here. We also continued to see this. A cPanel Analyst has recently been working on our system and think they have identified a fix. There is a support article you can follow, but I'm not sure if I can link to this. May be cPRex can add more detail.

cPJustinD · Jun 10, 2021

Hello phil99. Would you mind providing the ticket ID so that we can review the resolution information that was provided in the ticket?

phil99 · Jun 11, 2021

Hi, ticket #94327364.

dmart1n · Jun 11, 2021

I am also seeing issues with Backblaze offsite backups since Version 96.0.9

The problems I am seeing are:

1. When transferring files, it is no longer creating a new folder on Backblaze

It now always uses the same folder 2021-05-29/ and overwrites the backup from the day before.

It should create a new folder for each day - 2021-06-09 etc.

2021-06-09 is being saved to 2021-05-29/

[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] Initializing log file
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - parent starting
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - child starting
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon started.
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - started
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] Instantiating Object
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] Starting a "copy" operation on the "server3-v2" destination ID "6a8vXi_rRC0gbsB9MJJ6xRe5".
[2021-06-09 02:03:10 +0100] info [cpbackup_transporter] Validating destination path 2021-05-29/accounts/
[2021-06-09 02:03:11 +0100] info [cpbackup_transporter] Path exists

2021-06-06 is being saved to 2021-05-29/

[2021-06-06 02:03:11 +0100] info [cpbackup_transporter] Base path for destination is server3
[2021-06-06 02:03:11 +0100] info [cpbackup_transporter] Validating destination path server3/2021-05-28/accounts/
[2021-06-06 02:03:13 +0100] info [cpbackup_transporter] Path exists

All new backup are saved to 2021-05-29/

2. It is also skipping some backups and the error is

info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269.

3. I am also seeing this error

2021-06-09 04:19:26 +0100] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2021-06-09 04:19:26 +0100] info [cpbackup_transporter] Instantiating Object
[2021-06-09 04:19:28 +0100] info [cpbackup_transporter] Starting a "copy" operation on the "backup06" destination ID "RXeICSEibtmSh81YlG0oFuEP".
[2021-06-09 04:19:28 +0100] info [cpbackup_transporter] Base path for destination is server3
[2021-06-09 04:19:28 +0100] info [cpbackup_transporter] Validating destination path server3/2021-05-29/accounts/
[2021-06-09 04:19:29 +0100] info [cpbackup_transporter] Path exists
[2021-06-09 04:19:29 +0100] info [cpbackup_transporter] Uploading account backup /backup/2021-05-29/accounts/[hidden].tar.gz to server3/2021-05-29/accounts/[hidden].tar.gz (from 2021-05-29/accounts/[hidden].tar.gz)
[2021-06-09 04:19:29 +0100] info [cpbackup_transporter] Attempting to upload /backup/2021-05-29/accounts/[hidden].tar.gz to server3/2021-05-29/accounts/[hidden].tar.gz for destination: backup06
[2021-06-09 04:19:29 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-05-29/accounts/[hidden].tar.gz to server3/2021-05-29/accounts/[hidden].tar.gz for destination: backup06
[2021-06-09 04:20:46 +0100] warn [cpbackup_transporter] Upload attempt failed: rsync failed: command exited with code 255: Unknown rsync error
rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.2]
at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x13af588), "Upload attempt failed: rsync failed: command exited with code"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 633
Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x11916e8), Cpanel::Transport::Files::Rsync=HASH(0x242eb20), "/backup/2021-05-29/accounts/[hidden].tar.gz", "server3/2021-05-29/accounts/[hidden].tar.gz", Cpanel::LoggerAdapter=HASH(0x13af588)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333
Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x11916e8), Cpanel::TaskQueue::Task=HASH(0x390fac8), Cpanel::LoggerAdapter=HASH(0x13af588)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635
eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638
Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x3bb6188), CODE(0x2404bc8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643
Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2190298)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168
eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166
[2021-06-09 04:20:48 +0100] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-05-29/accounts/[hidden].tar.gz to server3/2021-05-29/accounts/[hidden].tar.gz for destination: backup06
[2021-06-09 04:21:29 +0100] warn [cpbackup_transporter] Upload attempt failed: rsync failed: command exited with code 255: Unknown rsync error
rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.2]
at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x13af588), "Upload attempt failed: rsync failed: command exited with code"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 633
Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x11916e8), Cpanel::Transport::Files::Rsync=HASH(0x242eb20), "/backup/2021-05-29/accounts/[hidden].tar.gz", "server3/2021-05-29/accounts/[hidden].tar.gz", Cpanel::LoggerAdapter=HASH(0x13af588)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333
Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x11916e8), Cpanel::TaskQueue::Task=HASH(0x390fac8), Cpanel::LoggerAdapter=HASH(0x13af588)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635
eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638
Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x3bb6188), CODE(0x2404bc8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643
Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2190298)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168
eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166

2021-06-06 02:09:28 +0100] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2021-06-06 02:09:28 +0100] info [cpbackup_transporter] Instantiating Object
[2021-06-06 02:09:29 +0100] info [cpbackup_transporter] Starting a "copy" operation on the "backup06" destination ID "RXeICSEibtmSh81YlG0oFuEP".
[2021-06-06 02:09:29 +0100] info [cpbackup_transporter] Base path for destination is server3
[2021-06-06 02:09:29 +0100] info [cpbackup_transporter] Validating destination path server3/2021-05-28/accounts/
[2021-06-06 02:09:30 +0100] info [cpbackup_transporter] Path exists
[2021-06-06 02:09:30 +0100] info [cpbackup_transporter] Uploading account backup /backup/2021-05-28/accounts/[hidden].tar.gz to server3/2021-05-28/accounts/[hidden].tar.gz (from 2021-05-28/accounts/[hidden].tar.gz)
[2021-06-06 02:09:30 +0100] info [cpbackup_transporter] Attempting to upload /backup/2021-05-28/accounts/[hidden].tar.gz to server3/2021-05-28/accounts/[hidden].tar.gz for destination: backup06
[2021-06-06 02:09:30 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-05-28/accounts/[hidden].tar.gz to server3/2021-05-28/accounts/[hidden].tar.gz for destination: backup06
[2021-06-06 02:09:30 +0100] warn [cpbackup_transporter] Upload attempt failed: rsync failed: command exited with code 23: partial transfer due to error
rsync: change_dir "/backup/2021-05-28/accounts" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2]
at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x20e5888), "Upload attempt failed: rsync failed: command exited with code"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 633
Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1ec76e8), Cpanel::Transport::Files::Rsync=HASH(0x3ea51e8), "/backup/2021-05-28/accounts/[hidden].tar.gz", "server3/2021-05-28/accounts/[hidden].tar.gz", Cpanel::LoggerAdapter=HASH(0x20e5888)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333
Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1ec76e8), Cpanel::TaskQueue::Task=HASH(0x3e7bc90), Cpanel::LoggerAdapter=HASH(0x20e5888)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635
eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638
Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x2f97350), CODE(0x325a1a8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643
Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2ec65b8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168
eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166
[2021-06-06 02:09:32 +0100] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-05-28/accounts/[hidden].tar.gz to server3/2021-05-28/accounts/[hidden].tar.gz for destination: backup06

4. and this error: SSL connect attempt failed because of handshake problems

[2021-05-26 04:02:08 +0100] info [cpbackup_transporter] Attempting to upload /backup/2021-05-26/accounts/[hidden].tar.gz to 2021-05-26/accounts/[hidden].tar.gz for destination: Backblaze Backups
[2021-05-26 04:02:08 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-05-26/accounts/[hidden].tar.gz to 2021-05-26/accounts/[hidden].tar.gz for destination: Backblaze Backups
[2021-05-26 04:50:53 +0100] info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 297.
[2021-05-26 04:50:53 +0100] info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269.
[2021-05-30 00:31:57 +0100] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID bfj2ry) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://api.backblazeb2.com/b2api/v1/b2_authorize_account” because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (SSL connect attempt failed because of handshake problems): Can't connect to api.backblazeb2.com:443 (SSL connect attempt failed because of handshake problems)

SSL connect attempt failed because of handshake problems at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.

at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
Cpanel::Transport::Files::Backblaze::_error(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), "HTTP::Network", HASH(0x2e6d8518)) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 273
Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 297
Cpanel::Transport::Files::Backblaze::_get_authorize_token(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), 1) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 268
Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), "content_type", "application/octet-stream", "api_ver", "v1", "custom_url", "[[HIDDEN]]"..., "payload", ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269
Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), "extra_headers", HASH(0x37e3770), "payload", CODE(0x3844800), "lives_on_error", 1, "request_type", ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269
Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__..., __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269
Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), "request_type", "POST", "lives_on_error", 1, "payload", CODE(0x3844800), "extra_headers", ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269

cPJustinD · Jun 11, 2021

There was a case filed on this. More information on this case can be found here:

cPanel Backup Transporter BackBlaze error: 500 Can't connect to api.backblazeb2.com:443

cPRex · Jun 15, 2021

Update - the fix is scheduled to be included in version 96.0.11.

dmart1n · Jun 17, 2021

Hi

All our servers have all been upgraded to 96.0.11

The path creation issues seems to be resolved as the path is being created.

The backups are not working due to the following error

[2021-06-16 23:23:47 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-16/accounts/[hidden].tar.gz to 2021-06-16/accounts/[hidden].tar.gz for destination: Backblaze Backups
[2021-06-17 00:06:04 +0100] info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 297.
[2021-06-17 00:06:04 +0100] info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269.

The servers are all crashing as a result and I have had to disable the offsite backups to Backblaze

Can you review and provide a solution?

Thanks

cPRex · Jun 17, 2021

@dmart1n - could you submit a ticket to our team so we can check this on one of your affected machines?

pyrographics · Jun 24, 2021

I'm in the same boat as dmart1n with exactly the same errors affecting at least two of my servers, I have a ticket open #94335602

cPRex · Jun 24, 2021

Our team has opened case CPANEL-37482 to track the issue on new tickets as this seems to be happening again after it was recently fixed. You'll be getting a reply to the ticket saying something similar most likely, but we are aware of the issue.

MindServer · Jul 1, 2021

Hi,

I have the version 94.0.12 and have the same problem: WHM/cPanel not finalize sending the backups to remote server:

Unable to send “/backup/2021-07-01/accounts/account71.tar.gz to destination "backup-server"

Unable to send “/backup/2021-07-01/accounts/account72.tar.gz to destination "backup-server"

Unable to send “/backup/2021-07-01/accounts/account73.tar.gz to destination "backup-server"

Unable to send “/backup/2021-07-01/accounts/account74.tar.gz to destination "backup-server"

Thank you very much. Have a nice day.

cPRex · Jul 1, 2021

@MindServer - we aren't expecting the issue to be resolved on version 94. The case mentioned above is still being investigated at this time.

MindServer · Jul 1, 2021

cPRex said:
@MindServer - we aren't expecting the issue to be resolved on version 94. The case mentioned above is still being investigated at this time.

Ok, thank you very much.

Please notify us in this post when the problem is solved. Have a nice day.

Thread starter	Similar threads	Forum	Replies	Date
T	In Progress CPANEL-42332 - Backups of a Suspended Domain	Backups	4	Feb 1, 2023
M	Never Ending Cpanel Account Restore	Backups	9	Jan 18, 2023
S	Is it possible to backup /etc from within cPanel?	Backups	1	Jan 13, 2023
I	Does cPanel backup accounts 1-by-1 or multiple at once?	Backups	1	Jan 12, 2023
M	Client cPanel Backup Issue after updating to latest theme	Backups	3	Nov 4, 2022

SOLVED CPANEL-36050 - Backblaze B2: Backup transport errors

Well-Known Member

Jurassic Moderator

Well-Known Member

Jurassic Moderator

Jurassic Moderator

Well-Known Member

Registered

Well-Known Member

Administrator

Well-Known Member

Registered

Administrator

Jurassic Moderator

Registered

Jurassic Moderator

Registered

Jurassic Moderator

Well-Known Member

Jurassic Moderator

Well-Known Member