In Progress CPANEL-40457 - Restarting cpbackup_transporter skips the current backup file

[AGY]

I occasionally see an issue where uploads to Backblaze slow or perhaps hang. I then restart the process with cpbackup_transporter --stop and then --start. It would appear that the shutdown process marks the file in progress done, because when it starts back up it grabs the next file.

[2022-04-11 15:29:56 -0400] info [cpbackup_transporter] Uploading account backup /backup/2022-04-11/accounts/account1.tar.gz to 2022-04-11/accounts/account1.tar.gz (from 2022-04-11/accounts/account1.tar.gz)
[2022-04-11 15:29:56 -0400] info [cpbackup_transporter] Attempting to upload /backup/2022-04-11/accounts/account1.tar.gz to 2022-04-11/accounts/account1.tar.gz for destination: Backblaze-CP
[2022-04-11 15:29:56 -0400] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2022-04-11/accounts/account1.tar.gz to 2022-04-11/accounts/account1.tar.gz for destination: Backblaze-CP

# /usr/local/cpanel/bin/cpbackup_transporter --stop
[2022-04-11 16:44:25 -0400] info [cpbackup_transporter] Initializing log file
Graceful shutdown of cPanel Backup Transporter Queue Daemon requested.
Graceful shutdown due to SIGHUP 34064
[2022-04-11 16:44:25 -0400] info [cpbackup_transporter] Graceful shutdown due to SIGHUP
Shutdown complete
[2022-04-11 16:44:25 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2022-04-11 16:44:25 -0400] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped.
[2022-04-11 16:45:38 -0400] info [cpbackup_transporter] cpbackup_transporter - started
[2022-04-11 16:45:38 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2022-04-11 16:45:38 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2022-04-11 16:45:38 -0400] info [cpbackup_transporter] Instantiating Object
[2022-04-11 16:45:38 -0400] info [cpbackup_transporter] Starting a "copy" operation on the "Backblaze-CP" destination ID "<snip>".
[2022-04-11 16:45:38 -0400] info [cpbackup_transporter] Validating destination path 2022-04-11/accounts/
[2022-04-11 16:45:40 -0400] info [cpbackup_transporter] Path exists
[2022-04-11 16:45:40 -0400] info [cpbackup_transporter] Uploading account backup /backup/2022-04-11/accounts/account2.tar.gz to 2022-04-11/accounts/account2.tar.gz (from 2022-04-11/accounts/account2.tar.gz)

When the transport finishes the skipped files will still exist in the backup directory and are not present in the Backblaze bucket.

cpanel_and_whm: 11.102.0
operating_system_name: centos
operating_system_version: 7

 

[cPRex]

Hey there! Do you know if you are using the B2 or S3 protocol? We see *much* better results when using the "S3Compatible" transport in the backup system than the B2.

 

[AGY]

We are using the B2 protocol. I'll flip it to S3 to see about fixing the periodic glitches, but the skipping files bit would likely still be there.

 

[cPRex]

If you can try that and then report back about any lingering issues I can make a case from there.

 

[AGY]

I made the switch last night and the troubles continue. I'm investigating the throughput issues with my provider (I can get 30 MB/s for a while and then it throttles back to a couple hundred KB/s), but the skipping backup bit is still the case. A backup being transported gets "lost" when cpbackup_transporter is stopped and restarted.

 

[phil99]

We have also seen this behaviour of losing the backup that was in progress when you stop and restart the cpbackup_transporter process. Again this was with Backblaze B2 using the B2 protocol. (It is not easy for us to swap protocols, but the bucket is not S3 enabled.)

 

[cPRex]

@AGY or @phil99 - could you submit a ticket to our team so we can see this on a server experiencing the issue?

 

[AGY]

Submitted #94436570.

 

[cPRex]

Thanks for that - I'm following along with that ticket on my end now.

 

[cPRex]

Our team was able to create case CPANEL-40457 with our developers to look into this behavior. You can follow along with this case at https://support.cpanel.net/hc/en-us...ansporter-may-leave-queued-uploads-unfinished for updates.