Unable to prune: The server has stopped responding - S3 - 11.52.1 (build 3)

[coursevector]

Two days in a row we've gotten S3 backup errors. It's happened 9 times in the pass month across 3 servers when they have been going smoothly for months. Wondering if something changed in the recent updates to cPanel.

12/18
Unable to prune transport “server4”
Unable to remove outdated backup: The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.

12/17
Unable to prune transport “server4”
Unable to remove outdated backup: The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.

12/16
Unable to prune transport “server10”
Unable to remove outdated backup: The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.

12/13
Unable to prune transport “server10”
Unable to remove outdated backup: The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.

12/2
Unable to prune transport “server14”
Unable to remove outdated backup: The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.

11/29
Unable to prune transport “server14”
Unable to remove outdated backup: The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.

11/22
Unable to prune transport “server14”
Unable to remove outdated backup: The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.

11/21
Unable to send “/backup/2015-11-21/accounts/boroughs.tar.gz” to destination “psabdns4”

11/18
Unable to prune transport “server14”
Unable to remove outdated backup: The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.

Here is an excerpt from the transport log for server4:

[2015-12-18 02:58:55 -0500] warn [cpbackup_transporter] Unable to prune:  The server has stopped responding at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 333.
at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
    Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0xd77cb8), "Unable to prune:  The server has stopped responding at /usr/l"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 529
    Cpanel::Backup::Queue::transport_backup::attempt_to_prune_destination(Cpanel::Backup::Queue::transport_backup=HASH(0x14cdf48), Cpanel::Transport::Files::AmazonS3=HASH(0x23fee40), 1, Cpanel::LoggerAdapter=HASH(0xd77cb8)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 216
    Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x14cdf48), cPanel::TaskQueue::Task=HASH(0x2623948), Cpanel::LoggerAdapter=HASH(0xd77cb8)) called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 582
    eval {...} called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 585
    cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 238
    eval {...} called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 238
    cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x17ddc00), CODE(0x1d4d1d0)) called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 590
    cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x14db0d8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 151
    eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 149

 

[cPanelMichael]

Hello

Have you made any changes to the server that could limit connections to remote destinations (e.g. new firewall rules, router changes)?

Thank you.

 

[coursevector]

Michael,

Nothing that I have done recently. I had a DNS issue recently that involved my host (LiquidWeb) messing with the HOSTS file and the servers DNS. They changed the DNS from an internal network DNS to Google's DNS. But it's since been reversed back to how it's been. No messing with the firewall or anything else to my knowledge. My host said it's an issue with cPanel and it's connection with Amazon S3 and that they (you guys) will probably fix it in a patch soon.

 

[cPanelMichael]

My host said it's an issue with cPanel and it's connection with Amazon S3 and that they (you guys) will probably fix it in a patch soon.

Could you let us know the case number they are referencing? The case I see that's possibly related involves an issue where monthly and weekly backups do not get pruned in Amazon S3 buckets (#CPANEL-3326).

Thank you.

 

[coursevector]

They didn't reference a specific case:

After digging into this I have found that this issue looks to be a result of the most recent cpanel update to version 11.52.1 build 3. As this is an issue with the interaction between Cpanel and the Amazon S3 pod, we will have to wait for this issue to be corrected by Cpanel. I anticipate the fix being released within the next couple Cpanel updates, however I have no way to verify when the fix will be pushed out.

 

[cPanelMichael]

Feel free to open a support ticket using the link in my signature so we may review your backup logs and verify if the issue is related to any specific cases. You can post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[coursevector]

Support Request Id 7414677

I did this for the server that is having the issue most often, but the other two servers listed are most likely having the same issue.