[CPANEL-26887] Amazon S3 backup destination validation times out upon initial setup

[KristanM]

Hi all,
I have an issue backing up to S3 - I've read the threads about bucket names not containing dots and V4 bucket support - but I'm still seeing the following in the transport log:

[2018-12-14 11:49:18 +0000] info [cpbackup_transporter] Validating destination path /Backups/2018-12-14/system/
[2018-12-14 11:49:21 +0000] info [cpbackup_transporter] Making Path /Backups/2018-12-14/system/
[2018-12-14 11:49:23 +0000] warn [cpbackup_transporter] Path creation failed: PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x181eeb0), "Path creation failed: PermanentRedirect: The bucket you are "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 838
Cpanel::Backup::Queue::transport_backup::_validate_path(Cpanel::Backup::Queue::transport_backup=HASH(0x1676190), Cpanel::Transport::Files::AmazonS3=HASH(0x26be7f0), "/Backups/2018-12-14/system/system_files.tar", Cpanel::LoggerAdapter=HASH(0x181eeb0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 788
Cpanel::Backup::Queue::transport_backup::validate_path(Cpanel::Backup::Queue::transport_backup=HASH(0x1676190), Cpanel::Transport::Files::AmazonS3=HASH(0x26be7f0), "/Backups/2018-12-14/system/system_files.tar", Cpanel::LoggerAdapter=HASH(0x181eeb0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 255
Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1676190), cPanel::TaskQueue::Task=HASH(0x26b4900), Cpanel::LoggerAdapter=HASH(0x181eeb0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624
eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 233
eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 233
cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x26a4910), CODE(0x26a42c8)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x26a47c0)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165


I've tried buckets set up in eu-west-1 (Ireland) and eu-west-2 (London) and neither validate or succeed when trying to back up to them. I can happily connect to the bucket via the AWS CLI, upload files etc. however I do have to set the region. Is there some way to tell the backup transport to use a specific region? Or have I got something wrong somewhere?

Thanks!

 

[benito]

Hello!

Im trying to setup a S3 backup using a bucket in europe and fails validation. If i setup a bucket in USA it works fine.

I do not where to look to solve this problem.

Thanks

 

[benito]

@KristanM Hello! My EU bucket start working. Today i tried to validate and worked ok. I did nothing.

 

[KristanM]

Interesting - I left mine enabled, even though it wouldn't validate. I've just checked and it looks like the following day it started working and I now have 5 days of backups and a weekly correctly in my S3 bucket.

Has something changed I wonder, or do we need to wait some time between creating the bucket and using it for it to replicate somewhere.

Useful to know though, as I'm setting this up on several boxes now!

 

[cPanelMichael]

Hello @benito and @KristanM,

Despite backups generating successfully, I'd like to rule out any issues with the Amazon S3 backup destination validation process. Can you verify which cPanel & WHM version is installed on the affected system? Also, are you able to reproduce the validation issue on new S3 backup destinations (if you have the ability to test it)?

Thank you.

 

[benito]

Hello @benito and @KristanM,

Despite backups generating successfully, I'd like to rule out any issues with the Amazon S3 backup destination validation process. Can you verify which cPanel & WHM version is installed on the affected system? Also, are you able to reproduce the validation issue on new S3 backup destinations (if you have the ability to test it)?

Thank you.

Hello!

I created a new bucket in London, it gives:

"Error: Validation for transport “bucketname” failed: Timeout"

cPanel version v76.0.14. Will check it again tomorrow to see if it validates ok like the previous one.

 

[cPanelMichael]

I created a new bucket in London, it gives:

"Error: Validation for transport “bucketname” failed: Timeout"

Can you open a support ticket so we can take a closer look at the affected system and reproduce this issue? Post the ticket number here and I'll link this thread to it.

Thank you.

 

[benito]

Done, the ticket is 10995567

I created a new destination called cPanel Test Bucket on server.

 

[cPanelMichael]

Hello,

To update, internal case CPANEL-24550 CPANEL-26887 is now open to track reports of Amazon S3 backup destinations failing validation. In these reports, validation and backups subsequently succeed within 24 hours after the initial failed validation attempt. I'll monitor this case and update this thread with more information as it becomes available.

Thank you.

 

[KristanM]

Hello @benito and @KristanM,

Despite backups generating successfully, I'd like to rule out any issues with the Amazon S3 backup destination validation process. Can you verify which cPanel & WHM version is installed on the affected system? Also, are you able to reproduce the validation issue on new S3 backup destinations (if you have the ability to test it)?

Thank you.

v76.0.14 here too.

I've tried two new buckets, one in the US, one in the EU. Both fail to validate for me interestingly. The existing one still validates fine. Is there a log file for the validation process we can look at?

 

[KristanM]

It's certainly less than 24 hours to validate - the ones I created this morning now validate fine, so more like 6 hours perhaps?

 

[cPanelMichael]

Hello @KristanM,

I've updated this thread to reflect that this issue extends to buckets within North America, and relayed this information internally. I'll continue to monitor the internal case and update this thread with more information as it becomes available.

Is there a log file for the validation process we can look at?

There's no specific log file, but you can review the output from the following WHM API 1 function when testing validation:

WHM API 1 Functions - backup_destination_validate - Developer Documentation - cPanel Documentation

EX:

whmapi1 backup_destination_validate id=DESTINATION-ID disableonfail=0

To obtain the destination ID, you'd use the following WHM API 1 command:

whmapi1 backup_destination_list

Thank you.

 

[cPanelMichael]

Hello Everyone,

Can anyone confirm if this issue persists in cPanel & WHM version 78.0.20 (available in the CURRENT release tier) or higher?

Thank you.

 

[drakeba]

@cPanelMichael I know this is a few months old, but I can confirm having the same issue just now on WHM version 80.0.18. Created an s3 bucket in the Oregon region, validation timed out consistently. As a test, I validated against an existing s3 bucket from the N. Virginia region just fine.

 

[drakeba]

@cPanelMichael Just tried again about 90 minutes later, it validated. I assume this is an S3 issue, but hopefully this info helps rule it out on your end. (assuming you haven't done so already in the past few months)

 

[cPanelMichael]

Hello @drakeba,

Thanks for sharing the additional information. We're now tracking reports of this behavior as part of internal case CPANEL-26887. I've linked this thread to the case and will provide an update as soon as new information is available.

Thank you.

 

[cPanelLauren]

Hello,

I've just checked in on this case and it would appear that the development team for this has decided this isn't something they'll be able to address. The reasoning being as follows:

Putting enough information in the validation message would be incredibly messy. I will look into possibly giving a message in the transport setup. Unfortunately, there's no way for us to capture that it's a newly-created bucket, so we would have to give this message on every validation that fails on S3 for a bucket not found message.

If you feel that this is something that will be important to be included or if you have a different idea for how to mitigate it, I'd suggest opening a feature request for it. You can do so using the link in my signature. If you do open a feature request for this, please add the link here when it's accepted so others can view and vote on it.


Thanks!

 

[pacwebhosting]

Hi,

I have just encountered this exact same issue today.

After setting up an s3 bucket cpanel backup would not validate it but jetbackup would on the same server straight away.

I imagine that where cpanel is trying to validate it that it does indeed have to replicate to there as it worked hours later.

Mine was a UK bucket.

So maybe speak to the guys/partners at jetbackup and find out how they work out how to get it to validate straight away as they must have a method working out the location of the bucket and querying there

Thanks

Paul

 

[hoomanxx]

Hi,

I have same problem, When I want create s3 bucket, validation timed out consistently. my s3 bucket provider examined the problem and said: "cPanel's standard is not compatible with our services, you must set the region to default and cPanel have not this option".

How can I set the region to "default"?

Thanks

 

[cPanelLauren]

They give the following instructions on how to set the region: Choose an AWS Region - Amazon EMR