[CPANEL-26887] Amazon S3 backup destination validation times out upon initial setup

KristanM

Registered
Dec 14, 2018
4
0
1
UK
cPanel Access Level
Root Administrator
Hi all,
I have an issue backing up to S3 - I've read the threads about bucket names not containing dots and V4 bucket support - but I'm still seeing the following in the transport log:

[2018-12-14 11:49:18 +0000] info [cpbackup_transporter] Validating destination path /Backups/2018-12-14/system/
[2018-12-14 11:49:21 +0000] info [cpbackup_transporter] Making Path /Backups/2018-12-14/system/
[2018-12-14 11:49:23 +0000] warn [cpbackup_transporter] Path creation failed: PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x181eeb0), "Path creation failed: PermanentRedirect: The bucket you are "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 838
Cpanel::Backup::Queue::transport_backup::_validate_path(Cpanel::Backup::Queue::transport_backup=HASH(0x1676190), Cpanel::Transport::Files::AmazonS3=HASH(0x26be7f0), "/Backups/2018-12-14/system/system_files.tar", Cpanel::LoggerAdapter=HASH(0x181eeb0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 788
Cpanel::Backup::Queue::transport_backup::validate_path(Cpanel::Backup::Queue::transport_backup=HASH(0x1676190), Cpanel::Transport::Files::AmazonS3=HASH(0x26be7f0), "/Backups/2018-12-14/system/system_files.tar", Cpanel::LoggerAdapter=HASH(0x181eeb0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 255
Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1676190), cPanel::TaskQueue::Task=HASH(0x26b4900), Cpanel::LoggerAdapter=HASH(0x181eeb0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624
eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 233
eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 233
cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x26a4910), CODE(0x26a42c8)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x26a47c0)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165


I've tried buckets set up in eu-west-1 (Ireland) and eu-west-2 (London) and neither validate or succeed when trying to back up to them. I can happily connect to the bucket via the AWS CLI, upload files etc. however I do have to set the region. Is there some way to tell the backup transport to use a specific region? Or have I got something wrong somewhere?

Thanks!
 

KristanM

Registered
Dec 14, 2018
4
0
1
UK
cPanel Access Level
Root Administrator
Interesting - I left mine enabled, even though it wouldn't validate. I've just checked and it looks like the following day it started working and I now have 5 days of backups and a weekly correctly in my S3 bucket.

Has something changed I wonder, or do we need to wait some time between creating the bucket and using it for it to replicate somewhere.

Useful to know though, as I'm setting this up on several boxes now!
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello @benito and @KristanM,

Despite backups generating successfully, I'd like to rule out any issues with the Amazon S3 backup destination validation process. Can you verify which cPanel & WHM version is installed on the affected system? Also, are you able to reproduce the validation issue on new S3 backup destinations (if you have the ability to test it)?

Thank you.
 

benito

Well-Known Member
Jan 8, 2004
396
13
168
Mar del Plata - Argentina
cPanel Access Level
Root Administrator
Twitter
Hello @benito and @KristanM,

Despite backups generating successfully, I'd like to rule out any issues with the Amazon S3 backup destination validation process. Can you verify which cPanel & WHM version is installed on the affected system? Also, are you able to reproduce the validation issue on new S3 backup destinations (if you have the ability to test it)?

Thank you.
Hello!

I created a new bucket in London, it gives:

"Error: Validation for transport “bucketname” failed: Timeout"

cPanel version v76.0.14. Will check it again tomorrow to see if it validates ok like the previous one.
 
Last edited:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
I created a new bucket in London, it gives:

"Error: Validation for transport “bucketname” failed: Timeout"
Can you open a support ticket so we can take a closer look at the affected system and reproduce this issue? Post the ticket number here and I'll link this thread to it.

Thank you.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello,

To update, internal case CPANEL-24550 CPANEL-26887 is now open to track reports of Amazon S3 backup destinations failing validation. In these reports, validation and backups subsequently succeed within 24 hours after the initial failed validation attempt. I'll monitor this case and update this thread with more information as it becomes available.

Thank you.
 
Last edited:

KristanM

Registered
Dec 14, 2018
4
0
1
UK
cPanel Access Level
Root Administrator
Hello @benito and @KristanM,

Despite backups generating successfully, I'd like to rule out any issues with the Amazon S3 backup destination validation process. Can you verify which cPanel & WHM version is installed on the affected system? Also, are you able to reproduce the validation issue on new S3 backup destinations (if you have the ability to test it)?

Thank you.
v76.0.14 here too.

I've tried two new buckets, one in the US, one in the EU. Both fail to validate for me interestingly. The existing one still validates fine. Is there a log file for the validation process we can look at?
 
Last edited by a moderator:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello @KristanM,

I've updated this thread to reflect that this issue extends to buckets within North America, and relayed this information internally. I'll continue to monitor the internal case and update this thread with more information as it becomes available.

Is there a log file for the validation process we can look at?
There's no specific log file, but you can review the output from the following WHM API 1 function when testing validation:

WHM API 1 Functions - backup_destination_validate - Developer Documentation - cPanel Documentation

EX:

Code:
whmapi1 backup_destination_validate id=DESTINATION-ID disableonfail=0
To obtain the destination ID, you'd use the following WHM API 1 command:

Code:
whmapi1 backup_destination_list
Thank you.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello Everyone,

Can anyone confirm if this issue persists in cPanel & WHM version 78.0.20 (available in the CURRENT release tier) or higher?

Thank you.
 

drakeba

Registered
Jul 2, 2019
2
0
1
Chesapeake, VA
cPanel Access Level
Root Administrator
@cPanelMichael I know this is a few months old, but I can confirm having the same issue just now on WHM version 80.0.18. Created an s3 bucket in the Oregon region, validation timed out consistently. As a test, I validated against an existing s3 bucket from the N. Virginia region just fine.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,218
463
Hello @drakeba,

Thanks for sharing the additional information. We're now tracking reports of this behavior as part of internal case CPANEL-26887. I've linked this thread to the case and will provide an update as soon as new information is available.

Thank you.
 
  • Like
Reactions: drakeba

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,295
1,273
313
Houston
Hello,

I've just checked in on this case and it would appear that the development team for this has decided this isn't something they'll be able to address. The reasoning being as follows:

Putting enough information in the validation message would be incredibly messy. I will look into possibly giving a message in the transport setup. Unfortunately, there's no way for us to capture that it's a newly-created bucket, so we would have to give this message on every validation that fails on S3 for a bucket not found message.
If you feel that this is something that will be important to be included or if you have a different idea for how to mitigate it, I'd suggest opening a feature request for it. You can do so using the link in my signature. If you do open a feature request for this, please add the link here when it's accepted so others can view and vote on it.


Thanks!
 

pacwebhosting

Member
PartnerNOC
Jul 4, 2007
14
0
51
UK
cPanel Access Level
DataCenter Provider
Hi,

I have just encountered this exact same issue today.

After setting up an s3 bucket cpanel backup would not validate it but jetbackup would on the same server straight away.

I imagine that where cpanel is trying to validate it that it does indeed have to replicate to there as it worked hours later.

Mine was a UK bucket.

So maybe speak to the guys/partners at jetbackup and find out how they work out how to get it to validate straight away as they must have a method working out the location of the bucket and querying there :)

Thanks

Paul
 

hoomanxx

Registered
Dec 31, 2019
3
0
1
Turkey
cPanel Access Level
Reseller Owner
Hi,

I have same problem, When I want create s3 bucket, validation timed out consistently. my s3 bucket provider examined the problem and said: "cPanel's standard is not compatible with our services, you must set the region to default and cPanel have not this option".

How can I set the region to "default"?

Thanks