The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Amazon S3 Transport error

Discussion in 'Data Protection' started by AM2015, May 7, 2017.

Tags:
  1. AM2015

    AM2015 Member

    Joined:
    Jan 1, 2015
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I have backups configured to transfer daily to an Amazon S3 bucket on multiple accounts. This has worked fine in the past, but recently I have been receiving an error message on one very large account, though other accounts are transferring without incident. I thought the problem could be the size of the account, as Amazon has a 5 GB limit for transfers, so I did some maintenance to reduce size for transfer (such as deleting old log files) -- and reduced the size of the backup to 3.3 GB -- but the error persists.

    My cpbackup_transporter.log shows Amazon reporting a 400 error on upload, as follows:

    Code:
    [2017-05-07 02:13:23 -0700] info [cpbackup_transporter] Uploading /backup/2017-05-07/accounts/account1.tar.gz to /2017-05-07/account1.tar.gz
    [2017-05-07 02:13:23 -0700] info [cpbackup_transporter] Attempting to upload /backup/2017-05-07/accounts/account1.tar.gz to /2017-05- 07/account1.tar.gz for destination:  Amazon S3
    [2017-05-07 02:13:23 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2017-05-07/accounts/account1.tar.gz to  /2017-05-07/account1.tar.gz for destination:  Amazon S3
    [2017-05-07 02:16:22 -0700] warn [cpbackup_transporter] Upload attempt failed: Amazon::S3: Amazon responded with 400 Bad Request
     at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 134.
     at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
            Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1938ed8), "Upload attempt failed: Amazon::S3: Amazon responded with 400  "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 494
            Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1f1fa78), Cpanel:: Transport::Files::AmazonS3=HASH(0x24a5f08), "/backup/2017-05-07/accounts/account1.tar.gz", "/2017-05-07/account1.tar.gz", Cpanel: :LoggerAdapter=HASH(0x1938ed8)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 256
            Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1f1fa78), cPanel::TaskQueue: :Task=HASH(0x24134c0), Cpanel::LoggerAdapter=HASH(0x1938ed8)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel /TaskQueue.pm line 586
            eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 589
            cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 238
            eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 238
            cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x23e1d90), CODE(0x23e1ca0)) called at /usr/local/cpanel/3r dparty/perl/524/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 594
            cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x23e1dd8)) called at /usr/local/cpanel/bin/cpbackup_transporter line  151
            eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 149
    [2017-05-07 02:16:22 -0700] info [cpbackup_transporter] Error encountered
    [2017-05-07 02:16:22 -0700] info [cpbackup_transporter] Error is not a reference Amazon::S3: Amazon responded with 400 Bad Request
     at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 134.
    Each night 3 attempts are made for the upload for this account -- each time it fails, and then the other accounts on the server are backed up and uploaded to the S3 Bucket without a problem. I have increased timeout settings but that does not appear to be the issue.

    At this point I am at a loss as to what I can do to resolve this issue. ​
     
  2. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,407
    Likes Received:
    53
    Trophy Points:
    28
    Location:
    India
    cPanel Access Level:
    Root Administrator
  3. AM2015

    AM2015 Member

    Joined:
    Jan 1, 2015
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I've done that and it doesn't have helpful information. I wouldn't be posting here if I hadn't first exhausted other resources.
     
  4. rpvw

    rpvw Well-Known Member

    Joined:
    Jul 18, 2013
    Messages:
    260
    Likes Received:
    76
    Trophy Points:
    28
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    Hi AM2015,

    I don't use S3, but I was intrigued by your issue, and thought I would have a look for myself.
    As you rightly pointed out - the S3 docs unhelpfully report that the string "400 Bad Request" can refer to 51 different issues, without really giving you a clue as to what to look at (thus demonstrating how gratuitous one-liner answers can often have little or no value !)

    The only thing I picked up on was from the cPanel documentation that warns that the backup folder (if defined) must exist in your S3 bucket prior to attempting the upload.

    It appears that you are asking the backup system to copy a file from:
    /backup/2017-05-07/accounts/account1.tar.gz
    to
    /2017-05- 07/account1.tar.gz

    Is it possible this folder does not exist, or has some typo in the path ?

    If you look at the first line of the log extract you posted, the file path in the bucket is /2017-05-07/ but in the second line, the file path appears as /2017-05- 07/ - see the extra space between the 05- and the 07 ? Now this could, of course, have happened at copy/paste time when you created this post - but it may be worth a second look :)

    Sorry I can't be more specific, and my apologies in advance if you have already revised all your configuration. I can only suggest to check and re-check you have got all your settings right at the WHM backup end, AND that they coincide with what you have at the S3 end.

    Hope this is of some help.


    **edit**
    I just stumbled across a similar thread at WHM backup results in "Amazon responded with 400" that it may be worth keeping an eye on :)
     
    #4 rpvw, May 8, 2017
    Last edited: May 8, 2017
  5. Jon Hill

    Jon Hill Registered

    Joined:
    May 7, 2017
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Fort Worth, Texas
    cPanel Access Level:
    Root Administrator
    I successfully configured WHM to backup to S3 as an "Additional Destination." It worked for two daily backups, then failed for two files in the third backup. The error in WHM's log just says, "Amazon responded with 400." Amazon's page lists 51 possible causes for the 400 error code. Is there a way to find out more specifically what went wrong? I am posting this same question over on Amazon as well.

    Excerpt from the transporter log:
    Code:
    [2017-05-07 14:40:03 -0400] warn [cpbackup_transporter] Upload attempt failed: Amazon::S3: 400
     at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 134.
     at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x2512bf0), "Upload attempt failed: Amazon::S3: Amazon responded with 400 "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 494
    What I am running on:
    Code:
    /etc/redhat-release:CentOS Linux release 7.3.1611 (Core)
    /usr/local/cpanel/version:11.62.0.22
    /var/cpanel/envtype:virtuozzo
    CPANEL=stable
    Server version: Apache/2.4.25 (cPanel)
    Server built:   Apr  7 2017 15:35:17
    ea-php-cli Copyright 2016 cPanel, Inc.
    PHP 5.6.30 (cli) (built: Apr 29 2017 17:39:17)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    mysql  Ver 14.14 Distrib 5.6.35, for Linux (x86_64) using  EditLine wrapper
    
     
  6. Jon Hill

    Jon Hill Registered

    Joined:
    May 7, 2017
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Fort Worth, Texas
    cPanel Access Level:
    Root Administrator
    The account file that failed was 2.14 GB. I don't have the tar file for the system, which also failed. Today's backup seems to have worked. The account file that failed before is again 2.14 GB on the VPS. system.tar is 340.2 MB on S3.
     
  7. AM2015

    AM2015 Member

    Joined:
    Jan 1, 2015
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Thank you, but I think the error you noted is just a copy-paste error from my logs. The appropriate directory was created on S3 and all other accounts transferred without a problem.

    I'd note that I have had this system set up for about 2 years with no problems - it only seems to have started since a recent CPanel upgrade that made some changes to the backup system. I'm currently running v. 64 (build 19). The error started on April 28, so I'm thinking it could be related to this change reported with 64.0.17:
    You are right that the other thread you noted is reporting the same issue.
     
  8. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    37,086
    Likes Received:
    1,288
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello Everyone,

    I've merged together a couple of threads related to this topic.

    Internal case CPANEL-13033 is open to track reports of backups to remote Amazon S3 servers intermittently failing with error messages such as this:

    Code:
    info [cpbackup_transporter] Error is not a reference Amazon::S3: Amazon responded with 400 Bad Request
    We don't yet have a workaround, however I'll update this thread with more information on the status of this case as it becomes available.

    Thank you.
     
    eva2000 likes this.
  9. AM2015

    AM2015 Member

    Joined:
    Jan 1, 2015
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Thank you so much. In my case the one account is failing every day - it is also a large account (3.3 gigs) -- whereas several other smaller accounts and system backups are transferring without difficulty. The other accounts are between 60-400 MB. The system_files.tar is also transferring most times (had an error one day only), and is just under 1 GB.

    Of course I am happy to provide whatever information is needed to help debug in the future.
     
  10. AM2015

    AM2015 Member

    Joined:
    Jan 1, 2015
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I don't know why, but the transfer to Amazon S3 has now been working every day for 3 days. The file that was failing before is 3.1 gigs in the Amazon bucket. My CPanel has now upgraded to v. 64 (build 20) but I can see that the transfers were still failing 1-2 days after upgrade, so I don't think the upgrade is the reason. Maybe the problem was on Amazon's end? (anyway, I'll monitor and post again if I see problems)
     
  11. jhoongo

    jhoongo Registered

    Joined:
    May 23, 2017
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Vancouver, BC, Canada
    cPanel Access Level:
    Root Administrator
    I am experiencing the exact same issue where Amazon S3 backup transfer intermittently fail.
    Looks like some of large accounts are failing, but not all of large accounts failed.

    I am currently on cPanel & WHM 64.0 (build 22).
    Due to this issue, I consistently run out of disk usage for my entire VPS which results in malfunction of all of my clients' website..:(

    Please update us on the issue when it is resolved.

    Thanks,
     
  12. AM2015

    AM2015 Member

    Joined:
    Jan 1, 2015
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I still am experiencing the intermittent failures with larger accounts but I have felt it somewhat helpful to edit the etc/cpbackup-exclude.conf to reduce size of backups-- in my case simply by eliminating backup of the user tmp directories, where old log files are stored. I also have eliminating backups of stored backup files -- for example, within Wordpress installations or backups made and stored by the Softaculous add on. Doing that reduced the storage size of the largest account from 3 gigs down to 1 gig.

    Of course it would be better if I didn't have to do this at all -- but pending resolution of the primary issue, it does at least improve both the problem of AWS transport and impact of local storage of backups.
     
  13. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    37,086
    Likes Received:
    1,288
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    The case is still open to track reports of this issue, however thus far it seems related to performance and network issues on the Amazon S3 network. I recommend reaching out to Amazon if you notice any additional failures.

    Thank you.
     
  14. AM2015

    AM2015 Member

    Joined:
    Jan 1, 2015
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Thank you.

    I'm not surprised -because it has been inconsistent and only on very large files, I suspected the problem might be connectivity to the Amazon bucket.

    I found some information on the Amazon site that might be helpful:

    Amazon S3 Multipart Uploads — AWS SDK for PHP documentation

    This describes multi-part uploads - and although it says that files up to 5 gigs can be uploaded, it also says, "Amazon S3 customers are encouraged to use multipart uploads for objects greater than 100 MB."

    I don't know whether the current CPanel backup script uses this function or not - but if not, then that could be related to the problem.

    My problems have only occurred with very large files. I was able to resolve the problem with the large account by excluding nonessential items (like old log files) from the backup, but I have had difficulty with the system backup also failing to transport frequently.

    One feature request would be to provide more flexibility in the system backup as to how it is transported. I have reduced the size of the system backup for now by disabling the full mysql backup (preserving only account-based backup) -- but that wouldn't be an issue of the full database backup could be stored and transported separately.

    It would also be nice if the system backup and individual account backups could be placed on different schedules. Anyway, this is just a wish list for now.
     
  15. RWH Tech

    RWH Tech Well-Known Member

    Joined:
    Oct 1, 2015
    Messages:
    84
    Likes Received:
    16
    Trophy Points:
    8
    Location:
    Brazil
    cPanel Access Level:
    Root Administrator
    +1 on this and consistently on large files, but at random with some smaller ones.
     
  16. ljj3

    ljj3 Member

    Joined:
    Nov 7, 2014
    Messages:
    17
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Reseller Owner
    On two VPS's with several dozen accounts each, this began last week and has continued on ONE account (1.4GB). Reviewing the underlying logs, other accounts fail 1-2 times - on the third failure WHM gives up.

    Looking back, there were NO errors prior to last week. I would note that transfers from NYC to S3 (Northern VA) all work fine, while transfers from Salt Lake City have problems.

    Sounds like an S3 issue - but it also sounds like the multipart transfer would help. What are the plans - if any - to address (on cPanel end)?

    Thanks,

    - Lou
     
    #16 ljj3, Jul 1, 2017
    Last edited: Jul 1, 2017
  17. ljj3

    ljj3 Member

    Joined:
    Nov 7, 2014
    Messages:
    17
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Reseller Owner
    One more datapoint:

    Salt Lake City --> VA Fails
    Salt Lake City --> N. California Success

    NYC --> VA Success
     
  18. RWH Tech

    RWH Tech Well-Known Member

    Joined:
    Oct 1, 2015
    Messages:
    84
    Likes Received:
    16
    Trophy Points:
    8
    Location:
    Brazil
    cPanel Access Level:
    Root Administrator
    Mine quit erroring out on 6/9, with no changes done on this end, so I figured a WHM update took care of it...
     
  19. ljj3

    ljj3 Member

    Joined:
    Nov 7, 2014
    Messages:
    17
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Reseller Owner
    Any update?

    I moved my S3 backups to a different S3 center. Problem stopped for a week and now restarted!

    Also smaller accounts are failing too.
     
  20. ljj3

    ljj3 Member

    Joined:
    Nov 7, 2014
    Messages:
    17
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Reseller Owner
    I should add we are on 64.0.33

    BTW, on the Amazon S3 forums this has been raised with no help or solutions offered...
     
Loading...

Share This Page