Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Pending Publication [CPANEL-20584] Amazon S3 backup transport failure

Discussion in 'Data Protection' started by David Colter, Jul 26, 2018.

Tags:
  1. David Colter

    David Colter Member

    Joined:
    Jun 30, 2016
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    USA
    cPanel Access Level:
    Root Administrator
    Hello,

    Below is an excerpt of an email that's been received 3 days in a row now.

    Code:
    Preview of transport errors log:
    Unable to send “/backup/2018-07-26/backup_incomplete” to destination “WHM Backups”
    
    A note to include with this "suddenly occurring" message is that one week ago, I reconfigured WHM Backup Configuration to 'stop' keeping a copy of the backup on my server. It appears that system files and all accounts are being backed up, though. The S3 backups previous to a week ago did have a backup_incomplete item (?? folder or file) along with accounts and system in that dated backup folder.
     
    #1 David Colter, Jul 26, 2018
    Last edited by a moderator: Jul 26, 2018
  2. David Colter

    David Colter Member

    Joined:
    Jun 30, 2016
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    USA
    cPanel Access Level:
    Root Administrator
    Below is a tail output excerpt. There were 3 unsuccessful attempts to upload /2018-07-26/backup_incomplete, for reason: 'attempted to validate a file that does not exist'. This occurs after successfully uploading the other user account files and one system file.

    From log file /usr/local/cpanel/logs/cpbackup_transporter.log:
    Code:
    .......
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Uploading backup metadata /backup/2018-07-26/accounts/.master.meta to /2018-07-26/accounts/.master.meta (from 2018-07-26/accounts/.master.meta)
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Attempting to upload /backup/2018-07-26/accounts/.master.meta to /2018-07-26/accounts/.master.meta for destination:  WHM Backups
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2018-07-26/accounts/.master.meta to /2018-07-26/accounts/.master.meta for destination:  WHM Backups
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Successful transfer of /backup/2018-07-26/accounts/.master.meta to /2018-07-26/accounts/.master.meta for destination WHM Backups
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] The backup has been successfully uploaded at least once, now we will delete the local copy since keeplocal (0) is disabled.
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] There are no more files or directories in the backup path and keeplocal (0) is disabled, so now we will delete /backup/2018-07-26 entirely.
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Instantiating Object
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Starting a "copy" operation on the "WHM Backups" destination ID "c1lOWveFr53CVCO9kzutQA0g".
    
    [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Validating destination path /2018-07-26/
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Path exists
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Uploading account backup /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete (from 2018-07-26/backup_incomplete)
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Attempting to upload /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete for destination:  WHM Backups
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete for destination:  WHM Backups
    
    [2018-07-26 02:09:32 -0400] warn [cpbackup_transporter] Upload attempt failed: “Cpanel::Transport::Files::put” attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
    
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x23188c0), "Upload attempt failed: \x{e2}\x{80}\x{9c}Cpanel::Transport::Files::put\x{e2}\x{80}\x{9d} at"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609
    
    Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), Cpanel::Transport::Files::AmazonS3=HASH(0x2e4b938), "/backup/2018-07-26/backup_incomplete", "/2018-07-26/backup_incomplete", Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 301
    
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), cPanel::TaskQueue::Task=HASH(0x45062d8), Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624
    
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
    
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
    
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
    
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x3748778), CODE(0x2db4e28)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
    
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2d70810)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
    
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: “Cpanel::Transport::Files::put” attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete for destination:  WHM Backups
    
    [2018-07-26 02:09:32 -0400] warn [cpbackup_transporter] Upload attempt failed: “Cpanel::Transport::Files::put” attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
    
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x23188c0), "Upload attempt failed: \x{e2}\x{80}\x{9c}Cpanel::Transport::Files::put\x{e2}\x{80}\x{9d} at"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609
    
             Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), Cpanel::Transport::Files::AmazonS3=HASH(0x2e4b938), "/backup/2018-07-26/backup_incomplete", "/2018-07-26/backup_incomplete", Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 301
    
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), cPanel::TaskQueue::Task=HASH(0x45062d8), Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624
    
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
    
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
    
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
    
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x3748778), CODE(0x2db4e28)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
    
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2d70810)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
    
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: “Cpanel::Transport::Files::put” attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete for destination:  WHM Backups
    
    [2018-07-26 02:09:32 -0400] warn [cpbackup_transporter] Upload attempt failed: “Cpanel::Transport::Files::put” attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
    
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x23188c0), "Upload attempt failed: \x{e2}\x{80}\x{9c}Cpanel::Transport::Files::put\x{e2}\x{80}\x{9d} at"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609
    
        Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), Cpanel::Transport::Files::AmazonS3=HASH(0x2e4b938), "/backup/2018-07-26/backup_incomplete", "/2018-07-26/backup_incomplete", Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 301
    
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), cPanel::TaskQueue::Task=HASH(0x45062d8), Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624
    
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
    
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
    
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
    
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x3748778), CODE(0x2db4e28)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
    
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2d70810)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
    
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: “Cpanel::Transport::Files::put” attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
    
    [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] cpbackup_transporter - Waiting up to 299s for new tasks
    .......
     
  3. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,838
    Likes Received:
    276
    Trophy Points:
    193
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @David Colter


    Looking at the file name:
    /backup/2018-07-26/backup_incomplete

    It seems that there may have been an issue with the backup process, to begin with. Based on:

    Code:
    info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: “Cpanel::Transport::Files::put” attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete
    It seems that the file may not actually exist though it's in the backup metadata. Can you confirm that this is the case? You can run the following and reply with the output:

    Code:
    ls -lah /backup/2018-07-26/backup_incomplete
    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. David Colter

    David Colter Member

    Joined:
    Jun 30, 2016
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    USA
    cPanel Access Level:
    Root Administrator
    Remembering that I stopped backing up locally a few days ago, there is no folder /backup/2018-07-26. In fact, there was no 'dated' backup folder (locally) for each of the days an error notice was sent.
    Code:
    # ls -lah /backup/2018-07-26/backup_incomplete
    /bin/ls: cannot access /backup/2018-07-26/backup_incomplete: No such file or directory
    
    I re-instated local backups yesterday, and did not receive any warnings during the 2am backup run today.

    I would like to stop backing up locally and only have the S3 backups, without receiving a daily warning that something is wrong.

    Can you please explain why the metadata would be containing the file? Do the backups first get saved locally, then get transported and finally, the local is deleted?

    Or perhaps, suggest a solution?

    Thank you.
     
  5. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,838
    Likes Received:
    276
    Trophy Points:
    193
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    That makes a bit more sense as to why you received that error in that case.

    The only circumstance in which I would assume the metadata would contain data on the backup when it was later cancelled is if the backup process was killed or abruptly ended.

    This is exactly how this happens - the local backup is not deleted until the transport is successful. Any configuration of the automated backup system will have this same behavior as well.


    Is the error in regard to the incomplete backup only occurring when you're keeping backups locally or do you mean you stopped the transport?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. David Colter

    David Colter Member

    Joined:
    Jun 30, 2016
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    USA
    cPanel Access Level:
    Root Administrator
    I have not ever 'stopped' the transport.

    The error only occurs when a local copy is NOT being kept.
     
  7. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,838
    Likes Received:
    276
    Trophy Points:
    193
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @David Colter

    Thank you for clarifying. That is not the expected behavior, you should be able to transport to the remote server without keeping a local copy. I'm not sure what would cause that specific behavior in this case.

    Can you please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. David Colter

    David Colter Member

    Joined:
    Jun 30, 2016
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    USA
    cPanel Access Level:
    Root Administrator
    Your Support Request ID is: 9997463
     
  9. David Colter

    David Colter Member

    Joined:
    Jun 30, 2016
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    USA
    cPanel Access Level:
    Root Administrator
    That ticket had a response from S. Young, with
     
  10. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,838
    Likes Received:
    276
    Trophy Points:
    193
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @David Colter

    I have located the case as well and it looks as though it is specifically related to the backup_incomplete touch file you were seeing and Amazon S3. It will be fixed in v74 of cPanel/WHM then backported to 70 and 72. I'll update here as soon as I have confirmation that the issue is resolved


    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. David Colter

    David Colter Member

    Joined:
    Jun 30, 2016
    Messages:
    23
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    USA
    cPanel Access Level:
    Root Administrator
    In the case this may be relevant to the subject issue - every time /usr/local/cpanel/bin/backup runs, an email is sent alerting the following:

    Use of uninitialized value in pattern match (m//) at /usr/local/cpanel/Cpanel/Filesys.pm line 74.

    This is consistently reported, since the Backup Manager was set to not save a local copy of the backup and appears to occur very near the beginning of the script. Additionally, the server is also reporting, ~ 8 minutes later, "unable to send ... /backup_incomplete ... " to the S3 destination.
     
  12. Ishware

    Ishware Well-Known Member

    Joined:
    Nov 7, 2003
    Messages:
    194
    Likes Received:
    3
    Trophy Points:
    168
    Location:
    Panama City, FL
    cPanel Access Level:
    Root Administrator
    I'm also getting the message from the above every day:

    Use of uninitialized value in pattern match (m//) at /usr/local/cpanel/Cpanel/Filesys.pm line 74.

    Posting here so I can get any updated replies on that :)
     
  13. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,838
    Likes Received:
    276
    Trophy Points:
    193
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hello everyone,


    I would like to reiterate that this issue is fixed in a future version, we're just awaiting the release of the fix. If you're still experiencing issues with this once the build of v74 with the resolution included is released please let us know. We'll announce here as soon as the fix is published.

    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice