Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Backup to Amazon S3 Doesn't Obey Retention Rules

Discussion in 'Data Protection' started by HowardE, Jul 31, 2017.

  1. HowardE

    HowardE Member

    Joined:
    Aug 8, 2015
    Messages:
    15
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    Florida
    cPanel Access Level:
    Root Administrator
    It seems that since the update to cPanel/WHM a few weeks ago [possibly since build 30 of ver 64] (current version I have is cPanel & WHM 64.0 (build 36)) that the backup to S3 has stopped obeying the retention rules like maybe the s3 sync command isn't quite right?

    The local folder that I'm using /backups retains the correct number of backups, but the S3 destination does not.

    I have "Strictly enforce retention, regardless of backup success." checked, and I have checked the backup log, and see no notices or errors in that.

    Thanks.
     
    #1 HowardE, Jul 31, 2017
    Last edited by a moderator: Jul 31, 2017
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,425
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    Could you review the transport logs in the /usr/local/cpanel/logs/cpbackup_transporter/ directory and verify if you see any "prune" activity after the backups are copied to the remote destination? Also, could you verify if you are seeing multiple copies of the same backup archive on the Amazon S3 server?

    Thank you.
     
  3. geotex

    geotex Member

    Joined:
    Feb 14, 2013
    Messages:
    7
    Likes Received:
    2
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Last log entry after successful upload of last scheduled domain:

    [2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
    [2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do
    [2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped.

    No activity or term "prune" found, no mention of deletion sig. to amazon to delete past directory.

    Checked Amazon S3 per request, the backup files are stored in dated folders, folders past the requested four are not duplicate dates and are not being removed.

    I can send the cPanel log if needed, as it is not that large since we do not back up all accounts, only active customer domains.
     
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,425
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello @geotex,

    Internal case CPANEL-14906 is open to address an issue where backup pruning does not occur on Amazon S3 destinations when no folder is specified in the destination's configuration. I'll monitor this case and update this thread with more information as it becomes available.

    As a temporary workaround, you can create a subdirectory in your Amazon S3 instance, and then edit the "Folder" option for the Amazon S3 destination from within "WHM >> Backup Configuration" so that it uses that specific subdirectory. This should allow pruning to occur. Additionally, you may want to manually remove any existing backups that were not pruned from your Amazon S3 destination.

    Thank you.
     
    Kent Brockman likes this.
  5. HowardE

    HowardE Member

    Joined:
    Aug 8, 2015
    Messages:
    15
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    Florida
    cPanel Access Level:
    Root Administrator
    Hi Michael.

    Thanks for your reply. I thought I was subscribed, but didn't get an email?

    Anyway... with today's backup, it worked properly, and there is a "Prune" operation in the log.

    From a previous log, there was a prune, but it didn't actually do it. There were other copy operations AFTER the prune


    Code:
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Instantiating Object
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Starting a "prune" operation on the "xxx Bucket" destination ID "----------------------------".
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Performing prune operation, retaining 4 items on:  xxx Bucket
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do
    [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped.
    [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Initializing log file
    [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon started.
    [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - started
    [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
    [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
    [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Instantiating Object
    [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Starting a "copy" operation on the "xxx Bucket" destination ID "xxxxxxxxxxxxxxxxxxxxxxxx".
    
    
    However.... today's backup did, and the 'prune' operation was the last item in the log before "Exiting"
     
  6. geotex

    geotex Member

    Joined:
    Feb 14, 2013
    Messages:
    7
    Likes Received:
    2
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Thank you. Set up as recommended, we will see how that goes. I only do full server backups twice a week, so will take a couple of weeks to see if that fixes the issue.

    Looking forward to seeing your updates post.

    George
     
    cPanelMichael likes this.
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,425
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hi @HowardE,

    Do you have anything entered for the "Folder" option in the backup configuration for your Amazon S3 destination in "WHM >> Backup Configuration"?

    Thank you.
     
  8. HowardE

    HowardE Member

    Joined:
    Aug 8, 2015
    Messages:
    15
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    Florida
    cPanel Access Level:
    Root Administrator
    Hi Michael.

    I do not have a folder specified. The backup is to the root of this particular bucket. The backup transport appears to work fine, just does not prune.

    upload_2017-8-4_6-21-21.png

    If making a folder underneath the bucket will make a difference, I can try that?


    Code:
    [2017-08-04 05:58:15 -0400] info [cpbackup_transporter] Starting a "prune" operation on the "Howard's Bucket" destination ID "xxxxxxxxxxxxxxxxxxxxxxxxxxxx".
    [2017-08-04 05:58:15 -0400] info [cpbackup_transporter] Performing prune operation, retaining 4 items on:  Howard's Bucket
    
    It says it started to prune. I've enabled logging on that S3 bucket and will report what it says after the next backup.

    In the meantime, I'm creating a folder called 'backups' and moving the 4 existing backups there, and i'll let the backup run tonight.
     
    #8 HowardE, Aug 4, 2017
    Last edited: Aug 4, 2017
  9. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,425
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Yes, create the folder and then edit the destination in "WHM >> Backup Configuration" to define that folder name. This will act as a temporary workaround to the issue described on my earlier post.

    Thanks!
     
  10. HowardE

    HowardE Member

    Joined:
    Aug 8, 2015
    Messages:
    15
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    Florida
    cPanel Access Level:
    Root Administrator
    It doesn't say if there should be a slash or not.

    upload_2017-8-4_16-3-52.png
     
  11. HowardE

    HowardE Member

    Joined:
    Aug 8, 2015
    Messages:
    15
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    Florida
    cPanel Access Level:
    Root Administrator
    So far... adding the folder seems to have worked. I'll confirm after a few more backup cycles.

    Thanks.
     
  12. HowardE

    HowardE Member

    Joined:
    Aug 8, 2015
    Messages:
    15
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    Florida
    cPanel Access Level:
    Root Administrator
    After a few backups, I can confirm that this is working.
     
    cPanelMichael likes this.
  13. geotex

    geotex Member

    Joined:
    Feb 14, 2013
    Messages:
    7
    Likes Received:
    2
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Enough time has passed to exceed the backup retention, it seems the changes suggested earlier are working, the folder on amazon s3 is deleting now as instructed.
     
    cPanelMichael likes this.
  14. Kent Brockman

    Kent Brockman Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    1,160
    Likes Received:
    5
    Trophy Points:
    68
    Location:
    Buenos Aires, Argentina
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello guys, I'm having the same issue. I need an ETA for when may be expected that Internal case CPANEL-14906 will be finally solved. I manage nearly 25 cPanel servers that make daily backups of hundreds of accounts on to different S3 buckets, on 12 different Amazon accounts, with different combinations of daily/weekly retention rules, all this combined with different S3 lifecycles rules moving old files to Amazon Glacier.. So :) you can bet it's not cool nor easy to reconfigure everything to start using a directory within every bucket, on every Amazon account.
    I need to know when will this be addressed. Soon, I hope. (More than 1 month have pased, and this bug is costing more money and work to all of us)
     
  15. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,425
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hi @Kent Brockman,

    I don't have a specific time frame to offer at this time, but I've noted this forums thread in the internal case and will update this thread with new information as soon as it's available.

    Thank you.
     
    Kent Brockman likes this.
  16. Kent Brockman

    Kent Brockman Well-Known Member

    Joined:
    Jan 20, 2008
    Messages:
    1,160
    Likes Received:
    5
    Trophy Points:
    68
    Location:
    Buenos Aires, Argentina
    cPanel Access Level:
    Root Administrator
    Twitter:
    ok, I hope you guys can fix this soon. Thanks!
     
  17. RWH Tech

    RWH Tech Well-Known Member

    Joined:
    Oct 1, 2015
    Messages:
    85
    Likes Received:
    16
    Trophy Points:
    8
    Location:
    Brazil
    cPanel Access Level:
    Root Administrator
    I think it's cute that Cpanel introduces a bug like this with a major upgrade, then we have wait until the next major upgrade to have it fixed.
     
  18. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,425
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    I removed the previous posts as they were in reference to separate cases. The particular issue discussed on this thread is part of internal case CPANEL-14906. We may publish a resolution to cPanel 66, but no specific time frame or decision on that is available at this time. I'll update this thread as soon as new information is available.

    Thank you.
     
    RWH Tech likes this.
  19. infiniteeye

    infiniteeye Registered

    Joined:
    Jul 13, 2006
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    151
    We are suffering from this same issue.

    I first noticed when our S3 bill shot up to several hundred $'s

    Having to delete old backups every few days manually in S3 until its resolved, very frustrating. Have tried the 'put them into a sub folder within the bucket' fix and that did not work either.

    Posting here to track any updates / alternative suggestions.
     
  20. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,425
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Did you also edit the "Folder" option for the Amazon S3 destination from within "WHM >> Backup Configuration" so that it uses the specific subdirectory path you created? If so, feel free to open a support ticket using the link in my signature so we can take a closer look, as it's possible the issue you are facing is unrelated to case CPANEL-14906.

    Thank you.
     
Loading...

Share This Page