Backup to Amazon S3 Doesn't Obey Retention Rules

HowardE

Member
Aug 8, 2015
20
2
53
Florida
cPanel Access Level
Root Administrator
It seems that since the update to cPanel/WHM a few weeks ago [possibly since build 30 of ver 64] (current version I have is cPanel & WHM 64.0 (build 36)) that the backup to S3 has stopped obeying the retention rules like maybe the s3 sync command isn't quite right?

The local folder that I'm using /backups retains the correct number of backups, but the S3 destination does not.

I have "Strictly enforce retention, regardless of backup success." checked, and I have checked the backup log, and see no notices or errors in that.

Thanks.
 
Last edited by a moderator:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Hello,

Could you review the transport logs in the /usr/local/cpanel/logs/cpbackup_transporter/ directory and verify if you see any "prune" activity after the backups are copied to the remote destination? Also, could you verify if you are seeing multiple copies of the same backup archive on the Amazon S3 server?

Thank you.
 

geotex

Member
Feb 14, 2013
7
2
3
cPanel Access Level
Root Administrator
Last log entry after successful upload of last scheduled domain:

[2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do
[2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped.

No activity or term "prune" found, no mention of deletion sig. to amazon to delete past directory.

Checked Amazon S3 per request, the backup files are stored in dated folders, folders past the requested four are not duplicate dates and are not being removed.

I can send the cPanel log if needed, as it is not that large since we do not back up all accounts, only active customer domains.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Hello @geotex,

Internal case CPANEL-14906 is open to address an issue where backup pruning does not occur on Amazon S3 destinations when no folder is specified in the destination's configuration. I'll monitor this case and update this thread with more information as it becomes available.

As a temporary workaround, you can create a subdirectory in your Amazon S3 instance, and then edit the "Folder" option for the Amazon S3 destination from within "WHM >> Backup Configuration" so that it uses that specific subdirectory. This should allow pruning to occur. Additionally, you may want to manually remove any existing backups that were not pruned from your Amazon S3 destination.

Thank you.
 
  • Like
Reactions: Kent Brockman

HowardE

Member
Aug 8, 2015
20
2
53
Florida
cPanel Access Level
Root Administrator
Hi Michael.

Thanks for your reply. I thought I was subscribed, but didn't get an email?

Anyway... with today's backup, it worked properly, and there is a "Prune" operation in the log.

From a previous log, there was a prune, but it didn't actually do it. There were other copy operations AFTER the prune


Code:
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Instantiating Object
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Starting a "prune" operation on the "xxx Bucket" destination ID "----------------------------".
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Performing prune operation, retaining 4 items on:  xxx Bucket
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do
[2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped.
[2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Initializing log file
[2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon started.
[2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - started
[2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Instantiating Object
[2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Starting a "copy" operation on the "xxx Bucket" destination ID "xxxxxxxxxxxxxxxxxxxxxxxx".
However.... today's backup did, and the 'prune' operation was the last item in the log before "Exiting"
 

geotex

Member
Feb 14, 2013
7
2
3
cPanel Access Level
Root Administrator
Thank you. Set up as recommended, we will see how that goes. I only do full server backups twice a week, so will take a couple of weeks to see if that fixes the issue.

Looking forward to seeing your updates post.

George
 
  • Like
Reactions: cPanelMichael

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Hi @HowardE,

Do you have anything entered for the "Folder" option in the backup configuration for your Amazon S3 destination in "WHM >> Backup Configuration"?

Thank you.
 

HowardE

Member
Aug 8, 2015
20
2
53
Florida
cPanel Access Level
Root Administrator
Hi Michael.

I do not have a folder specified. The backup is to the root of this particular bucket. The backup transport appears to work fine, just does not prune.

upload_2017-8-4_6-21-21.png

If making a folder underneath the bucket will make a difference, I can try that?


Code:
[2017-08-04 05:58:15 -0400] info [cpbackup_transporter] Starting a "prune" operation on the "Howard's Bucket" destination ID "xxxxxxxxxxxxxxxxxxxxxxxxxxxx".
[2017-08-04 05:58:15 -0400] info [cpbackup_transporter] Performing prune operation, retaining 4 items on:  Howard's Bucket
It says it started to prune. I've enabled logging on that S3 bucket and will report what it says after the next backup.

In the meantime, I'm creating a folder called 'backups' and moving the 4 existing backups there, and i'll let the backup run tonight.
 
Last edited:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
If making a folder underneath the bucket will make a difference, I can try that?
Yes, create the folder and then edit the destination in "WHM >> Backup Configuration" to define that folder name. This will act as a temporary workaround to the issue described on my earlier post.

Thanks!
 

Kent Brockman

Well-Known Member
PartnerNOC
Jan 20, 2008
1,268
61
178
Buenos Aires, Argentina
cPanel Access Level
Root Administrator
Hello guys, I'm having the same issue. I need an ETA for when may be expected that Internal case CPANEL-14906 will be finally solved. I manage nearly 25 cPanel servers that make daily backups of hundreds of accounts on to different S3 buckets, on 12 different Amazon accounts, with different combinations of daily/weekly retention rules, all this combined with different S3 lifecycles rules moving old files to Amazon Glacier.. So :) you can bet it's not cool nor easy to reconfigure everything to start using a directory within every bucket, on every Amazon account.
I need to know when will this be addressed. Soon, I hope. (More than 1 month have pased, and this bug is costing more money and work to all of us)
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Hi @Kent Brockman,

I don't have a specific time frame to offer at this time, but I've noted this forums thread in the internal case and will update this thread with new information as soon as it's available.

Thank you.
 
  • Like
Reactions: Kent Brockman

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
I think it's cute that Cpanel introduces a bug like this with a major upgrade, then we have wait until the next major upgrade to have it fixed.
Hello,

I removed the previous posts as they were in reference to separate cases. The particular issue discussed on this thread is part of internal case CPANEL-14906. We may publish a resolution to cPanel 66, but no specific time frame or decision on that is available at this time. I'll update this thread as soon as new information is available.

Thank you.
 
  • Like
Reactions: RWH Tech

infiniteeye

Registered
Jul 13, 2006
1
0
151
We are suffering from this same issue.

I first noticed when our S3 bill shot up to several hundred $'s

Having to delete old backups every few days manually in S3 until its resolved, very frustrating. Have tried the 'put them into a sub folder within the bucket' fix and that did not work either.

Posting here to track any updates / alternative suggestions.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Having to delete old backups every few days manually in S3 until its resolved, very frustrating. Have tried the 'put them into a sub folder within the bucket' fix and that did not work either.
Did you also edit the "Folder" option for the Amazon S3 destination from within "WHM >> Backup Configuration" so that it uses the specific subdirectory path you created? If so, feel free to open a support ticket using the link in my signature so we can take a closer look, as it's possible the issue you are facing is unrelated to case CPANEL-14906.

Thank you.