S3 Backups not being deleted

platinum

Active Member
Nov 21, 2004
30
0
156
There's a few threads I've seen, but as far as I could work out - the current version of cPanel with S3 backups is meant to behave the same was as regular backups, i.e. only retain the latest set of weekly/monthly/daily.

My daily backups are just building up each day, causing a ton of space to be used. I don't really want to use the auto-expire feature of files, because it would be better to explicitly have them deleted by the backup system.

grep BACKUP_DAILY_RETENTION /var/cpanel/backups/config = 1

Also checked that my Amazon bucket policy allows for deletion of files, it's happening the same on 2 servers I'm testing with.

Edit: should also add, on WHM 11.50.0 (build 9)
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,267
463

platinum

Active Member
Nov 21, 2004
30
0
156
Ah I see - I thought that might have been implemented as the last reply on that feature request says it does work, and the most recent thread about it here was asking about how to stop it deleting files.

I also get these messages in the transporter log, which say it's attempting to prune?

Code:
[2015-06-05 08:04:16 +1000] info [cpbackup_transporter] Starting a "prune" operation on the "Server-Name" destination ID XXXXX
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,267
463
[2015-06-05 08:04:16 +1000] info [cpbackup_transporter] Starting a "prune" operation on the "Server-Name" destination ID XXXXX
Hello,

Yes, this message indicates it's pruning the expired backups on the remote destination.

Thank you.
 

platinum

Active Member
Nov 21, 2004
30
0
156
Well this is confusing haha.

So it does prune expired backups on a remote destination (in this case S3) ?
What qualifies an "expired" backup - as at the moment it seems like it's just storing everything.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,267
463
Yes, for instance, daily backups are replaced with new daily backups. There is an internal case open (#111053) to address an issue where Amazon backups are not being pruned from Amazon S3 destination servers, however you should see errors in /usr/local/cpanel/logs/cpbackup_transporter.log when this happens.

Thank you.
 

LostNerd

Well-Known Member
Mar 12, 2014
258
12
18
Hastings, East Sussex, UK
cPanel Access Level
Root Administrator
Twitter
There is also another internal case open (#187065) which has recently been discovered to be relevant to the S3 platform, not just FTP:
Code:
187065 - Pruning remote backups can fail when FTP directory listing format is different than expected
The definition I've received from P. Elsner is:
When pruning old backups, we list the directory contents and we look for a specific directory structure..

$string !~ /^[rwxsStT-]{9}$/

That means the files have to begin with a permissions string that starts with rwxsSrT-
For cPanel Staff:
Support Ticket ID: 6578307 for more info on what I have quoted.
 

platinum

Active Member
Nov 21, 2004
30
0
156
Yes, for instance, daily backups are replaced with new daily backups. There is an internal case open (#111053) to address an issue where Amazon backups are not being pruned from Amazon S3 destination servers, however you should see errors in /usr/local/cpanel/logs/cpbackup_transporter.log when this happens.

Thank you.
Here's my log:

Code:
[2015-06-11 12:08:11 +1000] info [cpbackup_transporter] Starting a "prune" operation on the "XXXXXX" destination ID "XXXXXX".
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped.
Nothing has been removed, I've got a backlog of about 10 daily backups built up again.

Is there anything else I can check or is this a bug?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,267
463
Feel free to open a support ticket using the link in my signature so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome.

Thank you.