S3 Backups not being deleted

[platinum]

There's a few threads I've seen, but as far as I could work out - the current version of cPanel with S3 backups is meant to behave the same was as regular backups, i.e. only retain the latest set of weekly/monthly/daily.

My daily backups are just building up each day, causing a ton of space to be used. I don't really want to use the auto-expire feature of files, because it would be better to explicitly have them deleted by the backup system.

grep BACKUP_DAILY_RETENTION /var/cpanel/backups/config = 1

Also checked that my Amazon bucket policy allows for deletion of files, it's happening the same on 2 servers I'm testing with.

Edit: should also add, on WHM 11.50.0 (build 9)

 

[cPanelMichael]

Hello,

There's a thread on this at:

https://forums.cpanel.net/threads/amazon-s3-backups-not-deleting.435631/#post-1880071

As mentioned, it's currently not possible to configure retention for Amazon S3, but there's an open feature request for that functionality.

Thank you.

 

[platinum]

Ah I see - I thought that might have been implemented as the last reply on that feature request says it does work, and the most recent thread about it here was asking about how to stop it deleting files.

I also get these messages in the transporter log, which say it's attempting to prune?

[2015-06-05 08:04:16 +1000] info [cpbackup_transporter] Starting a "prune" operation on the "Server-Name" destination ID XXXXX

 

[cPanelMichael]

[2015-06-05 08:04:16 +1000] info [cpbackup_transporter] Starting a "prune" operation on the "Server-Name" destination ID XXXXX

Hello,

Yes, this message indicates it's pruning the expired backups on the remote destination.

Thank you.

 

[platinum]

Well this is confusing haha.

So it does prune expired backups on a remote destination (in this case S3) ?
What qualifies an "expired" backup - as at the moment it seems like it's just storing everything.

 

[cPanelMichael]

Yes, for instance, daily backups are replaced with new daily backups. There is an internal case open (#111053) to address an issue where Amazon backups are not being pruned from Amazon S3 destination servers, however you should see errors in /usr/local/cpanel/logs/cpbackup_transporter.log when this happens.

Thank you.

 

[LostNerd]

There is also another internal case open (#187065) which has recently been discovered to be relevant to the S3 platform, not just FTP:

187065 - Pruning remote backups can fail when FTP directory listing format is different than expected

The definition I've received from P. Elsner is:

When pruning old backups, we list the directory contents and we look for a specific directory structure..

$string !~ /^[rwxsStT-]{9}$/

That means the files have to begin with a permissions string that starts with rwxsSrT-

For cPanel Staff:
Support Ticket ID: 6578307 for more info on what I have quoted.

 

[platinum]

Yes, for instance, daily backups are replaced with new daily backups. There is an internal case open (#111053) to address an issue where Amazon backups are not being pruned from Amazon S3 destination servers, however you should see errors in /usr/local/cpanel/logs/cpbackup_transporter.log when this happens.

Thank you.

Here's my log:

[2015-06-11 12:08:11 +1000] info [cpbackup_transporter] Starting a "prune" operation on the "XXXXXX" destination ID "XXXXXX".
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cpbackup_transporter - Processing next task
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do
[2015-06-11 12:08:12 +1000] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped.

Nothing has been removed, I've got a backlog of about 10 daily backups built up again.

Is there anything else I can check or is this a bug?

 

[cPanelMichael]

Feel free to open a support ticket using the link in my signature so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome.

Thank you.