rsyc remote files not being removed

[Webdew]

I have incremental backups working to a remote server via rsync ( a QNAP in this case). I have daily x 7, weekly x 3 monthly x 3.

My understanding was that with incremental backups storage would be reduced with unchanged files not being duplicated. So a backup today would check if file A was previously backed up. If not a copy would be made in today's folder. If it has already been backed up AND had been changed since the last copy was taken an new backup copy would be taken. Otherwise if unchanged a symlink would be created to the previous copy.. thus saving space.

I had expected if I only had 5 days of backups in my backup settings I would only see a list of folders going back 5 days from today, and that effectively the oldest folder would contain all files from that date with the newer folders possibly having symlinks or changed folders. I had expected when a new backup was run the 'second oldest' folder would then be populated with updated files and the oldest folder would be removed along with its contents..

What I am seeing right now is a long list of daily and even weekly folders that are being left behind.

Right now I can see dailies going back to 04/05/2020 .. that is a lot of extra data potentially. When I look in this folder I still see /accounts/ and about 9 account folders within. As a look at more recent folders there are even more accounts being caught up... and even the /system/ folder.

I'm not sure if I can delete these older files .. or if potentially these are still legitimate 'files' with legitimate symlinks still referencing these to this day. I don't want to delete these myself and find I can't do a backup recovery for those accounts because files were removed.

So am I mis understanding what should be happening?
OR is there something wrong with my backups for this to be occurring.

I need to reclaim some space soon.

 

[cPanelLauren]

That doesn't sound right, just with the dailies alone. If you go look at at the backup logs do you see any errors with pruning? You can find them at /usr/local/cpanel/logs/cpbackup/

 

[Webdew]

That doesn't sound right, just with the dailies alone. If you go look at at the backup logs do you see any errors with pruning? You can find them at /usr/local/cpanel/logs/cpbackup/

Looks like there are 10 log files that roll through in that folder. The latest does have entry in it right towards the end of the file:

This log file is long - but not even close long enough to list all of the accounts that could be backed up. The entries in the log do not seem to be in alphabetical order so its difficult to tell how many were done or missed. Perhaps "nothing" changed in many of them?

[2020-08-18 02:32:22 +0800] info [backup] Successfully backed up account “someaccount” to “/home/backup/2020-08-18/accounts”
[2020-08-18 02:32:22 +0800] info [backup] Adding metadata information for someaccount to backup at /home/backup/2020-08-18
[2020-08-18 02:32:22 +0800] info [backup] Processing an incremental type backup for someaccount..
[2020-08-18 02:32:22 +0800] info [backup] Queuing daily backup copy of “someaccount” for transport of “/home/backup/2020-08-18/accounts/someaccount” to “2020-08-18/accounts/someaccount”
[2020-08-18 02:32:22 +0800] info [backup] This particular transport will be queued with keep_local = 0 , based on the need to copy weekly () and/or monthly () copies as well.
[2020-08-18 02:32:22 +0800] info [backup] Queuing transport of file: /home/backup/2020-08-18/accounts/someaccount
[2020-08-18 02:32:22 +0800] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:28734
[2020-08-18 02:32:22 +0800] info [backup] leaving queue_backup_transport_item
[2020-08-18 02:32:22 +0800] info [backup] Queuing transport of meta file: /home/backup/2020-08-18/accounts/.master.meta
[2020-08-18 02:32:22 +0800] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:28735
[2020-08-18 02:32:22 +0800] info [backup] leaving queue_backup_transport_item
[2020-08-18 02:32:22 +0800] info [backup] Queuing transport of file: /home/backup/2020-08-18/backup_incomplete
[2020-08-18 02:32:22 +0800] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:28736
[2020-08-18 02:32:22 +0800] info [backup] leaving queue_backup_transport_item
[2020-08-18 02:32:22 +0800] warn [backup] Pruning of backup files skipped due to errors. at /usr/local/cpanel/bin/backup line 395.
        bin::backup::run("bin::backup") called at /usr/local/cpanel/bin/backup line 120
[2020-08-18 02:32:22 +0800] info [backup] Queuing removal of staging directories since KEEPLOCAL is disabled.
[2020-08-18 02:32:22 +0800] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:28737
[2020-08-18 02:32:22 +0800] info [backup] leaving queue_backup_transport_item
[2020-08-18 02:32:22 +0800] info [backup] Queuing transport reporter
[2020-08-18 02:32:22 +0800] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:28738
[2020-08-18 02:32:22 +0800] info [backup] leaving queue_backup_transport_item
[2020-08-18 02:32:22 +0800] info [backup] Completed at Tue Aug 18 02:32:22 2020

I checked the last 4 and they all end the same way with the file reporting a prune error on different accounts at the point of failure and logging halts.

[2020-08-17 02:28:53 +0800] info [backup] Final state is Backup::PartialFailure (0)
[2020-08-17 02:28:53 +0800] info [backup] Sent Backup::PartialFailure notification.

I don't see these 'sent' notifications mentioned in the line above (and if I had I would have been looking at this a lot sooner) - but I do get plenty of other WHM emailed notices so I'm not sure where this is going to find out more.
It looks like this failure is always ending 30-40 min into the backup task running.

I didn't mention in the opening post above but I do have the backups set so that they are not kept locally and only remotely (removed after sync) and that appears to be happening fine (ie local storage is not getting choked up at all).

Thanks for the feed back - appreciate it.

 

[cPanelLauren]

This indicates that some of the backups aren't completing and in that instance, pruning won't happen by default (anytime there is a Partial Failure) recorded. The backup logs are created during backup time during every backup. You'd have to look through them to identify which account/s is experiencing issues.

 

[Webdew]

Is there a way for me to just 'start again'? If I just delete everything on the backup server will this be picked up and a full new backup be performed?
Looking at the log it seem to be different accounts each time that it halts on .. but I have no clue as to why. I have tripled checked and see no detailed email.
I guess the issue would just reoccur and if I don't know what it is it will halt with only a handful done at that time.

What are the potential issues that could cause this on either end?, given it seems to be a 30-40 minutes could it be a time out issue or memory allocation (I do not think it is disk).

I had thought I had a 'too much' back up data issue but now I suspect I just have a lot of fragments and quite possibly very few completely recoverable accounts.. which is a huge concern.