Backup retention does not remove old backups anymore?

[Benjamin D.]

Hi,

Recently, I've reached 100% of my home partition disk usage. I've checked what could have used that much hard disk space and quickly realized the WHM Backups do not remove the old backups anymore. It has always been set to BACKUP DAILY, RETENTION: 2 but this seems to have changed somehow, as even tough it's set on 2, I see last 5 days still in the backup directory. The files are all there, account files and system files for each of those daily backups.

Note that I've always been on FULL UNCOMPRESSED.

Anybody knows how to have it set like it was before, meaning that it only retains the last 2 days and nothing older than that? Thx

 

[Benjamin D.]

Bump after 24 hours. Anybody?

I do not have anything set in "Additional Destinations" if anybody is wondering. We're talking that my backups are only done in an arbitrary "Default Backup Directory" but they're not moved elsewhere afterwards, simple setup.

 

[cPanelMichael]

Hello,

By default, when backups partially complete (fails), the system will not delete the oldest backup, and it will continue to retain the oldest backups. This procedure ensures that the system retains at least one retained complete backup. Thus, you should review your backup logs in the /usr/local/cpanel/logs/cpbackup/ directory to see if backups are failing on a specific account. Additionally. you can read more about retention behavior at:

Backup Configuration - Retention Behavior

Thank you.

 

[Benjamin D.]

Very interesting. I've just checked inside yesterday's backup log and around the end of the file, I see this suspicious line, which seems to prematurely end the backup progress, any idea what that concretely means?

[2017-11-06 06:02:52 -0500] warn [backup] Pruning of backup files skipped due to errors. at /usr/local/cpanel/bin/backup line 402. bin::backup::run("bin::backup") called at /usr/local/cpanel/bin/backup line 108

 

[cPanelMichael]

[2017-11-06 06:02:52 -0500] warn [backup] Pruning of backup files skipped due to errors. at /usr/local/cpanel/bin/backup line 402. bin::backup::run("bin::backup") called at /usr/local/cpanel/bin/backup line 108

Hello,

It means that backups failed to prune because of errors during the backup generation process. You can mmanually review your backup logs in the /usr/local/cpanel/logs/cpbackup/ directory to see if backups are failing on a specific account. EX:

view /usr/local/cpanel/logs/cpbackup/XXXX.log

Then search through the log file for terms such as "error". This may help if you have never used "vi" or "view":

Using vi, the Unix Visual Editor

Thank you.

 

[Benjamin D.]

Yes I know how to use vi thanks. When I /error there's zero line beginning with "error", only a couple lines beginning with "warn" but among the whole file, the only 4 lines even containing the keyword "error" are:

/bin/gtar: /var/cpanel: file changed as we read it
1) /bin/gtar returned a non-fatal error code, 1 (Some files differ).

2) [2017-11-03 06:24:41 -0400] warn [backup] (XID ap8ma5) The command “/usr/local/cpanel/bin/cpuwatch” failed to report errors to the status file. at /usr/local/cpanel/Cpanel/Backup/Utility.pm line 225.
        Cpanel::Backup::Utility::__ANON__(Cpanel::Exception=HASH(0x21045b8)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 119
        Try::Tiny::try(CODE(0x2082ef8), Try::Tiny::Catch=REF(0x20f86e0), Try::Tiny::Finally=REF(0x20fcc68)) called at /usr/local/cpanel/Cpanel/Backup/Utility.pm line 235
        Cpanel::Backup::Utility::cpusystem(Cpanel::Backup::Utility=HASH(0x1f772d8), "pkgacct", "--nocompress", "chaletma", "/home/backup/2017-11-03/accounts", "backup") called at /usr/local/cpanel/bin/backup line 1530
        bin::backup::backup_accounts("/home/backup/2017-11-03/accounts") called at /usr/local/cpanel/bin/backup line 770
        bin::backup::__ANON__() called at /usr/local/cpanel/bin/backup line 777
        bin::backup::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 98
        eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 89
        Try::Tiny::try(CODE(0x1f9e540), Try::Tiny::Catch=REF(0x20447b8)) called at /usr/local/cpanel/bin/backup line 787
        bin::backup::_process_backups(HASH(0x1ec2130)) called at /usr/local/cpanel/bin/backup line 329
        bin::backup::run("bin::backup") called at /usr/local/cpanel/bin/backup line 108

3) [2017-11-03 06:24:41 -0400] warn [backup] Failed to back up account “chaletma”. at /usr/local/cpanel/bin/backup line 1534.
        bin::backup::backup_accounts("/home/backup/2017-11-03/accounts") called at /usr/local/cpanel/bin/backup line 770
        bin::backup::__ANON__() called at /usr/local/cpanel/bin/backup line 777
        bin::backup::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 98
        eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 89
        Try::Tiny::try(CODE(0x1f9e540), Try::Tiny::Catch=REF(0x20447b8)) called at /usr/local/cpanel/bin/backup line 787
        bin::backup::_process_backups(HASH(0x1ec2130)) called at /usr/local/cpanel/bin/backup line 329
        bin::backup::run("bin::backup") called at /usr/local/cpanel/bin/backup line 108

4) [2017-11-03 06:56:45 -0400] warn [backup] Pruning of backup files skipped due to errors. at /usr/local/cpanel/bin/backup line 402.  bin::backup::run("bin::backup") called at /usr/local/cpanel/bin/backup line 108

No idea what to do of this information.

 

[cPanelMichael]

3) [2017-11-03 06:24:41 -0400] warn [backup] Failed to back up account “chaletma”. at /usr/local/cpanel/bin/backup line 1534.

Hello,

This seems like the culprit. Search for that account username in the backup log to see what happened during the backup process. If those errors are the only thing you see for the account, try backing it up manually to see if it fails:

/scripts/pkgacct username

Thank you.

 

[Benjamin D.]

Well, you've seen what I posted. Your guess is as good as mine. I have NO IDEA what all that crashing code means, as it's all cPanel's scripts. Are you asking me to trace and debug cPanel's code?

Now running: /scripts/pkgacct chaletma

It's creating the archive... we'll see in 10-15 mins.

 

[cPanelMichael]

Hello,

I just wanted to check to verify searching for that account username in the backup logs didn't show any other output beyond what you posted. If that's all that appeared, then the next step would be to try manually packaging the account to see if it fails.

Thank you.

 

[Benjamin D.]

Allllllright. So 3 hours later, the backup has completed. Nothing wrong with it except at the end of the report, I can see this. Is this single locked/superuser file warning enough to make the whole server backup process fail? I can see on the last line: pkgacct completed

[2017-11-08 15:50:29 -0500] One or more files in the home directory were not readable and were not copied. Please review the home directory upon completion of transfer
[2017-11-08 15:50:29 -0500] WARN: Warnings(s) encountered in tar during archiving:
[2017-11-08 15:50:29 -0500] /home/chaletma/public_html/_BACKUP/archive.tar: Cannot open file: Permission denied at /scripts/pkgacct line 1470.
[2017-11-08 15:50:29 -0500] Done
[2017-11-08 15:50:29 -0500] pkgacctfile is: /home/cpmove-chaletma.tar.gz
[2017-11-08 15:50:44 -0500] md5sum is: 50c2d4357f90308b1f0925de64443bbc
[2017-11-08 15:50:44 -0500]
[2017-11-08 15:50:44 -0500] size is: 6302998629
[2017-11-08 15:50:44 -0500]
[2017-11-08 15:50:44 -0500] homesize is: 13220687872
[2017-11-08 15:50:44 -0500]
[2017-11-08 15:50:44 -0500] homefiles is: 1203317
[2017-11-08 15:50:44 -0500] pkgacct completed

 

[cPanelMichael]

[2017-11-08 15:50:29 -0500] One or more files in the home directory were not readable and were not copied. Please review the home directory upon completion of transfer
[2017-11-08 15:50:29 -0500] WARN: Warnings(s) encountered in tar during archiving:
[2017-11-08 15:50:29 -0500] /home/chaletma/public_html/_BACKUP/archive.tar: Cannot open file: Permission denied at /scripts/pkgacct line 1470.

Hello,

Yes, that's considered a partial failure and can prevent the pruning of backups. You will need to correct that error, or modify the retention settings per the document linked below to ensure backups are pruned:

Backup Configuration - Retention Behavior

Thank you.

 

[Benjamin D.]

Fair enough, but this cPanel behavior is kind of dangerous as it will fill the whole partition after a few days and it doesn't warn the sysadmin either. The problem is: It's fine that it keeps older, complete, backups, BUT it shouldn't keep the partially failed backups then! Why keep a bunch of "bad" partially finished backups until it fills the whole disk and potentially makes the server crash? Where's the logic in that?

Thanks for your help.

 

[cPanelMichael]

Hello,

While a partially failed backup may lack specific files, it's still a valid backup archive that can be restored and thus considered a safer alternative than no backup at all. There is some further discussion of this topic on the following thread:

Backup pruning behavior in cPanel 60

In "WHM Home » Server Contacts » Contact Manager", check to see if the following option is enabled under the "Notifications" tab:

Backup Failure

Make sure the contact priority is configured to notify you of backup failures, and you should receive a notification when a failure occurs.

Thank you.