Backup pruning behavior in cPanel 60

musti19 · Nov 16, 2016

Hi,
since last WHM Update, WHM dont delete the old backups after/while creating the new backups.
I dont use the legacy backup method and havent did change on configurations.
Where can be the problem?

cPanelMichael · Nov 16, 2016

Hello,

Please post the output from the following commands:

Code:

cat /usr/local/cpanel/version
cat /var/cpanel/backups/config

Thank you.

musti19 · Nov 16, 2016

11.60.0.24

and

Code:

BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 1,3,5
BACKUPDIR: /1_cpanel_backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'yes'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'yes'
BACKUPTYPE: compressed
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 3
BACKUP_MONTHLY_DATES: 1
BACKUP_MONTHLY_ENABLE: 'yes'
BACKUP_MONTHLY_RETENTION: 1
BACKUP_WEEKLY_DAY: 6
BACKUP_WEEKLY_ENABLE: 'yes'
BACKUP_WEEKLY_RETENTION: 2
ERRORTHRESHHOLD: 3
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 2700
MYSQLBACKUP: both
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'
USEBINARYPKGACCT: 'no'

cPanelMichael · Nov 16, 2016

Hello,

The following case was implemented in cPanel version 60:

Fixed case CPANEL-7446: Don't prune old backups if backup was a partial failure.

This is to ensure backups aren't pruned in situations where a partial backup failure occurs. Could you review the backup logs in /usr/local/cpanel/logs/cpbackup/ to see if you notice any entries for "PartialFailure"? If so, you'd want to review the logs to see which accounts partially failed to determine what happened.

Thanks!

musti19 · Nov 17, 2016

thanks for your help,
but it is normal that if a user have exceeded the quota take the backup-process to failure?

Code:

[2016-11-16 04:28:29 +0100] Copying SSL certificates, CSRs, and keys...[2016-11-16 04:28:29 +0100] warn [pkgacct] Failed to write the cache file â/home/web832/ssl/ssl.db.cacheâ (Cpanel::Exception::IO::CloseError/(XID vhnypy) 
The system failed to close a file handle for â/home/web832/ssl/ssl.db.cache.tmp.6793260â because of the following error: Disk quota exceeded
at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 77, <GEN0> line 1.
    Cpanel::Exception::create("IO::CloseError", ARRAY(0x34a51a8)) called at /usr/local/cpanel/Cpanel/Autodie/IO.pm line 57
    Cpanel::Autodie::IO::close(GLOB(0x344a4f8), "/home/web832/ssl/ssl.db.cache.tmp.6793260") called at /usr/local/cpanel/Cpanel/FileUtils/Write.pm line 125
    Cpanel::FileUtils::Write::_write_to_tmpfile("/home/web832/ssl/ssl.db.cache", "{}", 33188, CODE(0x18cfd68)) called at /usr/local/cpanel/Cpanel/FileUtils/Write.pm line 64
    Cpanel::FileUtils::Write::overwrite("/home/web832/ssl/ssl.db.cache", "{}", 33188) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 320
    Cpanel::CachedDataStore::__ANON__() called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80
    eval {...} called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71
    Try::Tiny::try(CODE(0x344ab70), Try::Tiny::Catch=REF(0x34357d8)) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 329
    Cpanel::CachedDataStore::_write_cache_file("/home/web832/ssl/ssl.db.cache", HASH(0x34bbd38), 33188) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 277
    Cpanel::CachedDataStore::loaddatastore("/home/web832/ssl/ssl.db", 0, undef, HASH(0x344a090)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 1223
    Cpanel::SSLStorage::_load_datastore(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 1267
    Cpanel::SSLStorage::_load_datastore_ro(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 848
    Cpanel::SSLStorage::__ANON__() called at /usr/local/cpanel/Cpanel/AccessIds/ReducedPrivileges.pm line 85
    eval {...} called at /usr/local/cpanel/Cpanel/AccessIds/ReducedPrivileges.pm line 85
    Cpanel::AccessIds::ReducedPrivileges::call_as_user(CODE(0x344a300), "web832", "web832") called at /usr/local/cpanel/Cpanel/SSLStorage/User.pm line 676
    Cpanel::SSLStorage::User::_execute_coderef(Cpanel::SSLStorage::User=HASH(0x19d1e40), CODE(0x344a300)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 850
    Cpanel::SSLStorage::_find(Cpanel::SSLStorage::User=HASH(0x19d1e40), "type", "certificate") called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 167
    Cpanel::SSLStorage::find_certificates(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 269
    Cpanel::SSLStorage::export(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at bin/pkgacct.pl line 2140
    eval {...} called at bin/pkgacct.pl line 2140
    Script::Pkgacct::__ANON__(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at bin/pkgacct.pl line 2157
    Script::Pkgacct::backup_ssl_for_user("web832", "/1_cpanel_backup/2016-11-16/accounts/web832", Cpanel::Output::Pkgacct=HASH(0x2930e08)) called at bin/pkgacct.pl line 619
    Script::Pkgacct::script("Script::Pkgacct", "web832", "/1_cpanel_backup/2016-11-16/accounts", "backup") called at bin/pkgacct.pl line 134
    Script::Pkgacct::__ANON__() called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80
    eval {...} called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71
    Try::Tiny::try(CODE(0x290bfe8), Try::Tiny::Catch=REF(0x18c1668)) called at bin/pkgacct.pl line 144
); this file will not be saved. at /usr/local/cpanel/Cpanel/Debug.pm line 30, <GEN0> line 1.

cPanelMichael · Nov 17, 2016

Hello,

Pruning is skipped if errors resulting in the failure to backup account an account occurs during the backup generation. This is to avoid
getting rid of potentially good backups in favor of erroneous ones.

To clarify, is the entire backup process stopping completely once that error is encountered, or is it only the pruning that fails?

Thank you.

stassy · Nov 21, 2016

Hi,

Exactly same problem here. Old backup files/folders are not removed (local and FTP). This problem started after 8th November.

Code:

> cat /usr/local/cpanel/version
11.60.0.24

> cat /var/cpanel/backups/config
---
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 0,1,2,3,4,5,6
BACKUPDIR: /backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'no'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'no'
BACKUPTYPE: compressed
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 31
BACKUP_MONTHLY_DATES: 1
BACKUP_MONTHLY_ENABLE: 'yes'
BACKUP_MONTHLY_RETENTION: 4
BACKUP_WEEKLY_DAY: 1
BACKUP_WEEKLY_ENABLE: 'yes'
BACKUP_WEEKLY_RETENTION: 4
ERRORTHRESHHOLD: 3
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 3600
MYSQLBACKUP: both
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'

> cd /usr/local/cpanel/logs/cpbackup/
> grep -i PartialFailure *
1478826001.log:[2016-11-11 03:13:43 +0100] info [backup] Final state is Backup::PartialFailure (0)
1478826001.log:[2016-11-11 03:13:44 +0100] info [backup] Sent Backup::PartialFailure notification.
1478912402.log:[2016-11-12 03:19:30 +0100] info [backup] Final state is Backup::PartialFailure (0)
1478912402.log:[2016-11-12 03:19:30 +0100] info [backup] Sent Backup::PartialFailure notification.
1478998801.log:[2016-11-13 03:23:21 +0100] info [backup] Final state is Backup::PartialFailure (0)
1478998801.log:[2016-11-13 03:23:21 +0100] info [backup] Sent Backup::PartialFailure notification.
1479085202.log:[2016-11-14 03:30:16 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479085202.log:[2016-11-14 03:30:16 +0100] info [backup] Sent Backup::PartialFailure notification.
1479171601.log:[2016-11-15 03:23:13 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479171601.log:[2016-11-15 03:23:13 +0100] info [backup] Sent Backup::PartialFailure notification.
1479258001.log:[2016-11-16 03:25:32 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479258001.log:[2016-11-16 03:25:34 +0100] info [backup] Sent Backup::PartialFailure notification.
1479344402.log:[2016-11-17 03:23:07 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479344402.log:[2016-11-17 03:23:07 +0100] info [backup] Sent Backup::PartialFailure notification.
1479430802.log:[2016-11-18 03:22:33 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479430802.log:[2016-11-18 03:22:34 +0100] info [backup] Sent Backup::PartialFailure notification.
1479517201.log:[2016-11-19 03:34:51 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479517201.log:[2016-11-19 03:34:51 +0100] info [backup] Sent Backup::PartialFailure notification.
1479603602.log:[2016-11-20 03:36:40 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479603602.log:[2016-11-20 03:36:40 +0100] info [backup] Sent Backup::PartialFailure notification.
1479690002.log:[2016-11-21 03:37:51 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479690002.log:[2016-11-21 03:37:52 +0100] info [backup] Sent Backup::PartialFailure notification.

Any idea ?

Thanks

cPanelMichael · Nov 21, 2016

stassy said:
1479690002.log:[2016-11-21 03:37:52 +0100] info [backup] Sent Backup::PartialFailure notification.

This suggests a partial backup failure occurred. Could you browse through a specific backup log using the "cat" or "view" command to see if you notice any error or failure messages related to any of the accounts? The log files are stored in the /usr/local/cpanel/logs/cpbackup/ directory.

Thank you.

musti19 · Nov 21, 2016

yes, error on accounts that overquota:

Code:

2016-11-21 02:41:32 +0100] Copying SSL certificates, CSRs, and keys...[2016-11-21 02:41:32 +0100] warn [pkgacct] Failed to write the cache file â/home/web1110/ssl/ssl.db.cacheâ (Cpanel::Exception::IO::CloseError/(XID 67e6mf) â/home/web1110/ssl/ssl.db.cache.tmp.93205435â: Disk quota exceeded
The system failed to close a file handle for â/home/web1110/ssl/ssl.db.cache.tmp.93205435â because of the following error: Disk quota exceeded
at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 77, <GEN0> line 1.
.
.
[2016-11-21 02:41:34 +0100] warn [pkgacct] could not acquire a lock for '/home/web1110/cpbackup-exclude.conf': Disk quota exceeded at /usr/local/cpanel/Cpanel/Debug.pm line 30.
    Cpanel::Debug::log_warn("could not acquire a lock for '/home/web1110/cpbackup-exclude."...) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 377

How can this be a reason to no complete backup processes..

stassy · Nov 22, 2016

cPanelMichael said:
This suggests a partial backup failure occurred. Could you browse through a specific backup log using the "cat" or "view" command to see if you notice any error or failure messages related to any of the accounts? The log files are stored in the /usr/local/cpanel/logs/cpbackup/ directory.

Thank you.

You were right, I didn't notice when I was looking in these files.

I've removed a useless database with phpMyAdmin but didn't remove it from the cPanel databases interface.
And that's why mysqldump failed, it was not able to select the removed database.

Quite hard for an error. Warning should be better

By the way, thanks again

cPanelMichael · Nov 22, 2016

musti19 said:
2016-11-21 02:41:32 +0100] Copying SSL certificates, CSRs, and keys...[2016-11-21 02:41:32 +0100] warn [pkgacct] Failed to write the cache file â/home/web1110/ssl/ssl.db.cacheâ (Cpanel::Exception::IO::CloseError/(XID 67e6mf) â/home/web1110/ssl/ssl.db.cache.tmp.93205435â: Disk quota exceeded The system failed to close a file handle for â/home/web1110/ssl/ssl.db.cache.tmp.93205435â because of the following error: Disk quota exceeded at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 77, <GEN0> line 1.

Yes, this is a considered a failure and thus backups are not pruned to avoid a situation where the new backup is corrupted or doesn't contain all of the website/account data.

stassy said:
I've removed a useless database with phpMyAdmin but didn't remove it from the cPanel databases interface.
And that's why mysqldump failed, it was not able to select the removed database.

Quite hard for an error. Warning should be better

Check to ensure "Backup Failure" notifications are enabled under "WHM >> Contact Manager". This will ensure you are notified when partial backup failures occur. Also, what's the specific error message you noticed in the backup logs related to the failure?

Thank you.

musti19 · Nov 22, 2016

and what is the solution? Until last update,there was not this problem..
I cannot go and delete the backups all three days manually,
because some accounts are overquota.

cPanelMichael · Nov 22, 2016

Hello @musti19,

Could you verify if it's the accounts over quota, or if your entire system is at 100% disk space usage? You can check with a command such as:

Code:

df -h

If it's individual accounts, you could disable the accounts over quota from backup generation via the following option in WHM:

"WHM Home » Backup » Backup User Selection"

Then, force a new backup generation via:

Code:

/usr/local/cpanel/bin/backup --force

This should prune the old backups since the system should not encounter backup failures with the over-quota accounts disabled.

Thank you.

stassy · Nov 22, 2016

cPanelMichael said:
Check to ensure "Backup Failure" notifications are enabled under "WHM >> Contact Manager". This will ensure you are notified when partial backup failures occur. Also, what's the specific error message you noticed in the backup logs related to the failure?

Thank you.

"Backup Failure" was not enabled in Notifications, only "Backup Failed To Finish" was. Now, I've changed this to enable both.

The error message in the log was:
[2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
[2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
/usr/bin/mysqlcheck: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
[2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
[2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump failed -- database may be corrupt
[2016-11-21 03:04:21 +0100] (787 bytes)
[2016-11-21 03:04:21 +0100] Failed to dump database stagtrip_test: The subprocess reported error number 2 when it ended.
[2016-11-21 03:04:21 +0100]
[2016-11-21 03:04:21 +0100] WARNING: There was a problem dumping one or more databases

Message is clear enough, but as I didn't receive any emails...

And this stagtrip_test database was removed by me with phpMyAdmin and not with the cPanel interface;

Thanks

musti19 · Nov 22, 2016

/dev/md2 1.8T 1.1T 625G 64% /
i checked some accounts and it is right, that they are overquota

should i now check every day, if user is overquota and disable it?

cPanelMichael · Nov 22, 2016

@stassy - Keep in mind that it's always advised to remove databases via cPanel or via an API function. Manually removing databases via the command line or phpMyAdmin doesn't disassociate the database from the cPanel account.

@musti19 - I recommend disabling backups for the accounts over quota first just to make sure that's the actual reason why pruning is failing. It's possible there's another issue you haven't noticed that's preventing the backup from completing successfully. Also, regarding the disk space issue, the following options under the "Notifications" tab in "WHM >> Tweak Settings" are useful for this purpose:

Account disk quota “critical” percentage
Notify admin or reseller when disk quota reaches “critical” state

This will ensure you are notified when accounts reach a critical disk space usage percentage.

Thank you.

mccreare · Nov 30, 2016

I'm having this problem too. Over quota accounts generate a Partial Failed error, then those backups are not removed, stalling the whole system. I can't give my customer's free space, and not backing up accounts isn't an option either. This doesn't work.

cPanelMichael · Dec 2, 2016

mccreare said:
I'm having this problem too. Over quota accounts generate a Partial Failed error, then those backups are not removed, stalling the whole system. I can't give my customer's free space, and not backing up accounts isn't an option either. This doesn't work.

Hello @mccreare,

cPanel accounts are not designed to function properly when at their disk space limit. This can lead to potential issues with services such as email. If the account has reached it's disk space limit, it's already in a state where it won't complete a full backup generation. Thus, in those cases, you should exclude it from the backups via "WHM Home » Backup » Backup User Selection".

Note that if the backup system were to prune backups when failures or partial failures occur, then it would create a situation where a previous valid backup is pruned, and no new backup is created, leaving the account with no usable backup archives. What you are seeking is new functionality, such as an option to temporarily increase an account's quota when backups are generated. I recommend submitting a feature request for the functionality you'd like to see via:

Submit A Feature Request

Let us know if you have any additional questions or concerns.

Thank you.

mccreare · Dec 2, 2016

No...
1. Shared hosts have to deal with customers going over quota all the time. So, it's not like there's anything we can do about that. I'm not going to beg every customer to get their act in gear each time they let their space go over so we can have functional backups again. With thousands of customers, that's absurd. We also offer backups to everyone, including anyone over quota, so these accounts need to be backed up anyway. There is no reason at all this should fail to work. Just back up the account properly with no failures, whether or not it is "over quota". Problem solved.
2. The system worked fine up until this most recent update, so I'm definitely not asking for a new feature. I'm asking you to un-break the system that was not broken until very recently.

NanoGame · Dec 4, 2016

Hello.

I have same problem. Please fix it.

[2016-12-04 21:48:33 +0300] info [backup] Final state is Backup::PartialFailure (0)
[2016-12-04 21:48:33 +0300] info [backup] Sent Backup::PartialFailure notification.

[2016-12-04 02:04:59 +0300] Copying SSL certificates, CSRs, and keys...[2016-12-04 02:04:59 +0300] warn [pkgacct] Failed to write the cache file “/home/delaemva/ssl/ssl.db.cache” (Cpanel::Exception::IO::CloseError/(XID 7bnwvg) Системе не удалось закрыть дескриптор файла для «/home/delaemva/ssl/ssl.db.cache.tmp.39861624» из-за следующей ошибки: Disk quota exceeded
The system failed to close a file handle for “/home/delaemva/ssl/ssl.db.cache.tmp.39861624” because of the following error: Disk quota exceeded

Thread starter	Similar threads	Forum	Replies	Date
A	Incremental backup stuck at "Pruning backup directory"	Backups	1	Dec 12, 2021
T	Manually Pruning older Backups	Backups	3	Nov 25, 2021
E	My daily saga with backup pruning errors	Backups	3	Feb 16, 2021
E	Pruning of backup files skipped apparently without reason	Backups	1	Dec 30, 2020
C	Backup Pruning Timeout	Backups	12	Jan 27, 2020

Backup pruning behavior in cPanel 60

Well-Known Member

Administrator

Well-Known Member

Administrator

Well-Known Member

Administrator

Member

Administrator

Well-Known Member

Member

Administrator

Well-Known Member

Administrator

Member

Well-Known Member

Administrator

Member

Administrator

Member

Member