Backup pruning behavior in cPanel 60

musti19

Well-Known Member
Jan 20, 2013
110
1
68
cPanel Access Level
Root Administrator
Hi,
since last WHM Update, WHM dont delete the old backups after/while creating the new backups.
I dont use the legacy backup method and havent did change on configurations.
Where can be the problem?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello,

Please post the output from the following commands:

Code:
cat /usr/local/cpanel/version
cat /var/cpanel/backups/config
Thank you.
 

musti19

Well-Known Member
Jan 20, 2013
110
1
68
cPanel Access Level
Root Administrator
11.60.0.24

and

Code:
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 1,3,5
BACKUPDIR: /1_cpanel_backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'yes'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'yes'
BACKUPTYPE: compressed
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 3
BACKUP_MONTHLY_DATES: 1
BACKUP_MONTHLY_ENABLE: 'yes'
BACKUP_MONTHLY_RETENTION: 1
BACKUP_WEEKLY_DAY: 6
BACKUP_WEEKLY_ENABLE: 'yes'
BACKUP_WEEKLY_RETENTION: 2
ERRORTHRESHHOLD: 3
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 2700
MYSQLBACKUP: both
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'
USEBINARYPKGACCT: 'no'
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello,

The following case was implemented in cPanel version 60:

Fixed case CPANEL-7446: Don't prune old backups if backup was a partial failure.

This is to ensure backups aren't pruned in situations where a partial backup failure occurs. Could you review the backup logs in /usr/local/cpanel/logs/cpbackup/ to see if you notice any entries for "PartialFailure"? If so, you'd want to review the logs to see which accounts partially failed to determine what happened.

Thanks!
 

musti19

Well-Known Member
Jan 20, 2013
110
1
68
cPanel Access Level
Root Administrator
thanks for your help,
but it is normal that if a user have exceeded the quota take the backup-process to failure?
Code:
[2016-11-16 04:28:29 +0100] Copying SSL certificates, CSRs, and keys...[2016-11-16 04:28:29 +0100] warn [pkgacct] Failed to write the cache file “/home/web832/ssl/ssl.db.cache” (Cpanel::Exception::IO::CloseError/(XID vhnypy) 
The system failed to close a file handle for “/home/web832/ssl/ssl.db.cache.tmp.6793260” because of the following error: Disk quota exceeded
at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 77, <GEN0> line 1.
    Cpanel::Exception::create("IO::CloseError", ARRAY(0x34a51a8)) called at /usr/local/cpanel/Cpanel/Autodie/IO.pm line 57
    Cpanel::Autodie::IO::close(GLOB(0x344a4f8), "/home/web832/ssl/ssl.db.cache.tmp.6793260") called at /usr/local/cpanel/Cpanel/FileUtils/Write.pm line 125
    Cpanel::FileUtils::Write::_write_to_tmpfile("/home/web832/ssl/ssl.db.cache", "{}", 33188, CODE(0x18cfd68)) called at /usr/local/cpanel/Cpanel/FileUtils/Write.pm line 64
    Cpanel::FileUtils::Write::overwrite("/home/web832/ssl/ssl.db.cache", "{}", 33188) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 320
    Cpanel::CachedDataStore::__ANON__() called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80
    eval {...} called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71
    Try::Tiny::try(CODE(0x344ab70), Try::Tiny::Catch=REF(0x34357d8)) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 329
    Cpanel::CachedDataStore::_write_cache_file("/home/web832/ssl/ssl.db.cache", HASH(0x34bbd38), 33188) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 277
    Cpanel::CachedDataStore::loaddatastore("/home/web832/ssl/ssl.db", 0, undef, HASH(0x344a090)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 1223
    Cpanel::SSLStorage::_load_datastore(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 1267
    Cpanel::SSLStorage::_load_datastore_ro(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 848
    Cpanel::SSLStorage::__ANON__() called at /usr/local/cpanel/Cpanel/AccessIds/ReducedPrivileges.pm line 85
    eval {...} called at /usr/local/cpanel/Cpanel/AccessIds/ReducedPrivileges.pm line 85
    Cpanel::AccessIds::ReducedPrivileges::call_as_user(CODE(0x344a300), "web832", "web832") called at /usr/local/cpanel/Cpanel/SSLStorage/User.pm line 676
    Cpanel::SSLStorage::User::_execute_coderef(Cpanel::SSLStorage::User=HASH(0x19d1e40), CODE(0x344a300)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 850
    Cpanel::SSLStorage::_find(Cpanel::SSLStorage::User=HASH(0x19d1e40), "type", "certificate") called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 167
    Cpanel::SSLStorage::find_certificates(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 269
    Cpanel::SSLStorage::export(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at bin/pkgacct.pl line 2140
    eval {...} called at bin/pkgacct.pl line 2140
    Script::Pkgacct::__ANON__(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at bin/pkgacct.pl line 2157
    Script::Pkgacct::backup_ssl_for_user("web832", "/1_cpanel_backup/2016-11-16/accounts/web832", Cpanel::Output::Pkgacct=HASH(0x2930e08)) called at bin/pkgacct.pl line 619
    Script::Pkgacct::script("Script::Pkgacct", "web832", "/1_cpanel_backup/2016-11-16/accounts", "backup") called at bin/pkgacct.pl line 134
    Script::Pkgacct::__ANON__() called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80
    eval {...} called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71
    Try::Tiny::try(CODE(0x290bfe8), Try::Tiny::Catch=REF(0x18c1668)) called at bin/pkgacct.pl line 144
); this file will not be saved. at /usr/local/cpanel/Cpanel/Debug.pm line 30, <GEN0> line 1.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello,

Pruning is skipped if errors resulting in the failure to backup account an account occurs during the backup generation. This is to avoid
getting rid of potentially good backups in favor of erroneous ones.

To clarify, is the entire backup process stopping completely once that error is encountered, or is it only the pruning that fails?

Thank you.
 

stassy

Member
Dec 30, 2004
5
0
151
Hi,

Exactly same problem here. Old backup files/folders are not removed (local and FTP). This problem started after 8th November.
Code:
> cat /usr/local/cpanel/version
11.60.0.24

> cat /var/cpanel/backups/config
---
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 0,1,2,3,4,5,6
BACKUPDIR: /backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'no'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'no'
BACKUPTYPE: compressed
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 31
BACKUP_MONTHLY_DATES: 1
BACKUP_MONTHLY_ENABLE: 'yes'
BACKUP_MONTHLY_RETENTION: 4
BACKUP_WEEKLY_DAY: 1
BACKUP_WEEKLY_ENABLE: 'yes'
BACKUP_WEEKLY_RETENTION: 4
ERRORTHRESHHOLD: 3
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 3600
MYSQLBACKUP: both
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'

> cd /usr/local/cpanel/logs/cpbackup/
> grep -i PartialFailure *
1478826001.log:[2016-11-11 03:13:43 +0100] info [backup] Final state is Backup::PartialFailure (0)
1478826001.log:[2016-11-11 03:13:44 +0100] info [backup] Sent Backup::PartialFailure notification.
1478912402.log:[2016-11-12 03:19:30 +0100] info [backup] Final state is Backup::PartialFailure (0)
1478912402.log:[2016-11-12 03:19:30 +0100] info [backup] Sent Backup::PartialFailure notification.
1478998801.log:[2016-11-13 03:23:21 +0100] info [backup] Final state is Backup::PartialFailure (0)
1478998801.log:[2016-11-13 03:23:21 +0100] info [backup] Sent Backup::PartialFailure notification.
1479085202.log:[2016-11-14 03:30:16 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479085202.log:[2016-11-14 03:30:16 +0100] info [backup] Sent Backup::PartialFailure notification.
1479171601.log:[2016-11-15 03:23:13 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479171601.log:[2016-11-15 03:23:13 +0100] info [backup] Sent Backup::PartialFailure notification.
1479258001.log:[2016-11-16 03:25:32 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479258001.log:[2016-11-16 03:25:34 +0100] info [backup] Sent Backup::PartialFailure notification.
1479344402.log:[2016-11-17 03:23:07 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479344402.log:[2016-11-17 03:23:07 +0100] info [backup] Sent Backup::PartialFailure notification.
1479430802.log:[2016-11-18 03:22:33 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479430802.log:[2016-11-18 03:22:34 +0100] info [backup] Sent Backup::PartialFailure notification.
1479517201.log:[2016-11-19 03:34:51 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479517201.log:[2016-11-19 03:34:51 +0100] info [backup] Sent Backup::PartialFailure notification.
1479603602.log:[2016-11-20 03:36:40 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479603602.log:[2016-11-20 03:36:40 +0100] info [backup] Sent Backup::PartialFailure notification.
1479690002.log:[2016-11-21 03:37:51 +0100] info [backup] Final state is Backup::PartialFailure (0)
1479690002.log:[2016-11-21 03:37:52 +0100] info [backup] Sent Backup::PartialFailure notification.
Any idea ?

Thanks
 
Last edited by a moderator:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
1479690002.log:[2016-11-21 03:37:52 +0100] info [backup] Sent Backup::PartialFailure notification.
This suggests a partial backup failure occurred. Could you browse through a specific backup log using the "cat" or "view" command to see if you notice any error or failure messages related to any of the accounts? The log files are stored in the /usr/local/cpanel/logs/cpbackup/ directory.

Thank you.
 

musti19

Well-Known Member
Jan 20, 2013
110
1
68
cPanel Access Level
Root Administrator
yes, error on accounts that overquota:
Code:
2016-11-21 02:41:32 +0100] Copying SSL certificates, CSRs, and keys...[2016-11-21 02:41:32 +0100] warn [pkgacct] Failed to write the cache file “/home/web1110/ssl/ssl.db.cache” (Cpanel::Exception::IO::CloseError/(XID 67e6mf) „/home/web1110/ssl/ssl.db.cache.tmp.93205435“: Disk quota exceeded
The system failed to close a file handle for “/home/web1110/ssl/ssl.db.cache.tmp.93205435” because of the following error: Disk quota exceeded
at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 77, <GEN0> line 1.
.
.
[2016-11-21 02:41:34 +0100] warn [pkgacct] could not acquire a lock for '/home/web1110/cpbackup-exclude.conf': Disk quota exceeded at /usr/local/cpanel/Cpanel/Debug.pm line 30.
    Cpanel::Debug::log_warn("could not acquire a lock for '/home/web1110/cpbackup-exclude."...) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 377
How can this be a reason to no complete backup processes..
 

stassy

Member
Dec 30, 2004
5
0
151
This suggests a partial backup failure occurred. Could you browse through a specific backup log using the "cat" or "view" command to see if you notice any error or failure messages related to any of the accounts? The log files are stored in the /usr/local/cpanel/logs/cpbackup/ directory.

Thank you.
You were right, I didn't notice when I was looking in these files.

I've removed a useless database with phpMyAdmin but didn't remove it from the cPanel databases interface.
And that's why mysqldump failed, it was not able to select the removed database.

Quite hard for an error. Warning should be better :)

By the way, thanks again
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
2016-11-21 02:41:32 +0100] Copying SSL certificates, CSRs, and keys...[2016-11-21 02:41:32 +0100] warn [pkgacct] Failed to write the cache file “/home/web1110/ssl/ssl.db.cache” (Cpanel::Exception::IO::CloseError/(XID 67e6mf) „/home/web1110/ssl/ssl.db.cache.tmp.93205435“: Disk quota exceeded The system failed to close a file handle for “/home/web1110/ssl/ssl.db.cache.tmp.93205435” because of the following error: Disk quota exceeded at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 77, <GEN0> line 1.
Yes, this is a considered a failure and thus backups are not pruned to avoid a situation where the new backup is corrupted or doesn't contain all of the website/account data.

I've removed a useless database with phpMyAdmin but didn't remove it from the cPanel databases interface.
And that's why mysqldump failed, it was not able to select the removed database.

Quite hard for an error. Warning should be better
Check to ensure "Backup Failure" notifications are enabled under "WHM >> Contact Manager". This will ensure you are notified when partial backup failures occur. Also, what's the specific error message you noticed in the backup logs related to the failure?

Thank you.
 

musti19

Well-Known Member
Jan 20, 2013
110
1
68
cPanel Access Level
Root Administrator
and what is the solution? Until last update,there was not this problem..
I cannot go and delete the backups all three days manually,
because some accounts are overquota.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello @musti19,

Could you verify if it's the accounts over quota, or if your entire system is at 100% disk space usage? You can check with a command such as:

Code:
df -h
If it's individual accounts, you could disable the accounts over quota from backup generation via the following option in WHM:

"WHM Home » Backup » Backup User Selection"

Then, force a new backup generation via:

Code:
/usr/local/cpanel/bin/backup --force
This should prune the old backups since the system should not encounter backup failures with the over-quota accounts disabled.

Thank you.
 

stassy

Member
Dec 30, 2004
5
0
151
Check to ensure "Backup Failure" notifications are enabled under "WHM >> Contact Manager". This will ensure you are notified when partial backup failures occur. Also, what's the specific error message you noticed in the backup logs related to the failure?

Thank you.
"Backup Failure" was not enabled in Notifications, only "Backup Failed To Finish" was. Now, I've changed this to enable both.

The error message in the log was:
[2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
[2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
/usr/bin/mysqlcheck: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
[2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
[2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump failed -- database may be corrupt
[2016-11-21 03:04:21 +0100] (787 bytes)
[2016-11-21 03:04:21 +0100] Failed to dump database stagtrip_test: The subprocess reported error number 2 when it ended.
[2016-11-21 03:04:21 +0100]
[2016-11-21 03:04:21 +0100] WARNING: There was a problem dumping one or more databases

Message is clear enough, but as I didn't receive any emails...

And this stagtrip_test database was removed by me with phpMyAdmin and not with the cPanel interface;

Thanks
 
Last edited:

musti19

Well-Known Member
Jan 20, 2013
110
1
68
cPanel Access Level
Root Administrator
/dev/md2 1.8T 1.1T 625G 64% /
i checked some accounts and it is right, that they are overquota

should i now check every day, if user is overquota and disable it?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
@stassy - Keep in mind that it's always advised to remove databases via cPanel or via an API function. Manually removing databases via the command line or phpMyAdmin doesn't disassociate the database from the cPanel account.

@musti19 - I recommend disabling backups for the accounts over quota first just to make sure that's the actual reason why pruning is failing. It's possible there's another issue you haven't noticed that's preventing the backup from completing successfully. Also, regarding the disk space issue, the following options under the "Notifications" tab in "WHM >> Tweak Settings" are useful for this purpose:

Account disk quota “critical” percentage
Notify admin or reseller when disk quota reaches “critical” state

This will ensure you are notified when accounts reach a critical disk space usage percentage.

Thank you.
 

mccreare

Member
Apr 23, 2003
18
1
153
I'm having this problem too. Over quota accounts generate a Partial Failed error, then those backups are not removed, stalling the whole system. I can't give my customer's free space, and not backing up accounts isn't an option either. This doesn't work.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
I'm having this problem too. Over quota accounts generate a Partial Failed error, then those backups are not removed, stalling the whole system. I can't give my customer's free space, and not backing up accounts isn't an option either. This doesn't work.
Hello @mccreare,

cPanel accounts are not designed to function properly when at their disk space limit. This can lead to potential issues with services such as email. If the account has reached it's disk space limit, it's already in a state where it won't complete a full backup generation. Thus, in those cases, you should exclude it from the backups via "WHM Home » Backup » Backup User Selection".

Note that if the backup system were to prune backups when failures or partial failures occur, then it would create a situation where a previous valid backup is pruned, and no new backup is created, leaving the account with no usable backup archives. What you are seeking is new functionality, such as an option to temporarily increase an account's quota when backups are generated. I recommend submitting a feature request for the functionality you'd like to see via:

Submit A Feature Request

Let us know if you have any additional questions or concerns.

Thank you.
 

mccreare

Member
Apr 23, 2003
18
1
153
No...
1. Shared hosts have to deal with customers going over quota all the time. So, it's not like there's anything we can do about that. I'm not going to beg every customer to get their act in gear each time they let their space go over so we can have functional backups again. With thousands of customers, that's absurd. We also offer backups to everyone, including anyone over quota, so these accounts need to be backed up anyway. There is no reason at all this should fail to work. Just back up the account properly with no failures, whether or not it is "over quota". Problem solved.
2. The system worked fine up until this most recent update, so I'm definitely not asking for a new feature. I'm asking you to un-break the system that was not broken until very recently.
 
  • Like
Reactions: musti19

NanoGame

Member
Oct 6, 2016
5
6
1
Russia
cPanel Access Level
Root Administrator
Hello.

I have same problem. Please fix it.

[2016-12-04 21:48:33 +0300] info [backup] Final state is Backup::PartialFailure (0)
[2016-12-04 21:48:33 +0300] info [backup] Sent Backup::PartialFailure notification.

[2016-12-04 02:04:59 +0300] Copying SSL certificates, CSRs, and keys...[2016-12-04 02:04:59 +0300] warn [pkgacct] Failed to write the cache file “/home/delaemva/ssl/ssl.db.cache” (Cpanel::Exception::IO::CloseError/(XID 7bnwvg) Системе не удалось закрыть дескриптор файла для «/home/delaemva/ssl/ssl.db.cache.tmp.39861624» из-за следующей ошибки: Disk quota exceeded
The system failed to close a file handle for “/home/delaemva/ssl/ssl.db.cache.tmp.39861624” because of the following error: Disk quota exceeded