Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Backup pruning behavior in cPanel 60

Discussion in 'Data Protection' started by musti19, Nov 16, 2016.

Tags:
  1. musti19

    musti19 Well-Known Member

    Joined:
    Jan 20, 2013
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Hi,
    since last WHM Update, WHM dont delete the old backups after/while creating the new backups.
    I dont use the legacy backup method and havent did change on configurations.
    Where can be the problem?
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,424
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    Please post the output from the following commands:

    Code:
    cat /usr/local/cpanel/version
    cat /var/cpanel/backups/config
    Thank you.
     
  3. musti19

    musti19 Well-Known Member

    Joined:
    Jan 20, 2013
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    11.60.0.24

    and

    Code:
    BACKUPACCTS: 'yes'
    BACKUPBWDATA: 'yes'
    BACKUPDAYS: 1,3,5
    BACKUPDIR: /1_cpanel_backup
    BACKUPENABLE: 'yes'
    BACKUPFILES: 'yes'
    BACKUPLOGS: 'yes'
    BACKUPMOUNT: 'no'
    BACKUPSUSPENDEDACCTS: 'yes'
    BACKUPTYPE: compressed
    BACKUP_DAILY_ENABLE: 'yes'
    BACKUP_DAILY_RETENTION: 3
    BACKUP_MONTHLY_DATES: 1
    BACKUP_MONTHLY_ENABLE: 'yes'
    BACKUP_MONTHLY_RETENTION: 1
    BACKUP_WEEKLY_DAY: 6
    BACKUP_WEEKLY_ENABLE: 'yes'
    BACKUP_WEEKLY_RETENTION: 2
    ERRORTHRESHHOLD: 3
    GZIPRSYNCOPTS: --rsyncable
    KEEPLOCAL: 1
    LINKDEST: 0
    LOCALZONESONLY: 'no'
    MAXIMUM_RESTORE_TIMEOUT: 21600
    MAXIMUM_TIMEOUT: 2700
    MYSQLBACKUP: both
    POSTBACKUP: 'no'
    PREBACKUP: -1
    PSQLBACKUP: 'no'
    USEBINARYPKGACCT: 'no'
     
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,424
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    The following case was implemented in cPanel version 60:

    Fixed case CPANEL-7446: Don't prune old backups if backup was a partial failure.

    This is to ensure backups aren't pruned in situations where a partial backup failure occurs. Could you review the backup logs in /usr/local/cpanel/logs/cpbackup/ to see if you notice any entries for "PartialFailure"? If so, you'd want to review the logs to see which accounts partially failed to determine what happened.

    Thanks!
     
  5. musti19

    musti19 Well-Known Member

    Joined:
    Jan 20, 2013
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    thanks for your help,
    but it is normal that if a user have exceeded the quota take the backup-process to failure?
    Code:
    [2016-11-16 04:28:29 +0100] Copying SSL certificates, CSRs, and keys...[2016-11-16 04:28:29 +0100] warn [pkgacct] Failed to write the cache file “/home/web832/ssl/ssl.db.cache” (Cpanel::Exception::IO::CloseError/(XID vhnypy) 
    The system failed to close a file handle for “/home/web832/ssl/ssl.db.cache.tmp.6793260” because of the following error: Disk quota exceeded
    at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 77, <GEN0> line 1.
        Cpanel::Exception::create("IO::CloseError", ARRAY(0x34a51a8)) called at /usr/local/cpanel/Cpanel/Autodie/IO.pm line 57
        Cpanel::Autodie::IO::close(GLOB(0x344a4f8), "/home/web832/ssl/ssl.db.cache.tmp.6793260") called at /usr/local/cpanel/Cpanel/FileUtils/Write.pm line 125
        Cpanel::FileUtils::Write::_write_to_tmpfile("/home/web832/ssl/ssl.db.cache", "{}", 33188, CODE(0x18cfd68)) called at /usr/local/cpanel/Cpanel/FileUtils/Write.pm line 64
        Cpanel::FileUtils::Write::overwrite("/home/web832/ssl/ssl.db.cache", "{}", 33188) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 320
        Cpanel::CachedDataStore::__ANON__() called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80
        eval {...} called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71
        Try::Tiny::try(CODE(0x344ab70), Try::Tiny::Catch=REF(0x34357d8)) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 329
        Cpanel::CachedDataStore::_write_cache_file("/home/web832/ssl/ssl.db.cache", HASH(0x34bbd38), 33188) called at /usr/local/cpanel/Cpanel/CachedDataStore.pm line 277
        Cpanel::CachedDataStore::loaddatastore("/home/web832/ssl/ssl.db", 0, undef, HASH(0x344a090)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 1223
        Cpanel::SSLStorage::_load_datastore(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 1267
        Cpanel::SSLStorage::_load_datastore_ro(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 848
        Cpanel::SSLStorage::__ANON__() called at /usr/local/cpanel/Cpanel/AccessIds/ReducedPrivileges.pm line 85
        eval {...} called at /usr/local/cpanel/Cpanel/AccessIds/ReducedPrivileges.pm line 85
        Cpanel::AccessIds::ReducedPrivileges::call_as_user(CODE(0x344a300), "web832", "web832") called at /usr/local/cpanel/Cpanel/SSLStorage/User.pm line 676
        Cpanel::SSLStorage::User::_execute_coderef(Cpanel::SSLStorage::User=HASH(0x19d1e40), CODE(0x344a300)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 850
        Cpanel::SSLStorage::_find(Cpanel::SSLStorage::User=HASH(0x19d1e40), "type", "certificate") called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 167
        Cpanel::SSLStorage::find_certificates(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at /usr/local/cpanel/Cpanel/SSLStorage.pm line 269
        Cpanel::SSLStorage::export(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at bin/pkgacct.pl line 2140
        eval {...} called at bin/pkgacct.pl line 2140
        Script::Pkgacct::__ANON__(Cpanel::SSLStorage::User=HASH(0x19d1e40)) called at bin/pkgacct.pl line 2157
        Script::Pkgacct::backup_ssl_for_user("web832", "/1_cpanel_backup/2016-11-16/accounts/web832", Cpanel::Output::Pkgacct=HASH(0x2930e08)) called at bin/pkgacct.pl line 619
        Script::Pkgacct::script("Script::Pkgacct", "web832", "/1_cpanel_backup/2016-11-16/accounts", "backup") called at bin/pkgacct.pl line 134
        Script::Pkgacct::__ANON__() called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80
        eval {...} called at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71
        Try::Tiny::try(CODE(0x290bfe8), Try::Tiny::Catch=REF(0x18c1668)) called at bin/pkgacct.pl line 144
    ); this file will not be saved. at /usr/local/cpanel/Cpanel/Debug.pm line 30, <GEN0> line 1.
     
  6. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,424
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    Pruning is skipped if errors resulting in the failure to backup account an account occurs during the backup generation. This is to avoid
    getting rid of potentially good backups in favor of erroneous ones.

    To clarify, is the entire backup process stopping completely once that error is encountered, or is it only the pruning that fails?

    Thank you.
     
  7. stassy

    stassy Member

    Joined:
    Dec 30, 2004
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    151
    Hi,

    Exactly same problem here. Old backup files/folders are not removed (local and FTP). This problem started after 8th November.
    Code:
    > cat /usr/local/cpanel/version
    11.60.0.24
    
    > cat /var/cpanel/backups/config
    ---
    BACKUPACCTS: 'yes'
    BACKUPBWDATA: 'yes'
    BACKUPDAYS: 0,1,2,3,4,5,6
    BACKUPDIR: /backup
    BACKUPENABLE: 'yes'
    BACKUPFILES: 'yes'
    BACKUPLOGS: 'no'
    BACKUPMOUNT: 'no'
    BACKUPSUSPENDEDACCTS: 'no'
    BACKUPTYPE: compressed
    BACKUP_DAILY_ENABLE: 'yes'
    BACKUP_DAILY_RETENTION: 31
    BACKUP_MONTHLY_DATES: 1
    BACKUP_MONTHLY_ENABLE: 'yes'
    BACKUP_MONTHLY_RETENTION: 4
    BACKUP_WEEKLY_DAY: 1
    BACKUP_WEEKLY_ENABLE: 'yes'
    BACKUP_WEEKLY_RETENTION: 4
    ERRORTHRESHHOLD: 3
    GZIPRSYNCOPTS: --rsyncable
    KEEPLOCAL: 1
    LINKDEST: 0
    LOCALZONESONLY: 'no'
    MAXIMUM_RESTORE_TIMEOUT: 21600
    MAXIMUM_TIMEOUT: 3600
    MYSQLBACKUP: both
    POSTBACKUP: 'no'
    PREBACKUP: -1
    PSQLBACKUP: 'no'
    
    > cd /usr/local/cpanel/logs/cpbackup/
    > grep -i PartialFailure *
    1478826001.log:[2016-11-11 03:13:43 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1478826001.log:[2016-11-11 03:13:44 +0100] info [backup] Sent Backup::PartialFailure notification.
    1478912402.log:[2016-11-12 03:19:30 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1478912402.log:[2016-11-12 03:19:30 +0100] info [backup] Sent Backup::PartialFailure notification.
    1478998801.log:[2016-11-13 03:23:21 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1478998801.log:[2016-11-13 03:23:21 +0100] info [backup] Sent Backup::PartialFailure notification.
    1479085202.log:[2016-11-14 03:30:16 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1479085202.log:[2016-11-14 03:30:16 +0100] info [backup] Sent Backup::PartialFailure notification.
    1479171601.log:[2016-11-15 03:23:13 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1479171601.log:[2016-11-15 03:23:13 +0100] info [backup] Sent Backup::PartialFailure notification.
    1479258001.log:[2016-11-16 03:25:32 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1479258001.log:[2016-11-16 03:25:34 +0100] info [backup] Sent Backup::PartialFailure notification.
    1479344402.log:[2016-11-17 03:23:07 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1479344402.log:[2016-11-17 03:23:07 +0100] info [backup] Sent Backup::PartialFailure notification.
    1479430802.log:[2016-11-18 03:22:33 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1479430802.log:[2016-11-18 03:22:34 +0100] info [backup] Sent Backup::PartialFailure notification.
    1479517201.log:[2016-11-19 03:34:51 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1479517201.log:[2016-11-19 03:34:51 +0100] info [backup] Sent Backup::PartialFailure notification.
    1479603602.log:[2016-11-20 03:36:40 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1479603602.log:[2016-11-20 03:36:40 +0100] info [backup] Sent Backup::PartialFailure notification.
    1479690002.log:[2016-11-21 03:37:51 +0100] info [backup] Final state is Backup::PartialFailure (0)
    1479690002.log:[2016-11-21 03:37:52 +0100] info [backup] Sent Backup::PartialFailure notification.
    
    Any idea ?

    Thanks
     
    #7 stassy, Nov 21, 2016
    Last edited by a moderator: Nov 21, 2016
  8. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,424
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    This suggests a partial backup failure occurred. Could you browse through a specific backup log using the "cat" or "view" command to see if you notice any error or failure messages related to any of the accounts? The log files are stored in the /usr/local/cpanel/logs/cpbackup/ directory.

    Thank you.
     
  9. musti19

    musti19 Well-Known Member

    Joined:
    Jan 20, 2013
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    yes, error on accounts that overquota:
    Code:
    2016-11-21 02:41:32 +0100] Copying SSL certificates, CSRs, and keys...[2016-11-21 02:41:32 +0100] warn [pkgacct] Failed to write the cache file “/home/web1110/ssl/ssl.db.cache” (Cpanel::Exception::IO::CloseError/(XID 67e6mf) „/home/web1110/ssl/ssl.db.cache.tmp.93205435“: Disk quota exceeded
    The system failed to close a file handle for “/home/web1110/ssl/ssl.db.cache.tmp.93205435” because of the following error: Disk quota exceeded
    at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 77, <GEN0> line 1.
    .
    .
    [2016-11-21 02:41:34 +0100] warn [pkgacct] could not acquire a lock for '/home/web1110/cpbackup-exclude.conf': Disk quota exceeded at /usr/local/cpanel/Cpanel/Debug.pm line 30.
        Cpanel::Debug::log_warn("could not acquire a lock for '/home/web1110/cpbackup-exclude."...) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 377
    
    How can this be a reason to no complete backup processes..
     
  10. stassy

    stassy Member

    Joined:
    Dec 30, 2004
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    151
    You were right, I didn't notice when I was looking in these files.

    I've removed a useless database with phpMyAdmin but didn't remove it from the cPanel databases interface.
    And that's why mysqldump failed, it was not able to select the removed database.

    Quite hard for an error. Warning should be better :)

    By the way, thanks again
     
  11. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,424
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Yes, this is a considered a failure and thus backups are not pruned to avoid a situation where the new backup is corrupted or doesn't contain all of the website/account data.

    Check to ensure "Backup Failure" notifications are enabled under "WHM >> Contact Manager". This will ensure you are notified when partial backup failures occur. Also, what's the specific error message you noticed in the backup logs related to the failure?

    Thank you.
     
  12. musti19

    musti19 Well-Known Member

    Joined:
    Jan 20, 2013
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    and what is the solution? Until last update,there was not this problem..
    I cannot go and delete the backups all three days manually,
    because some accounts are overquota.
     
  13. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,424
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello @musti19,

    Could you verify if it's the accounts over quota, or if your entire system is at 100% disk space usage? You can check with a command such as:

    Code:
    df -h
    If it's individual accounts, you could disable the accounts over quota from backup generation via the following option in WHM:

    "WHM Home » Backup » Backup User Selection"

    Then, force a new backup generation via:

    Code:
    /usr/local/cpanel/bin/backup --force
    This should prune the old backups since the system should not encounter backup failures with the over-quota accounts disabled.

    Thank you.
     
  14. stassy

    stassy Member

    Joined:
    Dec 30, 2004
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    151
    "Backup Failure" was not enabled in Notifications, only "Backup Failed To Finish" was. Now, I've changed this to enable both.

    The error message in the log was:
    [2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
    [2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
    /usr/bin/mysqlcheck: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
    [2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump: Got error: 1049: Unknown database 'stagtrip_test' when selecting the database
    [2016-11-21 03:04:21 +0100] stagtrip_test: mysqldump failed -- database may be corrupt
    [2016-11-21 03:04:21 +0100] (787 bytes)
    [2016-11-21 03:04:21 +0100] Failed to dump database stagtrip_test: The subprocess reported error number 2 when it ended.
    [2016-11-21 03:04:21 +0100]
    [2016-11-21 03:04:21 +0100] WARNING: There was a problem dumping one or more databases

    Message is clear enough, but as I didn't receive any emails...

    And this stagtrip_test database was removed by me with phpMyAdmin and not with the cPanel interface;

    Thanks
     
    #14 stassy, Nov 22, 2016
    Last edited: Nov 22, 2016
  15. musti19

    musti19 Well-Known Member

    Joined:
    Jan 20, 2013
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    /dev/md2 1.8T 1.1T 625G 64% /
    i checked some accounts and it is right, that they are overquota

    should i now check every day, if user is overquota and disable it?
     
  16. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,424
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    @stassy - Keep in mind that it's always advised to remove databases via cPanel or via an API function. Manually removing databases via the command line or phpMyAdmin doesn't disassociate the database from the cPanel account.

    @musti19 - I recommend disabling backups for the accounts over quota first just to make sure that's the actual reason why pruning is failing. It's possible there's another issue you haven't noticed that's preventing the backup from completing successfully. Also, regarding the disk space issue, the following options under the "Notifications" tab in "WHM >> Tweak Settings" are useful for this purpose:

    Account disk quota “critical” percentage
    Notify admin or reseller when disk quota reaches “critical” state

    This will ensure you are notified when accounts reach a critical disk space usage percentage.

    Thank you.
     
  17. mccreare

    mccreare Member

    Joined:
    Apr 23, 2003
    Messages:
    19
    Likes Received:
    1
    Trophy Points:
    153
    I'm having this problem too. Over quota accounts generate a Partial Failed error, then those backups are not removed, stalling the whole system. I can't give my customer's free space, and not backing up accounts isn't an option either. This doesn't work.
     
  18. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,658
    Likes Received:
    1,424
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello @mccreare,

    cPanel accounts are not designed to function properly when at their disk space limit. This can lead to potential issues with services such as email. If the account has reached it's disk space limit, it's already in a state where it won't complete a full backup generation. Thus, in those cases, you should exclude it from the backups via "WHM Home » Backup » Backup User Selection".

    Note that if the backup system were to prune backups when failures or partial failures occur, then it would create a situation where a previous valid backup is pruned, and no new backup is created, leaving the account with no usable backup archives. What you are seeking is new functionality, such as an option to temporarily increase an account's quota when backups are generated. I recommend submitting a feature request for the functionality you'd like to see via:

    Submit A Feature Request

    Let us know if you have any additional questions or concerns.

    Thank you.
     
  19. mccreare

    mccreare Member

    Joined:
    Apr 23, 2003
    Messages:
    19
    Likes Received:
    1
    Trophy Points:
    153
    No...
    1. Shared hosts have to deal with customers going over quota all the time. So, it's not like there's anything we can do about that. I'm not going to beg every customer to get their act in gear each time they let their space go over so we can have functional backups again. With thousands of customers, that's absurd. We also offer backups to everyone, including anyone over quota, so these accounts need to be backed up anyway. There is no reason at all this should fail to work. Just back up the account properly with no failures, whether or not it is "over quota". Problem solved.
    2. The system worked fine up until this most recent update, so I'm definitely not asking for a new feature. I'm asking you to un-break the system that was not broken until very recently.
     
    musti19 likes this.
  20. NanoGame

    NanoGame Registered

    Joined:
    Oct 6, 2016
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Russia
    cPanel Access Level:
    Root Administrator
    Hello.

    I have same problem. Please fix it.

    [2016-12-04 21:48:33 +0300] info [backup] Final state is Backup::PartialFailure (0)
    [2016-12-04 21:48:33 +0300] info [backup] Sent Backup::PartialFailure notification.

    [2016-12-04 02:04:59 +0300] Copying SSL certificates, CSRs, and keys...[2016-12-04 02:04:59 +0300] warn [pkgacct] Failed to write the cache file “/home/delaemva/ssl/ssl.db.cache” (Cpanel::Exception::IO::CloseError/(XID 7bnwvg) Системе не удалось закрыть дескриптор файла для «/home/delaemva/ssl/ssl.db.cache.tmp.39861624» из-за следующей ошибки: Disk quota exceeded
    The system failed to close a file handle for “/home/delaemva/ssl/ssl.db.cache.tmp.39861624” because of the following error: Disk quota exceeded
     
Loading...

Share This Page