backup_incomplete file, not deleted old backups

postcd

Well-Known Member
Oct 22, 2010
721
20
68
find /backup -name backup_incomplete|sort -h
./2017-10-02/backup_incomplete
./2017-10-03/backup_incomplete
./2017-10-04/backup_incomplete
./2017-10-06/backup_incomplete
./2017-10-07/backup_incomplete
./2017-10-09/backup_incomplete
./2017-10-10/backup_incomplete
./2017-10-11/backup_incomplete
./2017-10-12/backup_incomplete
./2017-10-13/backup_incomplete
./2017-10-14/backup_incomplete
./2017-10-16/backup_incomplete
./2017-10-17/backup_incomplete
./2017-10-18/backup_incomplete
./monthly/2017-10-01/backup_incomplete
./weekly/2017-10-08/backup_incomplete
./weekly/2017-10-15/backup_incomplete
Hello, these files are empty, but makes me wonder what is wrong. At same time i have set Retention 5 in Daily backups section of the WHM/Backup configuration. So it is strange older backups are not deleted per above output. The "Strictly enforce retention, regardless of backup success." is not ticked.

Update: temporarily until this is solved, i setup a cronjob to delete all daily backups in /backup except 7-1=6 last/newest backups:
50 22 * * * root for dailybackupdir in $(ls -t -A1 /backup|grep -|tail -n +7);do /usr/bin/ionice -c2 -n7 find /backup/"$dailybackupdir" -delete;done
 
Last edited:

rpvw

Well-Known Member
Jul 18, 2013
1,101
466
113
UK
cPanel Access Level
Root Administrator
The docs suggest that the pruning of backups using the Strictly enforce retention, regardless of backup success switch will only happen after at least 1 (one) completed successful backup
Each Strictly enforce retention, regardless of backup success setting overrides the standard behavior for that backup type. If you select this option, the system will always retain only the desired number of backups with at least one complete backup.
Backup Configuration - Documentation - cPanel Documentation
 

cPWilliamL

cP Technical Analyst II
Staff member
May 15, 2017
258
30
103
America
cPanel Access Level
Root Administrator
@postcd, That cron seems a bit dangerous to me. If, for whatever reason, '$dailybackupdir' isn't set, you'll delete all your backups. You may consider setting nounset:
BashFAQ - Greg's Wiki

Really, the entire forloop could be replaced by one find command.

Regardless, I would recommend investigating why your backups are failing. You can use the oneliner below to check all backup logs for all errors that would lead to an incomplete backup(paste the entire command at once):
Code:
for log in /usr/local/cpanel/logs/cpbackup/*; do printf -- "------------------------------\n%s:\n" "$log"; while read err; do grep "$err" "$log"; done <<EOF;done
Unable to get user id
Unable to load cPanel user data
You cannot copy the root user
pkgacct failed to copy daily backup
Could not use daily backup
Bailing out
The REMOTE_PASSWORD variable is missing
Unable to find domain name
Exiting with error code
Could not remove directory
Hook denied execution of pkgacct
Could not open
Could not chmod
Could not rename
failed to create the working dir
Unable to fork
Unable to waitpid
Unable to open
Failure dumping
Unable to read
does not appear to be valid XML
Could not create directory
mysqldump: Got error
mysqldump failed
EOF
 

postcd

Well-Known Member
Oct 22, 2010
721
20
68
Backups was not created, i run above mentioned script to find errors in backup logs and this was found:
[2017-11-01 03:28:01 +0000] Bailing out.. you must set a valid destination for backups
... many same errors ...
[2017-11-01 03:28:01 +0000] Bailing out.. you must set a valid destination for backups
[2017-11-01 03:28:02 +0000] warn [backup] Unable to open “/backup/monthly”: No such file or directory at /usr/local/cpanel/bin/backup line 1856.
In WHM / Backup / Backup Configuration, the "Default Backup Directory" is set to "/backup".

The directory exist, but is empty:
Code:
# ls -la /backup; stat /backup
total 8
drwx--x--x  2 root root 4096 Nov  1 02:00 ./
drwxr-xr-x 32 root root 4096 Oct 31 10:40 ../
  File: `/backup'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: f1h/241d        Inode: 41829777    Links: 2
Access: (0711/drwx--x--x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-11-01 03:28:02.426860500 +0000
Modify: 2017-11-01 02:00:06.000000000 +0000
Change: 2017-11-01 03:22:49.607780815 +0000
I ran the backup manually from command line: /usr/local/cpanel/bin/backup --force
and it seems to finished and files are there in /backup and no "backup_incomplete" file.
[2017-11-01 10:21:55 +0000] info [backup] Completed at Wed Nov 1 10:21:55 2017
[2017-11-01 10:21:56 +0000] info [backup] Final state is Backup::Success (0)
[2017-11-01 10:21:56 +0000] info [backup] Sent Backup::Success notification.

a few days passed and backups are now created automatically without issue. Do not know why it was nto working.
 

veronicabend

Well-Known Member
Feb 25, 2005
76
2
158
I am experiencing the issue of old backups not being removed. There is a backup_incomplete file in the backup folder. It seems to create the backups for all accounts. I am not detecting why it would not delete the older backups. I run the command below, it just printed the log file names and the dashes line under each, nothing else.

I need to detect what is causing this, I am now having to remember to manually go and delete older backups periodically.

Thanks.

@postcd, That cron seems a bit dangerous to me. If, for whatever reason, '$dailybackupdir' isn't set, you'll delete all your backups. You may consider setting nounset:
BashFAQ - Greg's Wiki

Really, the entire forloop could be replaced by one find command.

Regardless, I would recommend investigating why your backups are failing. You can use the oneliner below to check all backup logs for all errors that would lead to an incomplete backup(paste the entire command at once):
Code:
for log in /usr/local/cpanel/logs/cpbackup/*; do printf -- "------------------------------\n%s:\n" "$log"; while read err; do grep "$err" "$log"; done <<EOF;done
Unable to get user id
Unable to load cPanel user data
You cannot copy the root user
pkgacct failed to copy daily backup
Could not use daily backup
Bailing out
The REMOTE_PASSWORD variable is missing
Unable to find domain name
Exiting with error code
Could not remove directory
Hook denied execution of pkgacct
Could not open
Could not chmod
Could not rename
failed to create the working dir
Unable to fork
Unable to waitpid
Unable to open
Failure dumping
Unable to read
does not appear to be valid XML
Could not create directory
mysqldump: Got error
mysqldump failed
EOF
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
9,791
1,529
313
cPanel Access Level
Root Administrator
That one-liner scans for some of the most common errors, but it can't catch everything. I'd recommend checking the backup logs directly in /usr/local/cpanel/logs/cpbackup/ to see if you can find the reason the backup is not completing properly. If the current backup fails, the older one will not be pruned, leading to the behavior you're seeing.
 

veronicabend

Well-Known Member
Feb 25, 2005
76
2
158
Thanks . I have already looked and have not detected anything. The log file is quite long. There are several accounts. I searched for "Error" or "error" and found nothing. I manually scanned some rows but everything looks the same. I don't know what else to look.

I see this near the end of the file when it is finishing:

Code:
[2021-07-14 03:17:52 +0000] warn [backup] Pruning of backup files skipped due to errors. at /usr/local/cpanel/bin/backup line 395.
        bin::backup::run("bin::backup") called at /usr/local/cpanel/bin/backup line 120
That one-liner scans for some of the most common errors, but it can't catch everything. I'd recommend checking the backup logs directly in /usr/local/cpanel/logs/cpbackup/ to see if you can find the reason the backup is not completing properly. If the current backup fails, the older one will not be pruned, leading to the behavior you're seeing.
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
9,791
1,529
313
cPanel Access Level
Root Administrator
Sometimes the errors are difficult to track down for sure. I've definitely had backup issues where the log wasn't as helpful as I wanted.

It might be best to submit a ticket to our team directly so we can take a look at the machine and track down what is happening. If you're able to do that, please post the ticket number here so I can follow along and keep this thread updated.
 

veronicabend

Well-Known Member
Feb 25, 2005
76
2
158
I tried to create a Support Ticket from inside WHM, but it is asking for a cPanel ID and I don't know which one it needs.

Sometimes the errors are difficult to track down for sure. I've definitely had backup issues where the log wasn't as helpful as I wanted.

It might be best to submit a ticket to our team directly so we can take a look at the machine and track down what is happening. If you're able to do that, please post the ticket number here so I can follow along and keep this thread updated.
 

veronicabend

Well-Known Member
Feb 25, 2005
76
2
158
I don't access the cPanel store. I have a dedicated server with cpanel license.
I will try sending the email.

The ID would be what you use to access the cPanel Store at cPanel Store - Cart. If you're not sure what that is, send a message to [email protected], which will also start the ticket process as well, as the Customer Service team can then move that over to our Technical Support department.