backup_incomplete file, not deleted old backups

[postcd]

find /backup -name backup_incomplete|sort -h

./2017-10-02/backup_incomplete
./2017-10-03/backup_incomplete
./2017-10-04/backup_incomplete
./2017-10-06/backup_incomplete
./2017-10-07/backup_incomplete
./2017-10-09/backup_incomplete
./2017-10-10/backup_incomplete
./2017-10-11/backup_incomplete
./2017-10-12/backup_incomplete
./2017-10-13/backup_incomplete
./2017-10-14/backup_incomplete
./2017-10-16/backup_incomplete
./2017-10-17/backup_incomplete
./2017-10-18/backup_incomplete
./monthly/2017-10-01/backup_incomplete
./weekly/2017-10-08/backup_incomplete
./weekly/2017-10-15/backup_incomplete

Hello, these files are empty, but makes me wonder what is wrong. At same time i have set Retention 5 in Daily backups section of the WHM/Backup configuration. So it is strange older backups are not deleted per above output. The "Strictly enforce retention, regardless of backup success." is not ticked.

Update: temporarily until this is solved, i setup a cronjob to delete all daily backups in /backup except 7-1=6 last/newest backups:
50 22 * * * root for dailybackupdir in $(ls -t -A1 /backup|grep -|tail -n +7);do /usr/bin/ionice -c2 -n7 find /backup/"$dailybackupdir" -delete;done

 

[rpvw]

The docs suggest that the pruning of backups using the Strictly enforce retention, regardless of backup success switch will only happen after at least 1 (one) completed successful backup

Each Strictly enforce retention, regardless of backup success setting overrides the standard behavior for that backup type. If you select this option, the system will always retain only the desired number of backups with at least one complete backup.

Backup Configuration - Documentation - cPanel Documentation

 

[cPWilliamL]

@postcd, That cron seems a bit dangerous to me. If, for whatever reason, '$dailybackupdir' isn't set, you'll delete all your backups. You may consider setting nounset:
BashFAQ - Greg's Wiki

Really, the entire forloop could be replaced by one find command.

Regardless, I would recommend investigating why your backups are failing. You can use the oneliner below to check all backup logs for all errors that would lead to an incomplete backup(paste the entire command at once):

for log in /usr/local/cpanel/logs/cpbackup/*; do printf -- "------------------------------\n%s:\n" "$log"; while read err; do grep "$err" "$log"; done <<EOF;done
Unable to get user id
Unable to load cPanel user data
You cannot copy the root user
pkgacct failed to copy daily backup
Could not use daily backup
Bailing out
The REMOTE_PASSWORD variable is missing
Unable to find domain name
Exiting with error code
Could not remove directory
Hook denied execution of pkgacct
Could not open
Could not chmod
Could not rename
failed to create the working dir
Unable to fork
Unable to waitpid
Unable to open
Failure dumping
Unable to read
does not appear to be valid XML
Could not create directory
mysqldump: Got error
mysqldump failed
EOF

 

[postcd]

Backups was not created, i run above mentioned script to find errors in backup logs and this was found:

[2017-11-01 03:28:01 +0000] Bailing out.. you must set a valid destination for backups
... many same errors ...
[2017-11-01 03:28:01 +0000] Bailing out.. you must set a valid destination for backups
[2017-11-01 03:28:02 +0000] warn [backup] Unable to open “/backup/monthly”: No such file or directory at /usr/local/cpanel/bin/backup line 1856.

In WHM / Backup / Backup Configuration, the "Default Backup Directory" is set to "/backup".

The directory exist, but is empty:

# ls -la /backup; stat /backup
total 8
drwx--x--x  2 root root 4096 Nov  1 02:00 ./
drwxr-xr-x 32 root root 4096 Oct 31 10:40 ../
  File: `/backup'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: f1h/241d        Inode: 41829777    Links: 2
Access: (0711/drwx--x--x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-11-01 03:28:02.426860500 +0000
Modify: 2017-11-01 02:00:06.000000000 +0000
Change: 2017-11-01 03:22:49.607780815 +0000

I ran the backup manually from command line: /usr/local/cpanel/bin/backup --force
and it seems to finished and files are there in /backup and no "backup_incomplete" file.
[2017-11-01 10:21:55 +0000] info [backup] Completed at Wed Nov 1 10:21:55 2017
[2017-11-01 10:21:56 +0000] info [backup] Final state is Backup::Success (0)
[2017-11-01 10:21:56 +0000] info [backup] Sent Backup::Success notification.

a few days passed and backups are now created automatically without issue. Do not know why it was nto working.

 

[veronicabend]

I am experiencing the issue of old backups not being removed. There is a backup_incomplete file in the backup folder. It seems to create the backups for all accounts. I am not detecting why it would not delete the older backups. I run the command below, it just printed the log file names and the dashes line under each, nothing else.

I need to detect what is causing this, I am now having to remember to manually go and delete older backups periodically.

Thanks.

@postcd, That cron seems a bit dangerous to me. If, for whatever reason, '$dailybackupdir' isn't set, you'll delete all your backups. You may consider setting nounset:
BashFAQ - Greg's Wiki

Really, the entire forloop could be replaced by one find command.

Regardless, I would recommend investigating why your backups are failing. You can use the oneliner below to check all backup logs for all errors that would lead to an incomplete backup(paste the entire command at once):

for log in /usr/local/cpanel/logs/cpbackup/*; do printf -- "------------------------------\n%s:\n" "$log"; while read err; do grep "$err" "$log"; done <<EOF;done
Unable to get user id
Unable to load cPanel user data
You cannot copy the root user
pkgacct failed to copy daily backup
Could not use daily backup
Bailing out
The REMOTE_PASSWORD variable is missing
Unable to find domain name
Exiting with error code
Could not remove directory
Hook denied execution of pkgacct
Could not open
Could not chmod
Could not rename
failed to create the working dir
Unable to fork
Unable to waitpid
Unable to open
Failure dumping
Unable to read
does not appear to be valid XML
Could not create directory
mysqldump: Got error
mysqldump failed
EOF

 

[cPRex]

That one-liner scans for some of the most common errors, but it can't catch everything. I'd recommend checking the backup logs directly in /usr/local/cpanel/logs/cpbackup/ to see if you can find the reason the backup is not completing properly. If the current backup fails, the older one will not be pruned, leading to the behavior you're seeing.

 

[veronicabend]

Thanks . I have already looked and have not detected anything. The log file is quite long. There are several accounts. I searched for "Error" or "error" and found nothing. I manually scanned some rows but everything looks the same. I don't know what else to look.

I see this near the end of the file when it is finishing:

[2021-07-14 03:17:52 +0000] warn [backup] Pruning of backup files skipped due to errors. at /usr/local/cpanel/bin/backup line 395.
        bin::backup::run("bin::backup") called at /usr/local/cpanel/bin/backup line 120

That one-liner scans for some of the most common errors, but it can't catch everything. I'd recommend checking the backup logs directly in /usr/local/cpanel/logs/cpbackup/ to see if you can find the reason the backup is not completing properly. If the current backup fails, the older one will not be pruned, leading to the behavior you're seeing.

 

[cPRex]

Sometimes the errors are difficult to track down for sure. I've definitely had backup issues where the log wasn't as helpful as I wanted.

It might be best to submit a ticket to our team directly so we can take a look at the machine and track down what is happening. If you're able to do that, please post the ticket number here so I can follow along and keep this thread updated.

 

[veronicabend]

I tried to create a Support Ticket from inside WHM, but it is asking for a cPanel ID and I don't know which one it needs.

Sometimes the errors are difficult to track down for sure. I've definitely had backup issues where the log wasn't as helpful as I wanted.

It might be best to submit a ticket to our team directly so we can take a look at the machine and track down what is happening. If you're able to do that, please post the ticket number here so I can follow along and keep this thread updated.

 

[cPRex]

The ID would be what you use to access the cPanel Store at cPanel Store - Cart. If you're not sure what that is, send a message to [email protected], which will also start the ticket process as well, as the Customer Service team can then move that over to our Technical Support department.

 

[veronicabend]

I don't access the cPanel store. I have a dedicated server with cpanel license.
I will try sending the email.

The ID would be what you use to access the cPanel Store at cPanel Store - Cart. If you're not sure what that is, send a message to [email protected], which will also start the ticket process as well, as the Customer Service team can then move that over to our Technical Support department.

 

[cPRex]

Sounds good!

 

[veronicabend]

I got assigned Inquiry ID 94343925

Sounds good!

 

[cPRex]

Thanks for that - I'm following along with that ticket on my end now as well.

 

[veronicabend]

Thank you.

Thanks for that - I'm following along with that ticket on my end now as well.

 

[cPRex]

I see our team was able to find an issue with a database that could not be dumped properly, leading to the backup failure.