Backup partial failure always

[sunmacet]

We have multiple servers that has weekly incremental backup scheduled from cPanel.

However these backups almost always (9 out of 10 times) finishes with backup partial failure. Usually there is 20-100 accounts missing from the backup.

No errors are found from backup logs about the missing accounts.

The only clue :

[2020-11-27 15:29:47 +0200] info [backup] checking backup for user2

After that there is nothing more about user2. Backup continues to next item but the backup for user2 will be missing. Exactly same is seen for all the missing backups.

/usr/local/cpanel/logs/cpbackup/1606435202.log

[2020-11-27 15:29:46 +0200] info [backup] Successfully backed up account ?user1? to ?/backup/weekly/2020-11-27/accounts?
[2020-11-27 15:29:46 +0200] info [backup] Adding metadata information for user1 to backup at /backup/weekly/2020-11-27
[2020-11-27 15:29:46 +0200] info [backup] Processing an incremental type backup for user1.
[2020-11-27 15:29:46 +0200] info [backup] Queuing weekly backup copy of ?user1? for transport of ?/backup/weekly/2020-11-27/accounts/user1? to ?weekly/2020-11-27/accounts/user1?
[2020-11-27 15:29:46 +0200] info [backup] This particular transport will be queued with keep_local = 1 , based on the need to copy weekly (1) and/or monthly () copies as well.
[2020-11-27 15:29:46 +0200] info [backup] Queuing transport of file: /backup/weekly/2020-11-27/accounts/user1
[2020-11-27 15:29:46 +0200] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:288218
[2020-11-27 15:29:47 +0200] info [backup] leaving queue_backup_transport_item
[2020-11-27 15:29:47 +0200] info [backup] checking backup for user2
[2020-11-27 15:29:47 +0200] info [backup] Queuing transport of meta file: /backup/weekly/2020-11-27/accounts/.master.meta
[2020-11-27 15:29:47 +0200] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:288219
[2020-11-27 15:29:47 +0200] info [backup] leaving queue_backup_transport_item
[2020-11-27 15:29:50 +0200] info [backup] Queuing transport of file: /backup/weekly/2020-11-27/backup_incomplete
[2020-11-27 15:29:50 +0200] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:288220
[2020-11-27 15:29:50 +0200] info [backup] leaving queue_backup_transport_item
[2020-11-27 15:29:50 +0200] info [backup] Pruning backup directory: /backup/weekly/2020-11-20
[2020-11-27 20:13:49 +0200] info [backup] Queuing prune operation for remote destination weekly backups
[2020-11-27 20:13:49 +0200] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:288221
[2020-11-27 20:13:50 +0200] info [backup] leaving queue_backup_transport_item
[2020-11-27 20:13:53 +0200] info [backup] Scheduling backup metadata vacuum
[2020-11-27 20:13:53 +0200] info [backup] Queuing transport reporter
[2020-11-27 20:13:53 +0200] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:288222
[2020-11-27 20:13:53 +0200] info [backup] leaving queue_backup_transport_item
[2020-11-27 20:13:53 +0200] info [backup] Completed at Fri Nov 27 20:13:53 2020
[2020-11-27 20:13:56 +0200] info [backup] Final state is Backup::PartialFailure (0)
[2020-11-27 20:13:56 +0200] info [backup] Sent Backup::PartialFailure notification.

The /var/log/messages and /usr/local/cpanel/logs/error_log has nothing relevant at the given timestamps.

Any idea what could cause this?

 

[andrew.n]

Can you attach the full log? we might be more clever based on that.

 

[sunmacet]

Can you attach the full log? we might be more clever based on that.

It would be more than 100 000 lines. And there is nothing to see. Full of completed backups (and therefore somewhat sensitive data) and then the missing accounts that has only the line "info [backup] checking backup for ..." and nothing more.

The .master.meta do not have the missing accounts. The /var/log/messages has nothing about the issue and also the cpanel error_log has only the lookup_mail_user() lines as usual.

I am asking if someone else has encountered this? Or has any idea where to look for more information other than cPanel backup log, /var/log/messages and cpanel error_log?

 

[cPRex]

The backup log is the best place to look for more details. I like to search for "error" or "warn" in the log to see if anything sticks out, but with such a large log file and you not finding anything obvious after you've already checked, this might be better handled through a ticket so our team can check things directly. If you do decide to submit a ticket, please post the number here so I can update the community with the results of our findings.