Pruning of backups from failed transports

[raudabaugh]

I have a daily backup that does an sftp of files to a remote sftp site. Past behaviour would retain any backup files that failed during the transport. Recently, the cpbackup process has changed and has been pruning all of the backup files, regardless of the success of the transport.

I'd like to revert back to the old process of not deleting backup files after a failed transport. It allows me to followup with these failed transports and finish the transfer. How can I do this without asking the server to retain all of the files regardless of backup success?

 

[cPanelLauren]

Hi @raudabaugh

This behavior is a result of the following setting being enabled at WHM>>Backup>>Backup Configuration ->
Strictly enforce retention, regardless of backup success.

For more information, read our Backup Configuration documentation.

 

[raudabaugh]

Just to be clear, until recently, the behaviour was completely different. I've always have had this enforce retention setting OFF, and the system would only delete those files that had successfully transferred to the remote destination. What happened?

Think about it. This is a valuable function. I'd prefer that any files that have been transported successfully to be removed from the production server. This is the way it worked for years, until recently.

 

[cPanelLauren]

Hi @raudabaugh


This setting defaults to off, if it was enabled it means someone had to have enabled it in the UI or through the backup configuration.

 

[raudabaugh]

Sorry, I'm not being clear enough.

The problem is not really an issue with the 'Enforce Retention" feature of the cpbackup. In fact, I've never set the switch to the 'on' position for a very good reason. My system does not have enough space to be retaining backups for any extended period of time. Hence the reason that i setup a SFTP transport process to move the daily backups to an 'off site' system that has the disk space I need.

HERE is the problem:

For years and years, the CPBACKUP system would only delete the backup files that were successfully transported to my SFTP site. It would retain any files that failed. This is a huge advantage where I could 'clean up' failed transports without regenerating. What happen to this behaviour? This is something the seemed to go away after the most recent whm/cpanel upgrades.

 

[cPanelLauren]

HI @raudabaugh

For years and years, the CPBACKUP system would only delete the backup files that were successfully transported to my SFTP site. It would retain any files that failed. This is a huge advantage where I could 'clean up' failed transports without regenerating. What happen to this behaviour? This is something the seemed to go away after the most recent whm/cpanel upgrades.

This is how it should be behaving still without "Enforce Retention" enabled. The backup logs should indicate what happened with the failed backups specifically. You can find them at /usr/local/cpanel/logs/cpbackup/

 

[raudabaugh]

Here is the email notification:

Transport errors encountered.

The system encountered errors during transport of the backup files.

Transport Backup Label Message
adrivesftp someusr Unable to send “/backup/2019-03-21/accounts/someusr.tar.gz” to destination “adrivesftp”
adrivesftp someusrtoo Unable to send “/backup/2019-03-21/accounts/someusrtoo.tar.gz” to destination “adrivesftp”


Below is a preview of the attached log file.

Preview of transport errors log:
Unable to send “/backup/2019-03-21/accounts/someusrtoo.tar.gz” to destination “adrivesftp”
Unable to send “/backup/2019-03-21/accounts/someusr.tar.gz” to destination “adrivesftp”

This notice is the result of a request from “cpbackup”.
The system generated this notice on Thursday, March 21, 2019 at 8:17:23 AM UTC.

Here is the cpbackup log entry for someusrtoo.tar.gz:

[2019-03-21 02:00:41 -0400] info [backup] Running account backup with target : /backup/2019-0
3-21/accounts
[2019-03-21 02:00:41 -0400] info [backup] checking backup for someusrtoo
[2019-03-21 02:00:41 -0400] info [backup] Backups ARE enabled for someusrtoo
[2019-03-21 02:00:41 -0400] info [backup] Calling pkgacct under cpuwatch to backup user “tavq
uote”
[2019-03-21 02:00:42 -0400] pkgacct started.
[2019-03-21 02:00:42 -0400] pkgacct version 10 - user : someusrtoo - tarball: 1 - target mysql
: default - split: 0 - incremental: 0 - homedir: 1 - mailman: 1 - backup: 1 - archive version
: 4 - running with uid 0
[2019-03-21 02:00:42 -0400] pkgacct using '/usr/local/cpanel/3rdparty/bin/pigz -6 --processes
1 --blocksize 4096 --rsyncable' to compress archives
[2019-03-21 02:00:42 -0400] pkgacct working dir : /backup/2019-03-21/accounts/someusrtoo
[2019-03-21 02:00:42 -0400] Copying Reseller Config...[2019-03-21 02:00:42 -0400] Done
[2019-03-21 02:00:42 -0400] Copying Suspension Info (if needed)...[2019-03-21 02:00:42 -0400]
Done
[2019-03-21 02:00:42 -0400] Copying installed SSL certificates and keys...[2019-03-21 02:00:4
2 -0400] Performing “ApacheTLS” component....
[2019-03-21 02:00:42 -0400] Completed “ApacheTLS” component.
[2019-03-21 02:00:42 -0400] Done
[2019-03-21 02:00:42 -0400] Copying Domain Keys....[2019-03-21 02:00:42 -0400] Done
[2019-03-21 02:00:42 -0400] Copying Counter Data....[2019-03-21 02:00:42 -0400] Done
[2019-03-21 02:00:42 -0400] Copying Bandwidth Data....[2019-03-21 02:00:42 -0400] Performing
“Bandwidth” component....
Summary databases … done!
[2019-03-21 02:00:42 -0400] Completed “Bandwidth” component.
[2019-03-21 02:00:42 -0400] Done
[2019-03-21 02:00:42 -0400] Copying Dns Zones.......someusrtoos.com...[2019-03-21 02:00:42 -040
0] Done
[2019-03-21 02:00:42 -0400] Copying Mail files....[2019-03-21 02:00:42 -0400] Done
[2019-03-21 02:00:42 -0400] Copying proftpd file....[2019-03-21 02:00:42 -0400] Done
[2019-03-21 02:00:42 -0400] Performing “Logs” component....
...log file sizes [3637 byte(s)]......someusrtoos.com-bytes_log......someusrtoos.com......tavquot
es.com-ssl_log...[2019-03-21 02:00:42 -0400] Completed “Logs” component.
[2019-03-21 02:00:42 -0400] Copy userdata...[2019-03-21 02:00:42 -0400] Done
[2019-03-21 02:00:42 -0400] Copy custom virtualhost templates...[2019-03-21 02:00:42 -0400] D
one
[2019-03-21 02:00:42 -0400] Copying mailman lists and archives....Done copying mailman lists
and archives.
[2019-03-21 02:00:42 -0400] Copying homedir.............
[2019-03-21 02:00:46 -0400] Done
[2019-03-21 02:00:46 -0400] Fixing up EA4 .htaccess blocks: /home/someusrtoo/public_html/.htacc
ess Done.
[2019-03-21 02:00:46 -0400] Performing “Postgresql” component....
[2019-03-21 02:00:46 -0400] Completed “Postgresql” component.
[2019-03-21 02:00:46 -0400] Performing “Mysql” component....
Load watching suspended due to SIGUSR1
[2019-03-21 02:00:46 -0400] Determining mysql dbs...[2019-03-21 02:00:46 -0400] ...mysqldump
version: 10.0.38...[2019-03-21 02:00:46 -0400] ...mysql version: 10.0...[2019-03-21 02:00:46
-0400] Saving mysql privs...[2019-03-21 02:00:46 -0400] Done
[2019-03-21 02:00:46 -0400] ...Done
[2019-03-21 02:00:46 -0400] Storing mysql dbs............
someusrtoo_db[2019-03-21 02:00:46 -0400] (34195 bytes)
[2019-03-21 02:00:46 -0400] ...Done
[2019-03-21 02:00:46 -0400] Completed “Mysql” component.
[2019-03-21 02:00:46 -0400] Performing “MysqlRemoteNotes” component....
[2019-03-21 02:00:46 -0400] Completed “MysqlRemoteNotes” component.
[2019-03-21 02:00:46 -0400] Copying cpuser file.......Load watching resumed due to SIGUSR2
[2019-03-21 02:00:46 -0400] Done
[2019-03-21 02:00:46 -0400] Copying crontab file.......[2019-03-21 02:00:46 -0400] Done
[2019-03-21 02:00:46 -0400] Performing “Quota” component....
[2019-03-21 02:00:46 -0400] Completed “Quota” component.
[2019-03-21 02:00:46 -0400] Performing “Integration” component....
[2019-03-21 02:00:46 -0400] Completed “Integration” component.
[2019-03-21 02:00:46 -0400] Performing “AuthnLinks” component....
[2019-03-21 02:00:46 -0400] Completed “AuthnLinks” component.
[2019-03-21 02:00:46 -0400] Performing “APITokens” component....
[2019-03-21 02:00:46 -0400] Completed “APITokens” component.
[2019-03-21 02:00:46 -0400] Performing “AutoSSL” component....
[2019-03-21 02:00:46 -0400] Completed “AutoSSL” component.
[2019-03-21 02:00:46 -0400] Storing Subdomains....
[2019-03-21 02:00:46 -0400] Done
[2019-03-21 02:00:46 -0400] Storing Parked Domains....
[2019-03-21 02:00:46 -0400] Done
[2019-03-21 02:00:46 -0400] Storing Addon Domains....
[2019-03-21 02:00:46 -0400] Performing “Password” component....
[2019-03-21 02:00:46 -0400] Completed “Password” component.
[2019-03-21 02:00:46 -0400] Performing “DigestShadow” component....
[2019-03-21 02:00:46 -0400] Completed “DigestShadow” component.
[2019-03-21 02:00:46 -0400] Copying shell.......[2019-03-21 02:00:46 -0400] Done
[2019-03-21 02:00:46 -0400] Performing “PublicContact” component....
[2019-03-21 02:00:46 -0400] Completed “PublicContact” component.
[2019-03-21 02:00:46 -0400] Performing “MailLimits” component....
[2019-03-21 02:00:46 -0400] Completed “MailLimits” component.
[2019-03-21 02:00:46 -0400] Creating Archive ........................
[2019-03-21 02:01:28 -0400] Done
[2019-03-21 02:01:28 -0400] pkgacctfile is: /backup/2019-03-21/accounts/someusrtoo.tar.gz
[2019-03-21 02:01:28 -0400]
[2019-03-21 02:01:28 -0400] size is: 50024317
[2019-03-21 02:01:28 -0400]
[2019-03-21 02:01:28 -0400] homesize is: 343334912
[2019-03-21 02:01:28 -0400]
[2019-03-21 02:01:28 -0400] homefiles is: 38626
[2019-03-21 02:01:28 -0400] pkgacct completed
[2019-03-21 02:01:28 -0400] info [backup] Successfully backed up account “someusrtoo” to “/back
up/2019-03-21/accounts”
[2019-03-21 02:01:28 -0400] info [backup] Adding metadata information for someusrtoo to backup
at /backup/2019-03-21
[2019-03-21 02:01:28 -0400] info [backup] Queuing daily backup copy of “someusrtoo” for transpo
rt of “/backup/2019-03-21/accounts/someusrtoo.tar.gz” to “2019-03-21/accounts/someusrtoo.tar.gz”
[2019-03-21 02:01:28 -0400] info [backup] This particular transport will be queued with keep_
local = 0 , based on the need to copy weekly () and/or monthly () copies as well.
[2019-03-21 02:01:28 -0400] info [backup] Queuing transport of file: /backup/2019-03-21/accou
nts/someusrtoo.tar.gz
[2019-03-21 02:01:28 -0400] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:527
[2019-03-21 02:01:28 -0400] info [backup] leaving queue_backup_transport_item

cpbackup_transporter.log did not have specific reference to someusrtoo.tar.gz, but there were transport errors from failed transport attempts.

Here is a copy of the server backup directory:

[email protected] [/backup]# ls -l
total 68
drwx--x--x 3 root root 4096 Mar 21 04:17 ./
drwxr-xr-x. 22 root root 4096 Feb 20 11:22 ../
-rw------- 1 root root 57344 Mar 21 04:17 transports.db
drwx--x--x 3 root root 4096 Mar 17 02:00 weekly/

 

[cPanelLauren]

Hi @raudabaugh

Interesting and the transport log didn't mention anything about pruning either? All transports should be logged if they were successful but in this instance, it seems like there was some failure that caused it to error out unexpectedly. At this point, it might be best to open a ticket which you can do using the link in my signature. Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


Thanks!

 

[raudabaugh]

11771623 - Backup files deleted after failed sFTP transport

 

[cPanelLauren]

Hi @raudabaugh

Great, I'm following that ticket now and I'll update here when there's any further information or the issue has been resolved.


Thanks!

 

[cPanelLauren]

Hello @raudabaugh

I just checked in on this ticket and it appears that there's a case open for this behavior - CPANEL-26053. We'll update here when the internal case is resolved.