Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Pruning of backups from failed transports

Discussion in 'Data Protection' started by raudabaugh, Mar 12, 2019.

  1. raudabaugh

    raudabaugh Member

    Joined:
    Jan 5, 2006
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    151
    I have a daily backup that does an sftp of files to a remote sftp site. Past behaviour would retain any backup files that failed during the transport. Recently, the cpbackup process has changed and has been pruning all of the backup files, regardless of the success of the transport.

    I'd like to revert back to the old process of not deleting backup files after a failed transport. It allows me to followup with these failed transports and finish the transfer. How can I do this without asking the server to retain all of the files regardless of backup success?
     
  2. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,707
    Likes Received:
    436
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @raudabaugh

    This behavior is a result of the following setting being enabled at WHM>>Backup>>Backup Configuration ->
    Strictly enforce retention, regardless of backup success.

    For more information, read our Backup Configuration documentation.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. raudabaugh

    raudabaugh Member

    Joined:
    Jan 5, 2006
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    151
    Just to be clear, until recently, the behaviour was completely different. I've always have had this enforce retention setting OFF, and the system would only delete those files that had successfully transferred to the remote destination. What happened?

    Think about it. This is a valuable function. I'd prefer that any files that have been transported successfully to be removed from the production server. This is the way it worked for years, until recently.
     
  4. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,707
    Likes Received:
    436
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @raudabaugh


    This setting defaults to off, if it was enabled it means someone had to have enabled it in the UI or through the backup configuration.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. raudabaugh

    raudabaugh Member

    Joined:
    Jan 5, 2006
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    151
    Sorry, I'm not being clear enough.

    The problem is not really an issue with the 'Enforce Retention" feature of the cpbackup. In fact, I've never set the switch to the 'on' position for a very good reason. My system does not have enough space to be retaining backups for any extended period of time. Hence the reason that i setup a SFTP transport process to move the daily backups to an 'off site' system that has the disk space I need.

    HERE is the problem:

    For years and years, the CPBACKUP system would only delete the backup files that were successfully transported to my SFTP site. It would retain any files that failed. This is a huge advantage where I could 'clean up' failed transports without regenerating. What happen to this behaviour? This is something the seemed to go away after the most recent whm/cpanel upgrades.
     
  6. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,707
    Likes Received:
    436
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    HI @raudabaugh

    This is how it should be behaving still without "Enforce Retention" enabled. The backup logs should indicate what happened with the failed backups specifically. You can find them at /usr/local/cpanel/logs/cpbackup/
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. raudabaugh

    raudabaugh Member

    Joined:
    Jan 5, 2006
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    151
    Here is the email notification:

    Transport errors encountered.

    The system encountered errors during transport of the backup files.

    Transport Backup Label Message
    adrivesftp someusr Unable to send “/backup/2019-03-21/accounts/someusr.tar.gz” to destination “adrivesftp”
    adrivesftp someusrtoo Unable to send “/backup/2019-03-21/accounts/someusrtoo.tar.gz” to destination “adrivesftp”


    Below is a preview of the attached log file.
    Code:
    Preview of transport errors log:
    Unable to send “/backup/2019-03-21/accounts/someusrtoo.tar.gz” to destination “adrivesftp”
    Unable to send “/backup/2019-03-21/accounts/someusr.tar.gz” to destination “adrivesftp”
    
    This notice is the result of a request from “cpbackup”.
    The system generated this notice on Thursday, March 21, 2019 at 8:17:23 AM UTC.
    
    Here is the cpbackup log entry for someusrtoo.tar.gz:
    
    [2019-03-21 02:00:41 -0400] info [backup] Running account backup with target : /backup/2019-0
    3-21/accounts
    [2019-03-21 02:00:41 -0400] info [backup] checking backup for someusrtoo
    [2019-03-21 02:00:41 -0400] info [backup] Backups ARE enabled for someusrtoo
    [2019-03-21 02:00:41 -0400] info [backup] Calling pkgacct under cpuwatch to backup user “tavq
    uote”
    [2019-03-21 02:00:42 -0400] pkgacct started.
    [2019-03-21 02:00:42 -0400] pkgacct version 10 - user : someusrtoo - tarball: 1 - target mysql
    : default - split: 0 - incremental: 0 - homedir: 1 - mailman: 1 - backup: 1 - archive version
    : 4 - running with uid 0
    [2019-03-21 02:00:42 -0400] pkgacct using '/usr/local/cpanel/3rdparty/bin/pigz -6 --processes
    1 --blocksize 4096 --rsyncable' to compress archives
    [2019-03-21 02:00:42 -0400] pkgacct working dir : /backup/2019-03-21/accounts/someusrtoo
    [2019-03-21 02:00:42 -0400] Copying Reseller Config...[2019-03-21 02:00:42 -0400] Done
    [2019-03-21 02:00:42 -0400] Copying Suspension Info (if needed)...[2019-03-21 02:00:42 -0400]
    Done
    [2019-03-21 02:00:42 -0400] Copying installed SSL certificates and keys...[2019-03-21 02:00:4
    2 -0400] Performing “ApacheTLS” component....
    [2019-03-21 02:00:42 -0400] Completed “ApacheTLS” component.
    [2019-03-21 02:00:42 -0400] Done
    [2019-03-21 02:00:42 -0400] Copying Domain Keys....[2019-03-21 02:00:42 -0400] Done
    [2019-03-21 02:00:42 -0400] Copying Counter Data....[2019-03-21 02:00:42 -0400] Done
    [2019-03-21 02:00:42 -0400] Copying Bandwidth Data....[2019-03-21 02:00:42 -0400] Performing
    “Bandwidth” component....
    Summary databases … done!
    [2019-03-21 02:00:42 -0400] Completed “Bandwidth” component.
    [2019-03-21 02:00:42 -0400] Done
    [2019-03-21 02:00:42 -0400] Copying Dns Zones.......someusrtoos.com...[2019-03-21 02:00:42 -040
    0] Done
    [2019-03-21 02:00:42 -0400] Copying Mail files....[2019-03-21 02:00:42 -0400] Done
    [2019-03-21 02:00:42 -0400] Copying proftpd file....[2019-03-21 02:00:42 -0400] Done
    [2019-03-21 02:00:42 -0400] Performing “Logs” component....
    ...log file sizes [3637 byte(s)]......someusrtoos.com-bytes_log......someusrtoos.com......tavquot
    es.com-ssl_log...[2019-03-21 02:00:42 -0400] Completed “Logs” component.
    [2019-03-21 02:00:42 -0400] Copy userdata...[2019-03-21 02:00:42 -0400] Done
    [2019-03-21 02:00:42 -0400] Copy custom virtualhost templates...[2019-03-21 02:00:42 -0400] D
    one
    [2019-03-21 02:00:42 -0400] Copying mailman lists and archives....Done copying mailman lists
    and archives.
    [2019-03-21 02:00:42 -0400] Copying homedir.............
    [2019-03-21 02:00:46 -0400] Done
    [2019-03-21 02:00:46 -0400] Fixing up EA4 .htaccess blocks: /home/someusrtoo/public_html/.htacc
    ess Done.
    [2019-03-21 02:00:46 -0400] Performing “Postgresql” component....
    [2019-03-21 02:00:46 -0400] Completed “Postgresql” component.
    [2019-03-21 02:00:46 -0400] Performing “Mysql” component....
    Load watching suspended due to SIGUSR1
    [2019-03-21 02:00:46 -0400] Determining mysql dbs...[2019-03-21 02:00:46 -0400] ...mysqldump
    version: 10.0.38...[2019-03-21 02:00:46 -0400] ...mysql version: 10.0...[2019-03-21 02:00:46
    -0400] Saving mysql privs...[2019-03-21 02:00:46 -0400] Done
    [2019-03-21 02:00:46 -0400] ...Done
    [2019-03-21 02:00:46 -0400] Storing mysql dbs............
    someusrtoo_db[2019-03-21 02:00:46 -0400] (34195 bytes)
    [2019-03-21 02:00:46 -0400] ...Done
    [2019-03-21 02:00:46 -0400] Completed “Mysql” component.
    [2019-03-21 02:00:46 -0400] Performing “MysqlRemoteNotes” component....
    [2019-03-21 02:00:46 -0400] Completed “MysqlRemoteNotes” component.
    [2019-03-21 02:00:46 -0400] Copying cpuser file.......Load watching resumed due to SIGUSR2
    [2019-03-21 02:00:46 -0400] Done
    [2019-03-21 02:00:46 -0400] Copying crontab file.......[2019-03-21 02:00:46 -0400] Done
    [2019-03-21 02:00:46 -0400] Performing “Quota” component....
    [2019-03-21 02:00:46 -0400] Completed “Quota” component.
    [2019-03-21 02:00:46 -0400] Performing “Integration” component....
    [2019-03-21 02:00:46 -0400] Completed “Integration” component.
    [2019-03-21 02:00:46 -0400] Performing “AuthnLinks” component....
    [2019-03-21 02:00:46 -0400] Completed “AuthnLinks” component.
    [2019-03-21 02:00:46 -0400] Performing “APITokens” component....
    [2019-03-21 02:00:46 -0400] Completed “APITokens” component.
    [2019-03-21 02:00:46 -0400] Performing “AutoSSL” component....
    [2019-03-21 02:00:46 -0400] Completed “AutoSSL” component.
    [2019-03-21 02:00:46 -0400] Storing Subdomains....
    [2019-03-21 02:00:46 -0400] Done
    [2019-03-21 02:00:46 -0400] Storing Parked Domains....
    [2019-03-21 02:00:46 -0400] Done
    [2019-03-21 02:00:46 -0400] Storing Addon Domains....
    [2019-03-21 02:00:46 -0400] Performing “Password” component....
    [2019-03-21 02:00:46 -0400] Completed “Password” component.
    [2019-03-21 02:00:46 -0400] Performing “DigestShadow” component....
    [2019-03-21 02:00:46 -0400] Completed “DigestShadow” component.
    [2019-03-21 02:00:46 -0400] Copying shell.......[2019-03-21 02:00:46 -0400] Done
    [2019-03-21 02:00:46 -0400] Performing “PublicContact” component....
    [2019-03-21 02:00:46 -0400] Completed “PublicContact” component.
    [2019-03-21 02:00:46 -0400] Performing “MailLimits” component....
    [2019-03-21 02:00:46 -0400] Completed “MailLimits” component.
    [2019-03-21 02:00:46 -0400] Creating Archive ........................
    [2019-03-21 02:01:28 -0400] Done
    [2019-03-21 02:01:28 -0400] pkgacctfile is: /backup/2019-03-21/accounts/someusrtoo.tar.gz
    [2019-03-21 02:01:28 -0400]
    [2019-03-21 02:01:28 -0400] size is: 50024317
    [2019-03-21 02:01:28 -0400]
    [2019-03-21 02:01:28 -0400] homesize is: 343334912
    [2019-03-21 02:01:28 -0400]
    [2019-03-21 02:01:28 -0400] homefiles is: 38626
    [2019-03-21 02:01:28 -0400] pkgacct completed
    [2019-03-21 02:01:28 -0400] info [backup] Successfully backed up account “someusrtoo” to “/back
    up/2019-03-21/accounts”
    [2019-03-21 02:01:28 -0400] info [backup] Adding metadata information for someusrtoo to backup
    at /backup/2019-03-21
    [2019-03-21 02:01:28 -0400] info [backup] Queuing daily backup copy of “someusrtoo” for transpo
    rt of “/backup/2019-03-21/accounts/someusrtoo.tar.gz” to “2019-03-21/accounts/someusrtoo.tar.gz”
    [2019-03-21 02:01:28 -0400] info [backup] This particular transport will be queued with keep_
    local = 0 , based on the need to copy weekly () and/or monthly () copies as well.
    [2019-03-21 02:01:28 -0400] info [backup] Queuing transport of file: /backup/2019-03-21/accou
    nts/someusrtoo.tar.gz
    [2019-03-21 02:01:28 -0400] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:527
    [2019-03-21 02:01:28 -0400] info [backup] leaving queue_backup_transport_item
    
    cpbackup_transporter.log did not have specific reference to someusrtoo.tar.gz, but there were transport errors from failed transport attempts.
    
    Here is a copy of the server backup directory:

    Code:
    root@server1 [/backup]# ls -l
    total 68
    drwx--x--x 3 root root 4096 Mar 21 04:17 ./
    drwxr-xr-x. 22 root root 4096 Feb 20 11:22 ../
    -rw------- 1 root root 57344 Mar 21 04:17 transports.db
    drwx--x--x 3 root root 4096 Mar 17 02:00 weekly/ 
    
     
    #7 raudabaugh, Mar 21, 2019
    Last edited by a moderator: Mar 21, 2019
  8. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,707
    Likes Received:
    436
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @raudabaugh

    Interesting and the transport log didn't mention anything about pruning either? All transports should be logged if they were successful but in this instance, it seems like there was some failure that caused it to error out unexpectedly. At this point, it might be best to open a ticket which you can do using the link in my signature. Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. raudabaugh

    raudabaugh Member

    Joined:
    Jan 5, 2006
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    151
    11771623 - Backup files deleted after failed sFTP transport
     
  10. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,707
    Likes Received:
    436
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @raudabaugh

    Great, I'm following that ticket now and I'll update here when there's any further information or the issue has been resolved.


    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,707
    Likes Received:
    436
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hello @raudabaugh

    I just checked in on this ticket and it appears that there's a case open for this behavior - CPANEL-26053. We'll update here when the internal case is resolved.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice