Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Error Pruning - ssh slave failed: timed out

Discussion in 'Data Protection' started by sp3ctre69, May 3, 2018.

Tags:
  1. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    Hi Michael,

    I am still getting this most nights. The problem is when we log tickets about this we don't seem to get the same level of interest/investigation than we sometimes do on the forum. My last ticket about this issue was 9366189. Feel free to take a look but they said it was my end. The problem is I was using an Ubuntu server for SFTP and was told I should use CentOS, so I installed a fresh CentOS box and still have the issue. Like others have said, I am not using SSD's for the backup server, but there is a level of write-cache.

    I am getting tired of logging tickets for this for first line support to shut it down saying it is my issue. I think there are sufficient people reporting this for it to be considered a real issue now.

    BTW, it happened last night on build 70.0.31. I was hoping your backup changes might have fixed it.
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hi @sp3ctre69,

    Let's start from the beginning to see if we can determine what exactly is causing the backup failures. First, could you run the commands referenced below on the affected cPanel server and let us know the output?

    Code:
    cat /usr/local/cpanel/version
    cat /var/cpanel/backups/config
    cat /var/cpanel/backups/$$$.backup_destination 
    Replace "/$$$.backup_destination" with the actual file name associated with the remote backup destination on your system, and ensure to exclude the hostname, ID, and password from the output. Additionally, can you provide us with the error messages that appear in /usr/local/cpanel/logs/cpbackup_transporter.log from the most recent backup transport?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    Thanks.. here you go:

    Code:
    cat /usr/local/cpanel/version
    11.70.0.31
    
    Code:
    cat /var/cpanel/backups/config
    ---
    BACKUPACCTS: 'yes'
    BACKUPBWDATA: 'yes'
    BACKUPDAYS: 0,1,2,3,4,5,6
    BACKUPDIR: /backup
    BACKUPENABLE: 'yes'
    BACKUPFILES: 'yes'
    BACKUPLOGS: 'no'
    BACKUPMOUNT: 'no'
    BACKUPSUSPENDEDACCTS: 'no'
    BACKUPTYPE: incremental
    BACKUP_DAILY_ENABLE: 'yes'
    BACKUP_DAILY_RETENTION: 5
    BACKUP_MONTHLY_DATES: 1,15
    BACKUP_MONTHLY_ENABLE: 'yes'
    BACKUP_MONTHLY_RETENTION: 1
    BACKUP_WEEKLY_DAY: 0
    BACKUP_WEEKLY_ENABLE: 'no'
    BACKUP_WEEKLY_RETENTION: 4
    CHECK_MIN_FREE_SPACE: 1
    ERRORTHRESHHOLD: 3
    FORCE_PRUNE_DAILY: 0
    FORCE_PRUNE_MONTHLY: 0
    FORCE_PRUNE_WEEKLY: 0
    GZIPRSYNCOPTS: --rsyncable
    KEEPLOCAL: 1
    LINKDEST: 0
    LOCALZONESONLY: 'no'
    MAXIMUM_RESTORE_TIMEOUT: 21600
    MAXIMUM_TIMEOUT: 14400
    MIN_FREE_SPACE: 5
    MIN_FREE_SPACE_UNIT: percent
    MYSQLBACKUP: accounts
    POSTBACKUP: 'no'
    PREBACKUP: -1
    PSQLBACKUP: 'no'
    
    Code:
    cat SFTPrsync_UID_**ID**.backup_destination
    ---
    authtype: password
    disabled: 0
    host: ***myhostname***
    id: **ID**
    name: SFTPrsync
    password: ***mypassword***
    path: /home/****/backup/***servername***/
    port: 22
    timeout: 300
    type: Rsync
    upload_system_backup: 1
    username: **myusername**
    
     
  4. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    ... and the log file...

    Code:
    l08NIt4V[2018-05-03 02:36:39 +0100] info [cpbackup_transporter] Pruning backup directory:  home/**path**/2018-04-28, from SFTPrsync
    [2018-05-03 02:42:44 +0100] warn [cpbackup_transporter] Error pruning home/**path**/2018-04-28 from SFTPrsync:  ssh slave failed: timed out at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
            Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x259af70), "Error pruning home/**path**/2018-04-28 from SF"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 724
            Cpanel::Backup::Queue::transport_backup::attempt_to_prune_destination(Cpanel::Backup::Queue::transport_backup=HASH(0x2573190), Cpanel::Transport::Files::Rsync=HASH(0x37226b8), 5, undef, Cpanel::LoggerAdapter=HASH(0x259af70)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 233
            Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2573190), cPanel::TaskQueue::Task=HASH(0x37226e8), Cpanel::LoggerAdapter=HASH(0x259af70)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624
            eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
            cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
            eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
            cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x2f01af8), CODE(0x33b9af8)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
            cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2591d20)) called at /usr/local/cpanel/bin/cpbackup_transporter line 163
            eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 161
    [2018-05-03 02:42:44 +0100] info [cpbackup_transporter] warn [cpbackup_transporter] Error pruning home/**path**/2018-04-28 from SFTPrsync:  ssh slave failed: timed out
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello @sp3ctre69,

    I suspect this is happening because a file transported during the backup process isn't writable by the user configured to transport the files to your remote rsync destination. We do have a couple of open internal cases related to this issue, but first let's make sure that's what is actually causing the issue on your system. Can you confirm if the username configured as part of your Rsync backup destination is a non-root user? If so, do you have root access to the rsync destination, or is your access limited to a non-privileged user?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    Hi,

    Yes, the user is non-root, but I do have full root access to the server.
     
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Are you able to temporarily modify the Rsync backup destination so that it uses root instead of a non-root user? If the issue no longer occurs after switching to the root user, then that would confirm the issue is related to one of the cases referenced in my previous response.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    weird, I can logon as root but when I use root in the backup settings it won't validate.
     
  9. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hi @sp3ctre69,

    Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    Not tried key based yet, but it looks like it does connect ok, it just fails to write the test file. If I change the destination location to /tmp/ it validates fine. Not sure why root would not be able to write the file though.

    Error is "Could not upload test file: child exited with code 1"

    If I rsync over ssh from the server to the backup location specified in this setup I can transfer files fine as root.
     
    #10 sp3ctre69, May 4, 2018
    Last edited: May 4, 2018
  11. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello @sp3ctre69,

    If you create a new Rsync destination with these settings instead of editing the existing destination, does it make a difference?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    No, same error when I create a new destination
     
  13. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello @sp3ctre69,

    The backup directory on remote systems is always created relative to the home directory of the logged in user on the remote system. Thus, you'll need to use the following value for the backup directory when configuring the Rsync destination since the user was changed to root:

    Code:
    /../home/backups/servername/
    I used this value on a test system and it worked well. Let me know if this helps.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    Thanks, looks like it validated ok now. Will see how it runs tonight.
     
  15. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello @sp3ctre69,

    Let us know how it goes.

    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    Hi,

    It worked ok last night, so far so good, although it didn’t used to fail every time so will see how it goes in the next day or two.
     
    cPanelMichael likes this.
  17. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    For reference, the internal case numbers associated with this issue are CPANEL-18049 and CPANEL-17941. CPANEL-17941 is open to improve the logging so that this issue is better detected during troubleshooting. CPANEL-18049 is open to determine if this is preventable by checking for this scenario during the destination validation process or by modifying permission values. I'll monitor both cases and update this thread with more information on the status of each case as it becomes available.

    Thank you.

    Update: Internal case CPANEL-17941 is planned for inclusion with cPanel & WHM version 76.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    Seems to have worked every night since changing over to root.

    Cheers,

    Jim
     
    cPanelMichael likes this.
  19. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    43,870
    Likes Received:
    1,811
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hi Jim,

    I'm glad to see it's working well thus far. I'll keep an eye on those cases and update this thread once a solution is published.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  20. sp3ctre69

    sp3ctre69 Well-Known Member

    Joined:
    Aug 14, 2006
    Messages:
    103
    Likes Received:
    3
    Trophy Points:
    168
    Looks like I'm getting quite a few other errors in the transporter now:

    Code:
    rsync failed: command exited with code 12: error in rsync protocol data stream
     
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice