Error Pruning - ssh slave failed: timed out

sp3ctre69

Well-Known Member
Aug 14, 2006
107
4
168
Hi Michael,

I am still getting this most nights. The problem is when we log tickets about this we don't seem to get the same level of interest/investigation than we sometimes do on the forum. My last ticket about this issue was 9366189. Feel free to take a look but they said it was my end. The problem is I was using an Ubuntu server for SFTP and was told I should use CentOS, so I installed a fresh CentOS box and still have the issue. Like others have said, I am not using SSD's for the backup server, but there is a level of write-cache.

I am getting tired of logging tickets for this for first line support to shut it down saying it is my issue. I think there are sufficient people reporting this for it to be considered a real issue now.

BTW, it happened last night on build 70.0.31. I was hoping your backup changes might have fixed it.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hi @sp3ctre69,

Let's start from the beginning to see if we can determine what exactly is causing the backup failures. First, could you run the commands referenced below on the affected cPanel server and let us know the output?

Code:
cat /usr/local/cpanel/version
cat /var/cpanel/backups/config
cat /var/cpanel/backups/$$$.backup_destination
Replace "/$$$.backup_destination" with the actual file name associated with the remote backup destination on your system, and ensure to exclude the hostname, ID, and password from the output. Additionally, can you provide us with the error messages that appear in /usr/local/cpanel/logs/cpbackup_transporter.log from the most recent backup transport?

Thank you.
 

sp3ctre69

Well-Known Member
Aug 14, 2006
107
4
168
Thanks.. here you go:

Code:
cat /usr/local/cpanel/version
11.70.0.31
Code:
cat /var/cpanel/backups/config
---
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 0,1,2,3,4,5,6
BACKUPDIR: /backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'no'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'no'
BACKUPTYPE: incremental
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 5
BACKUP_MONTHLY_DATES: 1,15
BACKUP_MONTHLY_ENABLE: 'yes'
BACKUP_MONTHLY_RETENTION: 1
BACKUP_WEEKLY_DAY: 0
BACKUP_WEEKLY_ENABLE: 'no'
BACKUP_WEEKLY_RETENTION: 4
CHECK_MIN_FREE_SPACE: 1
ERRORTHRESHHOLD: 3
FORCE_PRUNE_DAILY: 0
FORCE_PRUNE_MONTHLY: 0
FORCE_PRUNE_WEEKLY: 0
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 14400
MIN_FREE_SPACE: 5
MIN_FREE_SPACE_UNIT: percent
MYSQLBACKUP: accounts
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'
Code:
cat SFTPrsync_UID_**ID**.backup_destination
---
authtype: password
disabled: 0
host: ***myhostname***
id: **ID**
name: SFTPrsync
password: ***mypassword***
path: /home/****/backup/***servername***/
port: 22
timeout: 300
type: Rsync
upload_system_backup: 1
username: **myusername**
 

sp3ctre69

Well-Known Member
Aug 14, 2006
107
4
168
... and the log file...

Code:
l08NIt4V[2018-05-03 02:36:39 +0100] info [cpbackup_transporter] Pruning backup directory:  home/**path**/2018-04-28, from SFTPrsync
[2018-05-03 02:42:44 +0100] warn [cpbackup_transporter] Error pruning home/**path**/2018-04-28 from SFTPrsync:  ssh slave failed: timed out at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x259af70), "Error pruning home/**path**/2018-04-28 from SF"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 724
        Cpanel::Backup::Queue::transport_backup::attempt_to_prune_destination(Cpanel::Backup::Queue::transport_backup=HASH(0x2573190), Cpanel::Transport::Files::Rsync=HASH(0x37226b8), 5, undef, Cpanel::LoggerAdapter=HASH(0x259af70)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 233
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2573190), cPanel::TaskQueue::Task=HASH(0x37226e8), Cpanel::LoggerAdapter=HASH(0x259af70)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x2f01af8), CODE(0x33b9af8)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2591d20)) called at /usr/local/cpanel/bin/cpbackup_transporter line 163
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 161
[2018-05-03 02:42:44 +0100] info [cpbackup_transporter] warn [cpbackup_transporter] Error pruning home/**path**/2018-04-28 from SFTPrsync:  ssh slave failed: timed out
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello @sp3ctre69,

I suspect this is happening because a file transported during the backup process isn't writable by the user configured to transport the files to your remote rsync destination. We do have a couple of open internal cases related to this issue, but first let's make sure that's what is actually causing the issue on your system. Can you confirm if the username configured as part of your Rsync backup destination is a non-root user? If so, do you have root access to the rsync destination, or is your access limited to a non-privileged user?

Thank you.
 

sp3ctre69

Well-Known Member
Aug 14, 2006
107
4
168
Hello @sp3ctre69,

I suspect this is happening because a file transported during the backup process isn't writable by the user configured to transport the files to your remote rsync destination. We do have a couple of open internal cases related to this issue, but first let's make sure that's what is actually causing the issue on your system. Can you confirm if the username configured as part of your Rsync backup destination is a non-root user? If so, do you have root access to the rsync destination, or is your access limited to a non-privileged user?

Thank you.
Hi,

Yes, the user is non-root, but I do have full root access to the server.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Yes, the user is non-root, but I do have full root access to the server.
Are you able to temporarily modify the Rsync backup destination so that it uses root instead of a non-root user? If the issue no longer occurs after switching to the root user, then that would confirm the issue is related to one of the cases referenced in my previous response.

Thank you.
 

sp3ctre69

Well-Known Member
Aug 14, 2006
107
4
168
weird, I can logon as root but when I use root in the backup settings it won't validate.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
weird, I can logon as root but when I use root in the backup settings it won't validate.
Hi @sp3ctre69,

Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination?

Thank you.
 

sp3ctre69

Well-Known Member
Aug 14, 2006
107
4
168
Hi @sp3ctre69,

Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination?

Thank you.
Not tried key based yet, but it looks like it does connect ok, it just fails to write the test file. If I change the destination location to /tmp/ it validates fine. Not sure why root would not be able to write the file though.

Error is "Could not upload test file: child exited with code 1"

If I rsync over ssh from the server to the backup location specified in this setup I can transfer files fine as root.
 
Last edited:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Not tried key based yet, but it looks like it does connect ok, it just fails to write the test file. If I change the destination location to /tmp/ it validates fine. Not sure why root would not be able to write the file though.

Error is "Could not upload test file: child exited with code 1"
Hello @sp3ctre69,

If you create a new Rsync destination with these settings instead of editing the existing destination, does it make a difference?

Thank you.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello @sp3ctre69,

The backup directory on remote systems is always created relative to the home directory of the logged in user on the remote system. Thus, you'll need to use the following value for the backup directory when configuring the Rsync destination since the user was changed to root:

Code:
/../home/backups/servername/
I used this value on a test system and it worked well. Let me know if this helps.

Thank you.
 

sp3ctre69

Well-Known Member
Aug 14, 2006
107
4
168
Hi @sp3ctre69,

Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination?

Thank you.
Thanks, looks like it validated ok now. Will see how it runs tonight.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello,

For reference, the internal case numbers associated with this issue are CPANEL-18049 and CPANEL-17941. CPANEL-17941 is open to improve the logging so that this issue is better detected during troubleshooting. CPANEL-18049 is open to determine if this is preventable by checking for this scenario during the destination validation process or by modifying permission values. I'll monitor both cases and update this thread with more information on the status of each case as it becomes available.

Thank you.

Update: Internal case CPANEL-17941 is planned for inclusion with cPanel & WHM version 76.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hi Jim,

I'm glad to see it's working well thus far. I'll keep an eye on those cases and update this thread once a solution is published.

Thank you.
 

sp3ctre69

Well-Known Member
Aug 14, 2006
107
4
168
Looks like I'm getting quite a few other errors in the transporter now:

Code:
rsync failed: command exited with code 12: error in rsync protocol data stream