Error Pruning - ssh slave failed: timed out

[sp3ctre69]

Hi Michael,

I am still getting this most nights. The problem is when we log tickets about this we don't seem to get the same level of interest/investigation than we sometimes do on the forum. My last ticket about this issue was 9366189. Feel free to take a look but they said it was my end. The problem is I was using an Ubuntu server for SFTP and was told I should use CentOS, so I installed a fresh CentOS box and still have the issue. Like others have said, I am not using SSD's for the backup server, but there is a level of write-cache.

I am getting tired of logging tickets for this for first line support to shut it down saying it is my issue. I think there are sufficient people reporting this for it to be considered a real issue now.

BTW, it happened last night on build 70.0.31. I was hoping your backup changes might have fixed it.

 

[cPanelMichael]

Hi @sp3ctre69,

Let's start from the beginning to see if we can determine what exactly is causing the backup failures. First, could you run the commands referenced below on the affected cPanel server and let us know the output?

cat /usr/local/cpanel/version
cat /var/cpanel/backups/config
cat /var/cpanel/backups/$$$.backup_destination

Replace "/$$$.backup_destination" with the actual file name associated with the remote backup destination on your system, and ensure to exclude the hostname, ID, and password from the output. Additionally, can you provide us with the error messages that appear in /usr/local/cpanel/logs/cpbackup_transporter.log from the most recent backup transport?

Thank you.

 

[sp3ctre69]

Thanks.. here you go:

cat /usr/local/cpanel/version
11.70.0.31

cat /var/cpanel/backups/config
---
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 0,1,2,3,4,5,6
BACKUPDIR: /backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'no'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'no'
BACKUPTYPE: incremental
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 5
BACKUP_MONTHLY_DATES: 1,15
BACKUP_MONTHLY_ENABLE: 'yes'
BACKUP_MONTHLY_RETENTION: 1
BACKUP_WEEKLY_DAY: 0
BACKUP_WEEKLY_ENABLE: 'no'
BACKUP_WEEKLY_RETENTION: 4
CHECK_MIN_FREE_SPACE: 1
ERRORTHRESHHOLD: 3
FORCE_PRUNE_DAILY: 0
FORCE_PRUNE_MONTHLY: 0
FORCE_PRUNE_WEEKLY: 0
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 14400
MIN_FREE_SPACE: 5
MIN_FREE_SPACE_UNIT: percent
MYSQLBACKUP: accounts
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'

cat SFTPrsync_UID_**ID**.backup_destination
---
authtype: password
disabled: 0
host: ***myhostname***
id: **ID**
name: SFTPrsync
password: ***mypassword***
path: /home/****/backup/***servername***/
port: 22
timeout: 300
type: Rsync
upload_system_backup: 1
username: **myusername**

 

[sp3ctre69]

... and the log file...

l08NIt4V[2018-05-03 02:36:39 +0100] info [cpbackup_transporter] Pruning backup directory:  home/**path**/2018-04-28, from SFTPrsync
[2018-05-03 02:42:44 +0100] warn [cpbackup_transporter] Error pruning home/**path**/2018-04-28 from SFTPrsync:  ssh slave failed: timed out at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x259af70), "Error pruning home/**path**/2018-04-28 from SF"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 724
        Cpanel::Backup::Queue::transport_backup::attempt_to_prune_destination(Cpanel::Backup::Queue::transport_backup=HASH(0x2573190), Cpanel::Transport::Files::Rsync=HASH(0x37226b8), 5, undef, Cpanel::LoggerAdapter=HASH(0x259af70)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 233
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2573190), cPanel::TaskQueue::Task=HASH(0x37226e8), Cpanel::LoggerAdapter=HASH(0x259af70)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x2f01af8), CODE(0x33b9af8)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2591d20)) called at /usr/local/cpanel/bin/cpbackup_transporter line 163
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 161
[2018-05-03 02:42:44 +0100] info [cpbackup_transporter] warn [cpbackup_transporter] Error pruning home/**path**/2018-04-28 from SFTPrsync:  ssh slave failed: timed out

 

[cPanelMichael]

Hello @sp3ctre69,

I suspect this is happening because a file transported during the backup process isn't writable by the user configured to transport the files to your remote rsync destination. We do have a couple of open internal cases related to this issue, but first let's make sure that's what is actually causing the issue on your system. Can you confirm if the username configured as part of your Rsync backup destination is a non-root user? If so, do you have root access to the rsync destination, or is your access limited to a non-privileged user?

Thank you.

 

[sp3ctre69]

Hello @sp3ctre69,

I suspect this is happening because a file transported during the backup process isn't writable by the user configured to transport the files to your remote rsync destination. We do have a couple of open internal cases related to this issue, but first let's make sure that's what is actually causing the issue on your system. Can you confirm if the username configured as part of your Rsync backup destination is a non-root user? If so, do you have root access to the rsync destination, or is your access limited to a non-privileged user?

Thank you.

Hi,

Yes, the user is non-root, but I do have full root access to the server.

 

[cPanelMichael]

Yes, the user is non-root, but I do have full root access to the server.

Are you able to temporarily modify the Rsync backup destination so that it uses root instead of a non-root user? If the issue no longer occurs after switching to the root user, then that would confirm the issue is related to one of the cases referenced in my previous response.

Thank you.

 

[sp3ctre69]

weird, I can logon as root but when I use root in the backup settings it won't validate.

 

[cPanelMichael]

weird, I can logon as root but when I use root in the backup settings it won't validate.

Hi @sp3ctre69,

Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination?

Thank you.

 

[sp3ctre69]

Hi @sp3ctre69,

Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination?

Thank you.

Not tried key based yet, but it looks like it does connect ok, it just fails to write the test file. If I change the destination location to /tmp/ it validates fine. Not sure why root would not be able to write the file though.

Error is "Could not upload test file: child exited with code 1"

If I rsync over ssh from the server to the backup location specified in this setup I can transfer files fine as root.

 

[cPanelMichael]

Not tried key based yet, but it looks like it does connect ok, it just fails to write the test file. If I change the destination location to /tmp/ it validates fine. Not sure why root would not be able to write the file though.

Error is "Could not upload test file: child exited with code 1"

Hello @sp3ctre69,

If you create a new Rsync destination with these settings instead of editing the existing destination, does it make a difference?

Thank you.

 

[sp3ctre69]

Hello @sp3ctre69,

If you create a new Rsync destination with these settings instead of editing the existing destination, does it make a difference?

Thank you.

No, same error when I create a new destination

 

[cPanelMichael]

Hello @sp3ctre69,

The backup directory on remote systems is always created relative to the home directory of the logged in user on the remote system. Thus, you'll need to use the following value for the backup directory when configuring the Rsync destination since the user was changed to root:

/../home/backups/servername/

I used this value on a test system and it worked well. Let me know if this helps.

Thank you.

 

[sp3ctre69]

Hi @sp3ctre69,

Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination?

Thank you.

Thanks, looks like it validated ok now. Will see how it runs tonight.

 

[cPanelMichael]

Thanks, looks like it validated ok now. Will see how it runs tonight.

Hello @sp3ctre69,

Let us know how it goes.

Thanks!

 

[sp3ctre69]

Hello @sp3ctre69,

Let us know how it goes.

Thanks!

Hi,

It worked ok last night, so far so good, although it didn’t used to fail every time so will see how it goes in the next day or two.

 

[cPanelMichael]

Hello,

For reference, the internal case numbers associated with this issue are CPANEL-18049 and CPANEL-17941. CPANEL-17941 is open to improve the logging so that this issue is better detected during troubleshooting. CPANEL-18049 is open to determine if this is preventable by checking for this scenario during the destination validation process or by modifying permission values. I'll monitor both cases and update this thread with more information on the status of each case as it becomes available.

Thank you.

Update: Internal case CPANEL-17941 is planned for inclusion with cPanel & WHM version 76.

 

[sp3ctre69]

Seems to have worked every night since changing over to root.

Cheers,

Jim

 

[cPanelMichael]

Hi Jim,

I'm glad to see it's working well thus far. I'll keep an eye on those cases and update this thread once a solution is published.

Thank you.

 

[sp3ctre69]

Looks like I'm getting quite a few other errors in the transporter now:

rsync failed: command exited with code 12: error in rsync protocol data stream