Backup transport errors Unable to send file to destination - file incomplete

[xpace]

Hi all,

I know there are already few threads about this but this one is different:

I have 2 servers - identical with TMD hosting - system CentOS 7.8 kvm and cPanel 88.0.13
Bothe servers have different websites with databases with large amount of data and files between 10GB to 75GB
Backups are being created locally and all of them are being transported remotely.
Each server has 4-5 accounts
Here is the thing:
every account backup except one on each server is being transported successfully.
The accounts (compressed files) in question are over 50GB in size all the rest is just less but some are 45GB or somewhere around, so it seems like only anything above 50GB in size can not be transported or better say finished because the file exist on the remote location but only about 45GB of it instead of 75GB.
There is other file that is complete and is 47GB in size. Of course plenty of space left on the remote location.
It's complaining about timeout which I have changed to the highest possible - 21600 seconds
And this happens every single day on both machines.

Email report says: Unable to send “/disk2/backup/2020-08-18/accounts/xxx.tar.gz” to destination “QNAP-xxx”

Log file says:

[2020-08-18 13:53:11 +1000] info [cpbackup_transporter] Upload attempt #1 starting for /disk2/backup/2020-08-18/accounts/nortwest.tar.gz to DS_Files/DS1266/2020-08-18/accounts/nortwest.tar.gz for destination:  QNAP-NW
[2020-08-18 17:53:11 +1000] warn [cpbackup_transporter] Upload attempt failed: Time out reached for upload attempt #1
 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x19b3930), "Upload attempt failed: Time out reached for upload attempt #1\x{a}") called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 658
        Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x17b5210), Cpanel::Transport::Files::SFTP=HASH(0x2868748), "/disk2/backup/2020-08-18/accounts/nortwest.tar.gz", "DS_Files/DS1266/2020-08-18/accounts/nortwest.tar.gz", Cpanel::LoggerAdapter=HASH(0x19b3930)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x17b5210), Cpanel::TaskQueue::Task=HASH(0x2853688), Cpanel::LoggerAdapter=HASH(0x19b3930)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635
        eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638
        Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
        eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
        Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x2852b78), CODE(0x280cfc8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643
        Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2852f38)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166
[2020-08-18 17:53:15 +1000] info [cpbackup_transporter] Upload attempt #2 starting for /disk2/backup/2020-08-18/accounts/nortwest.tar.gz to DS_Files/DS1266/2020-08-18/accounts/nortwest.tar.gz for destination:  QNAP-NW
[2020-08-18 21:53:15 +1000] warn [cpbackup_transporter] Upload attempt failed: Time out reached for upload attempt #2
 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x19b3930), "Upload attempt failed: Time out reached for upload attempt #2\x{a}") called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 658
        Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x17b5210), Cpanel::Transport::Files::SFTP=HASH(0x2868748), "/disk2/backup/2020-08-18/accounts/nortwest.tar.gz", "DS_Files/DS1266/2020-08-18/accounts/nortwest.tar.gz", Cpanel::LoggerAdapter=HASH(0x19b3930)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x17b5210), Cpanel::TaskQueue::Task=HASH(0x2853688), Cpanel::LoggerAdapter=HASH(0x19b3930)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635
        eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638
        Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
        eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
        Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x2852b78), CODE(0x280cfc8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643
        Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2852f38)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166
[2020-08-18 21:53:19 +1000] info [cpbackup_transporter] Upload attempt #3 starting for /disk2/backup/2020-08-18/accounts/nortwest.tar.gz to DS_Files/DS1266/2020-08-18/accounts/nortwest.tar.gz for destination:  QNAP-NW
[2020-08-19 01:53:19 +1000] warn [cpbackup_transporter] Upload attempt failed: Time out reached for upload attempt #3
 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x19b3930), "Upload attempt failed: Time out reached for upload attempt #3\x{a}") called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 658
        Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x17b5210), Cpanel::Transport::Files::SFTP=HASH(0x2868748), "/disk2/backup/2020-08-18/accounts/nortwest.tar.gz", "DS_Files/DS1266/2020-08-18/accounts/nortwest.tar.gz", Cpanel::LoggerAdapter=HASH(0x19b3930)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x17b5210), Cpanel::TaskQueue::Task=HASH(0x2853688), Cpanel::LoggerAdapter=HASH(0x19b3930)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635
        eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638
        Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
        eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241
        Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x2852b78), CODE(0x280cfc8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643
        Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2852f38)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166

Thank you for your help

 

[cPanelLauren]

It is definitely a timeout as it's happening at precisely 4 hours (14400 seconds) every single time. Which timeout did you set to 21600 (6 hours)? Does this happen with the exact same backup every time on both servers or just any backup over 50GB? If the QNAP server uses an NFS filesystem (which many NAS systems use) there may be additional considerations for using them for backups as they're unsupported. This is noted in the backup configuration documentation as well:

We strongly recommend that you do not perform backups to remote filesystems (for example, NFS, CIFS, smbfs, or other types of network drive systems). While you can back up directly to a remote filesystem, cPanel & WHM does not support this configuration. We strongly recommend that you work with a qualified system administrator to manage this custom backup path and avoid potential risks.

 

[xpace]

It is definitely a timeout as it's happening at precisely 4 hours (14400 seconds) every single time. Which timeout did you set to 21600 (6 hours)? Does this happen with the exact same backup every time on both servers or just any backup over 50GB? If the QNAP server uses an NFS filesystem (which many NAS systems use) there may be additional considerations for using them for backups as they're unsupported. This is noted in the backup configuration documentation as well:

Hi Lauren,
Thank you for your reply.
Yes it happens with the exactly same backups which are bigger than 50GB, I can't tell if it would happen with different files over 50GB because I don't have such files.
Also the QNAP is using NFS as you have mentioned.
If it's unsupported I would love to know what the actual problem is with files bigger than 50GB. I can scp and rsync the files no problem so what's different in WHM/cPanel backup?
I am a sysadmin for this setup and can do adjustments but can not reformat the QNAP with different file system as there are other files and backups from other machines.
So what are my options?

Thank you