SOLVED Backup transport errors. Transfer aborted. No space left on device?

kabatak

Well-Known Member
Jun 10, 2009
149
8
68
We received a notice "Backup transport errors" with the log saying "452 Transfer aborted. No space left on device" but the destination still has plenty of space left (35% free space left). The account being backed up is just less than 1GB but the destination has 160GB free. The same backup destination is used by a different cpanel server and that backs up accounts with no issue. Any ideas what could cause this? Thanks.

This happens to several accounts of the server, here's snippet of one of the errors:
Unable to send “/backup/weekly/2018-07-14/accounts/xxxxx.tar.gz” to destination “xxxxx”
From/usr/local/cpanel/logs/cpbackup_transporter.log:
Code:
[2018-07-14 15:18:32 +0800] info [cpbackup_transporter] Instantiating Object
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  220 ProFTPD 1.3.3a Server (xxxxx) [178.32.222.126]
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  USER xxxxx
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  331 Password required for xxxxx
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  PASS ....
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  230 User xxxxx logged in
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  TYPE I
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  200 Type set to I
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Starting a "copy" operation on the "xxxxx" destination ID "xxxxx".
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Base path for destination is /
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Validating destination path /weekly/2018-07-14/accounts/
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  PASV
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  227 Entering Passive Mode (178,32,222,126,120,76).
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  LIST /weekly/2018-07-14/accounts/
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  150 Opening BINARY mode data connection for file list
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  226 Transfer complete
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Path exists
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Uploading account backup /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz to /weekly/2018-07-14/accounts/xxxxx.tar.gz (from weekly/2018-07-14/accounts/xxxxx.tar.$
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Attempting to upload /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz to /weekly/2018-07-14/accounts/xxxxx.tar.gz for destination:  xxxxx
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Upload attempt #1 starting for /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz to /weekly/2018-07-14/accounts/xxxxx.tar.gz for destination:  xxxxx
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  PASV
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  227 Entering Passive Mode (178,32,222,126,124,72).
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  STOR /weekly/2018-07-14/accounts/xxxxx.tar.gz
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  150 Opening BINARY mode data connection for /weekly/2018-07-14/accounts/xxxxx.tar.gz
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  �ABOR
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  452 Transfer aborted. No space left on device
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  226 Abort successful
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] put( /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz, /weekly/2018-07-14/accounts/xxxxx.tar.gz ) FAILED
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Abort successful

[2018-07-14 15:18:35 +0800] warn [cpbackup_transporter] Upload attempt failed: Abort successful
 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1d74078), "Upload attempt failed: Abort successful\x{a}") called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609
        Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1bdd190), Cpanel::Transport::Files::FTP=HASH(0x1bfa9b0), "/backup/weekly/2018-07-14/accounts/xxxxx.tar.$
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1bdd190), cPanel::TaskQueue::Task=HASH(0x30845d8), Cpanel::LoggerAdapter=HASH(0x1d74078)) called at /usr/local/cpane$
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x1bfa3e0), CODE(0x2899880)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x1bfa2d8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: Abort successful

[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  220 ProFTPD 1.3.3a Server (xxxxx) [178.32.222.126]
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  USER xxxxx
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  331 Password required for xxxxx
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  PASS ....
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  230 User xxxxx logged in
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  TYPE I
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  200 Type set to I
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Upload attempt #2 starting for /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz to /weekly/2018-07-14/accounts/xxxxx.tar.gz for destination:  xxxxx
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  PASV
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  227 Entering Passive Mode (178,32,222,126,124,18).
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  STOR /weekly/2018-07-14/accounts/xxxxx.tar.gz
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  150 Opening BINARY mode data connection for /weekly/2018-07-14/accounts/xxxxx.tar.gz
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  �ABOR
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  452 Transfer aborted. No space left on device
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  226 Abort successful
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] put( /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz, /weekly/2018-07-14/accounts/xxxxx.tar.gz ) FAILED
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Abort successful

[2018-07-14 15:18:37 +0800] warn [cpbackup_transporter] Upload attempt failed: Abort successful
 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1d74078), "Upload attempt failed: Abort successful\x{a}") called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609
        Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1bdd190), Cpanel::Transport::Files::FTP=HASH(0x1bfa9b0), "/backup/weekly/2018-07-14/accounts/xxxxx.tar.$
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1bdd190), cPanel::TaskQueue::Task=HASH(0x30845d8), Cpanel::LoggerAdapter=HASH(0x1d74078)) called at /usr/local/cpane$
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x1bfa3e0), CODE(0x2899880)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x1bfa2d8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165
[2018-07-14 15:18:39 +0800] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: Abort successful
 

kabatak

Well-Known Member
Jun 10, 2009
149
8
68
Hi,

No space left on device clearly means that the disk space is the issue.

#df -h
#df -hi

Can you provide the output of the above 2 commands here..
Hello, here's the output:

Code:
[email protected]:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             461G  278G  160G  64% /
tmpfs                 991M     0  991M   0% /lib/init/rw
udev                   10M  148K  9.9M   2% /dev
tmpfs                 991M     0  991M   0% /dev/shm
[email protected]:~# df -hi
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1                29M     75K     29M    1% /
tmpfs                   248K       5    248K    1% /lib/init/rw
udev                    248K    2.2K    246K    1% /dev
tmpfs                   248K       1    248K    1% /dev/shm
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,298
1,279
313
Houston
Hi @kabatak

Is there any quotas in place on the destination server? If you have root access you can run sun something like:

Code:
repquota -a
Also, is this occurring for all accounts or just one?

Thanks!
 

kabatak

Well-Known Member
Jun 10, 2009
149
8
68
Hi @kabatak

Is there any quotas in place on the destination server? If you have root access you can run sun something like:

Code:
repquota -a
Also, is this occurring for all accounts or just one?

Thanks!
I got "-bash: repquota: command not found" error running that command. This is Debian btw.

The "No space left on device" error appears to be happening on many accounts but not all of it.
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,298
1,279
313
Houston
Hi @kabatak

The error does clearly sound like it's a space issue though I can see that it seems you have enough space. To look into this further could you please open a ticket using the link in my signature? Once open please reply with the ticket ID so that we can follow up here with the outcome of the ticket.

Thanks!
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,298
1,279
313
Houston
Thanks @kabatak ,

I found your ticket and I'm now following it, I also noted it with the output of df you provided earlier. I'll update here when there's new information to report.

Thanks!
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,298
1,279
313
Houston
Hi @kabatak

We looked into this issue and it does indeed appear that the backup server was running out of space. The issue is that the backups that are transported are almost the same size as the available space in the destination server -

The scheduled backups generated are 150GB and 158GB:
Code:
[05:55:38 server [email protected] ~]cPs# du -shx /backup/weekly/2018-07-* 158G    /backup/weekly/2018-07-07 150G    /backup/weekly/2018-07-14
Because the pruning operation takes place after the successful transfer the server was indeed running out of space.


Please let us know if you have any questions or concerns in regard to this!


Thanks!
 
  • Like
Reactions: kabatak