Backup failed, not a readable file with fixed size

[coursevector]

Just had an issue with some backups and checked the transport logs and found this:

[2015-04-29 02:14:25 -0400] warn [cpbackup_transporter] Upload attempt failed: /backup/2015-04-29/accounts/psabdns3.tar.gz not a readable file with fixed size at /usr/local/cpanel/Cpanel/Transport/Files/AmazonS3.pm line 173.
at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 26
   Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1bda348), 'Upload attempt failed: /backup/2015-04-29/accounts/psabdns3.tar.gz not a readable file with fixed size at /usr/local/cpanel/Cpanel/Transport/Files/AmazonS3.pm line 173.\x0A') called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 376
   Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x2685fd8), Cpanel::Transport::Files::AmazonS3=HASH(0x27b9628), '/backup/2015-04-29/accounts/psabdns3.tar.gz', '/2015-04-29/psabdns3.tar.gz', Cpanel::LoggerAdapter=HASH(0x1bda348)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 199
   Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2685fd8), cPanel::TaskQueue::Task=HASH(0x2780ab8), Cpanel::LoggerAdapter=HASH(0x1bda348)) called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 582
   eval {...} called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 585
   cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 238
   eval {...} called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 238
   cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x27e1248), CODE(0x2d7ddb0)) called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 590
   cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x275db70)) called at /usr/local/cpanel/bin/cpbackup_transporter line 151
   eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 149

It then was unable to upload the file for 3 attempts. Immediately after that there was an error packing another account which never even created the file. It then failed to upload the non-existing file 3 times before giving up.

[2015-04-29 02:14:25 -0400] warn [cpbackup_transporter] Upload attempt failed: “Cpanel::Transport::Files::put” attempted to validate a file that does not exist: /backup/2015-04-29/accounts/millersv.tar.gz at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 26
etc......

BUT, after that it successfully uploaded 5 more accounts before finishing. We haven't had any issues with this until recently. It's been running updates this way for months now.

 

[cPanelMichael]

Hello,

The error messages you provided indicate that you might be running short on disk space. Try using the "df -h" and "df -i" commands to verify you are not close to your space and inode limits. It's possible the failed accounts are larger than the others, and thus require more space.

Thank you.

 

[coursevector]

It doesn't look to be the case, see below. Also, we don't store backups on the server itself, it uploads to S3 which helps keep space usage down. Thanks for the suggestion though.

[email protected] [~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 120G 23G 98G 19% /
none 1.0G 4.0K 1.0G 1% /dev

[email protected] [~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vzfs 1000000 465996 534004 47% /
none 262144 109 262035 1% /dev

 

[cPanelMichael]

You still need local space available for the backup generation itself, as it's generated on the local server and stored there temporarily until it's transferred to the remote destination. That being said, your server looks to have sufficient space available. How large is the account it failed at?

Thank you.

 

[coursevector]

Failed:
psabdns3 - 2383 MB
millersv - 517 MB

Successful:
bangorb - 195 MB
basicbud - 58 MB
berlin - 506 MB
boroughb - 51 MB
budgetbu - 205 MB
columbia - 1187 MB
girard - 793 MB
kingston - 164 MB
manheimb - 181 MB
mockups - 93 MB
nemo - 152 MB
newholl - 88 MB
police - 45 MB
staff - 550 MB
stras - 368 MB
webd - 37 MB

 

[cPanelMichael]

Feel free to open a support ticket using the link in my signature so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[coursevector]

Your Request id is: 6468735.

 

[cPanelMichael]

To update, it looks like the server reached it's memory limit during the generation of certain backups, resulting in the failure to successfully backup those accounts.

Thank you.