various issues with backup system

[raysolomon]

I've been having various issues ever since I disabled the legacy backup a few months ago and started using the new backup.

The backup log file (/usr/local/cpanel/logs/cpbackup/xxxxxxxxxx.log) shows the backup process just stops in the middle of an operation without an error message. It stops randomly. Sometimes 300+ accounts are backed up and other times its only 200+ accounts. I currently have 638 accounts to backup.

(Issue #1)
When I set the backup time in the "Cron Config" plugin page in WHM, it adds to the crontab correctly:
0 1 * * * /usr/local/cpanel/scripts/cpbackup

But then the next day somehow it automatically adds a second entry which causes another backup to be executed.
30 21 * * * /usr/local/cpanel/bin/backup
0 1 * * * /usr/local/cpanel/scripts/cpbackup

Then I get an email with an error message basically saying it was about to perform another backup operation, but it couldn't because another backup was being performed at that time.

Message:
"The backup was not able to be completed because timed out waiting for /bin/backup to finish"

So, having that second entry will cause backups to be performed twice if the crontab times are spaced far enough apart. I proved that this happens. I keep deleting the second entry, but it keeps getting added back again. Very annoying and its probably a bug.



(Issue #2)
Backups rarely complete with the new backup system, even when I use the force option to do it manually (/usr/local/cpanel/bin/backup --force).

For example, I get the following errors in this log file each day:
/usr/local/cpanel/logs/cpbackup_transporter.log

Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1c44dc0), 'Upload attempt failed: Can\'t use an undefined value as a symbol reference at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpan5/cpanel_lib/Net/FTP/dataconn.pm line 54.\x0A') called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 219

Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1c44dc0), 'Upload attempt failed: ALLO command ignored.\x0A') called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 219

Each day I get an email of either one of the following errors messages:
* Transport surpassed error threshhold with “51” errors
* The exact error was: The store local backup option is disabled and no active transports exist.
* Backup destination “XXXX.service.softlayer.com” has been disabled on “server.XXXX.com”

Then I have to go into WHM Backup Configuration and re-validate & re-enable the FTP destination.
Just so I can watch it try again the next day and fail miserably and disable itself again. It's kinda frustrating.

Since I have a lot of accounts to backup, I had to manually adjust the MAXIMUM_TIMEOUT and BACKUPFTPTIMEOUT, otherwise I would be getting an FTP timeout error.
I also adjusted the ERRORTHRESHHOLD to 50 to see if that would help complete whatever operation it was doing, but it does not help.

Here is my /var/cpanel/backups/config

---
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 0,1,2,3,4,5,6
BACKUPDIR: /home/cpbackuptmp
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'yes'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'yes'
BACKUPTYPE: compressed
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 1
BACKUP_MONTHLY_DATES: 1,15
BACKUP_MONTHLY_ENABLE: 'yes'
BACKUP_MONTHLY_RETENTION: 1
ERRORTHRESHHOLD: 50
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 0
LINKDEST: 0
LOCALZONESONLY: 'yes'
MAXIMUM_TIMEOUT: 43200
MYSQLBACKUP: accounts
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'
USEBINARYPKGACCT: 'no'

Here is my /etc/cpbackup.conf

BACKUPFTPPASSIVE yes
BACKUPFTPUSER SLXXXX-X
BACKUPBWDATA yes
BACKUPINC no
MYSQLBACKUP accounts
BACKUPDIR /backup
COMPRESSACCTS yes
GZIPRSYNCOPTS --rsyncable
BACKUPINT daily
BACKUP2 yes
BACKUPLOGS yes
BACKUPCHECK yes
BACKUPTYPE ftp
POSTBACKUP 0
BACKUPFTPDIR /cpbackup
USEBINARYPKGACCT yes
BACKUPRETMONTHLY 0
LINKDEST no
BACKUPFTPHOST XXXX.service.softlayer.com
BACKUPENABLE no
BACKUPRETDAILY 1
BACKUPFTPTIMEOUT 43200
BACKUPDAYS 0,1,2,3,4,5,6
PREBACKUP 0
BACKUPFILES no
LOCALZONESONLY no
BACKUPMOUNT no
BACKUPRETWEEKLY 1
DIEIFNOTMOUNTED no
BACKUPACCTS yes

Other notes:
* In Tweak Settings->Compression, I set the "Number of pigz processes" to 6 and "gzip compression level" to 6
* WHM 11.40.1 (build 7)

 

[cPanelMichael]

Hello

Issue 1. It's normal to see cron jobs for both of the following scripts:

/usr/local/cpanel/bin/backup
/usr/local/cpanel/scripts/cpbackup

The new backup script uses /usr/local/cpanel/bin/backup and the legacy backup script uses /usr/local/cpanel/scripts/cpbackup. Cron jobs are not added based on whether or not backups are enabled. The scripts automatically detect if backups are enabled and generate them if they are. The "/usr/local/cpanel/bin/backup" cron job is currently not available in the "Cron Config" plugin page, thus you are seeing the legacy backup cron job there.

Issue 2. Could you open a support ticket for this issue so we can take a closer look? You can post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[raysolomon]

The problem was with Softlayer's NAS backup system. I stopped using it shortly after creating this post and started storing my backups to a dedicated server away from softlayer. Never had a problem since and I'm happy. Softlayer's NAS failed to store my huge backups and this issue was simply too difficult to prove to them it was their problem. It was also much cheaper just buying a dedicated server than using their NAS anyway.