Filed with Developers [Case 48882] Backup Queue

[sirdopes]

I would like to have a queue for creating backups. For instance, when clicking on create full backup in cpanel, it would queue the request and check to see if any other backups were running and then proceed. This would prevent a reseller or someone to spawn 10+ backups at one time. Also, an option in whm to select how many concurrent backups are allowed to run.

[VeZoZ]

This is an awesome an idea and definitely should be implemented as soon as possible.

[Kent Brockman]

I have looked at this problem sometime. A backup queue is a good idea.

+1 for this request

[dibarra]

I agree that something like this is needed in cPanel currently. Resellers like to perform backups on all of their clients at once, which causes issues on populated machines.

+1

[InterServed]

+1 vote from me

[Kent Brockman]

Hi there people. Today I were thinking about this feature, and recalled when sometimes a backup process collides with another heavy weight process in course, like mailman sending a ton of mails and the load balance reach a spike of 18 in just 2 minutes of work. I have arranged to setup well separated cron hours for these tasks, but just because I do know how to avoid those scenarios doesn't mean that a simple queue will save problems to underpowered servers or novice admins. I think that the backup queue should also take in account the current load of the server and automatically pause itself until the load slow down to a reasonably normal value (that could set in the Backup screen as "Max Load after which backups are paused"), and issue an email alert to the sysadmin if the load don't get normalized in X minutes (also, set in the Backup screen: "Max High Load Time Wait before issuing an alert").
What do you think?

[InterServed]

Hi there people. Today I were thinking about this feature, and recalled when sometimes a backup process collides with another heavy weight process in course, like mailman sending a ton of mails and the load balance reach a spike of 18 in just 2 minutes of work. I have arranged to setup well separated cron hours for these tasks, but just because I do know how to avoid those scenarios doesn't mean that a simple queue will save problems to underpowered servers or novice admins. I think that the backup queue should also take in account the current load of the server and automatically pause itself until the load slow down to a reasonably normal value (that could set in the Backup screen as "Max Load after which backups are paused"), and issue an email alert to the sysadmin if the load don't get normalized in X minutes (also, set in the Backup screen: "Max High Load Time Wait before issuing an alert").
What do you think?

Load control with pause already exist into the current backup process.

here's an example of custom backup script i run for a customer:

#!/bin/bash
BROOT="/backup/custom_backups/domain.com/"
BACKUPDIR="/backup/custom_backups/domain.com/`date +%Y-%m-%d`"
CPUWATCH=/usr/local/cpanel/bin/cpuwatch
MAXLOAD=6.0
USER="customer_username"
BACKUPSCRIPT=/scripts/pkgacct
$CPUWATCH $MAXLOAD $BACKUPSCRIPT $USER
sleep 20
mkdir -p $BACKUPDIR ; mv /home/cpmove-customer_username.* $BACKUPDIR
cd $BROOT && find -mtime +4|xargs rm -rf

So as you can see i can control the max load for the backup process. In this example if the server load is >=6 then the backup script will be paused until load will decrease.

[Kent Brockman]

Great! And the native backup system in WHM does effectively check the load of the server?

[InterServed]

Great! And the native backup system in WHM does effectively check the load of the server?

Yes that's correct.

[cPanelDavidG]

Please review this report for accuracy and completeness before I submit this to our developers:

This feature will add a tweak setting that will determine how many instances of backups triggered by cPanel users via the cPanel interface will be permitted. This setting will only govern backups triggered via the cPanel interface and not include instances of the pkgacct script called by the cpbackup server-wide backup script.

The goal is to reduce resource consumption caused when a reseller with a dozen accounts decides to backup all their accounts simultaneously. The load of backing up several accounts simultaneously can be quite noticeable. The folks triggering these actions are not responsible for maintaining the server and may be unaware of the strain they are placing on the server.

If more than the designated number of permitted backups is taking place, then further requests will be queued until the number of running backups is below the threshold of permitted backup processes. This queue should be designed to only permit 1 entry per cPanel account as not to queue a backup for the same account twice.

[Kent Brockman]

I do agree.
"how many instances of backups triggered by cPanel users via the cPanel interface will be permitted": should be defaulted to 1 to cover actual scenario. Then sysadmins will decide if opening to more instances is a good idea for their different scenarios.

Let's hear some more comments

[Infopro]

Extra CPUs for server load in Tweak Settings works in a similar fashion and backups are set to obey this setting, correct?

[Kent Brockman]

Extra CPUs for server load in Tweak Settings works in a similar fashion and backups are set to obey this setting, correct?

Extra CPUs for server load defaults to 0. What's taken in account when set to zero? The number of CPUs?

[Infopro]

What else could it default to on a new installation of cPanel? It should be changed like many other settings.

The load average above the number of CPUs at which cpuwatch, cpanellogd, backups, and CPU stats consider the system to be in a critical load state. For example, a server with 4 physical CPUs and a value of 2 in this field will be considered “critical” in these cases once the load reaches 6.

[Kent Brockman]

I do agree, Infopro.
Actually, if several backup attempts are made and every one is detecting a critical load state, they will remain paused until the load is normalized again. A server with 4 CPUs may not be affected, but still may have performance issues, which may be avoided by queuing the backup tasks.