backup problem

[golden_eyes]

Hey,

I have over 250 account on my server, during my monitoring I noticed that the backup took some times 36 to 48 hours and some time less than 24 hours to be complete, while the cpbackup is set to be run daily when it exceed the 24 hours another backup process run so it make the server load average too high.

So is it possible to configure the backup to be wait the first running process until finished then wait to the nearest backup schedule or run the queued immediately.

Please help with this matter.

Sincerely,

[cPanelTristan]

Could you try skipping the backups to run every other day but unchecking Monday, Wednesday, Friday and Saturday on the schedule for the days it runs? This would leave Sunday, Tuesday, and Thursday for backups and allow it time to complete those backups.

Right now, there isn't an option to queue backups for the cpbackup feature (WHM > Backup > Configure Backup area).

Please note that to further reduce the time for the backups to run, you might also consider enabling this option in WHM > Backup > Configure Backup area:

EXPERIMENTAL: Use Hard Links for weekly and yearly backups to reduce disk usage and backup time.

Of note, while it indicates weekly and yearly, it is weekly and monthly instead. This feature, while experimental, will reduce the disk space and the time for backups to run as it creates hard links for weekly and monthly backups for files and folders that haven't had changes from the daily backup files or folders.

[golden_eyes]

Thank you for your advice, kindly find my backup configuration below:


Please note that as I said I want the backup daily not day after day, and some times the backup take less than 24 hours and some times take 36, 48 hours it depends on the server load, so what is your suggestion to reduce the backup time and speed up?

What is you opinion with the Incremental Backup? but at the same time I need to compress each account separately.

Sincerely,

[cPanelTristan]

The incremental backup will not work with compression. My suggestion is to use the hard links and disable compression, because you end up with less disk space used with the hard links anyway and wouldn't need compression at that point.

Next, the hard links (as I already noted) should decrease the backup time substantially. By decrease the backup time, it would be 1/2 to 2/3 less time to perform the backups with the hard links, since it won't actually create copies of duplicate data but link to that data. The daily backups will contain all the data and the monthly and weekly backups will contain data only when the files or folders are changed from the daily backup. Otherwise, the monthly and weekly files and folders will link to the daily backup files or folders for that user. This basically is higher level than the incremental backup b/c it isn't just copying what has changed, it is making a hard link for what has changed.

Prior to enabling the hard links feature, I'd suggest moving the existing backups on the machine somewhere, then re-running the hard links option at that point to create them. Any future backups should then maintain the necessary hard links for monthly and weekly other than when data has changed.

[golden_eyes]

With the above backup configuration I'm tacking a full backup on daily basis than there is another script I made to compress all accounts and archive it sequentially 09,10,11,12 of this month..

I don't know if if I used the hard links will keep compressing each account and delete the folder or will keep the folder and will never compress the accounts as usual.

[cPanelTristan]

When you compress the accounts, are you removing the folder then you mean? You cannot remove these folders and files due to them being hard links. You can certainly compress and archive the contents elsewhere, though.

Hard links doesn't delete the folder or file nor does it compress them. You are linking to files and folders in the daily backup that are the same, while data in the daily, weekly and monthly that are different will not be hard links.

It doesn't matter that it doesn't compress the data, though, because it takes up 1/2 to 2/3 less space to use these. That's the reason I'm suggesting trying it. It takes 1/2 to 2/3 less space and 1/2 to 2/3 less time.

[golden_eyes]

no I'm not deleting anything when I archive the previous backup, all what I do is compress the folder /backup/cpbackup/daily once I received a notification from the whm saying that the backup has been completed.

So with the above configuration the the whm script take backup of all accounts , compress each once and delete the folder.

in your case if the folder will never delete but it only compress it so in this case I will move *.tar.gz to another tmp directory and compress them, but please verify if I enabled the use hard links will still compress the accounts?

[cPanelTristan]

in your case if the folder will never delete but it only compress it so in this case I will move *.tar.gz to another tmp directory and compress them, but please verify if I enabled the use hard links will still compress the accounts?

As I said:

Hard links doesn't delete the folder or file nor does it compress them.

[golden_eyes]

mmmmmmmmmm, in this case I must use the "Used hard links" and will make another shell script to compress each account separately.

1. what is the different between the "use hard links" and incremental backup? while the incremental backup is backing up the changed files only? I think the first one will copy the changed files into the folder, unlike the second one it copy what changed only not at all < Please confirm

2. Is the "use hard links" like the rsync?

[cPanelTristan]

Incremental backups makes a copy in daily, weekly and monthly for every single file and folder and only backs up what has changed. It takes up 2 to 3x more disk space than hard links will take up.

Hard links backups up all files for the daily backups, then creates a hard link to the daily files and folders if they haven't changed for the weekly and monthly backups. If the daily has changed from the weekly and monthly, then the hard link isn't used for that file or folder.

The command hard links uses is based on rsync commands, yes.

[golden_eyes]

cooooooooool so I will go with the use hard links and please note that I'm not using the weekly and monthly I'm only use the daily backup so is there any problem with that?

[cPanelTristan]

If you aren't retaining weekly and monthly, this won't help you out for hard links. If you only are retaining and making daily backups entirely, then instead you'd want incremental as that is only the changed files being saved. I apologize for not realizing you weren't retaining weekly and monthly previous to now.

Incremental isn't going to work with compressed backups, but you can always compress them after they complete. Incremental backups should work faster than using compression for the backup times.

[golden_eyes]

ohhhh so what I supposed to do in order to make a daily backup only with reducing the server load average and speed up the time.

[cPanelTristan]

Change to incremental backups, disable compression and test out how that goes.