Incremental Backup questions

[reageo]

I have to make a backup for an account that is 180 GB. I noticed that the bulk of space is taken up by files in public_html/upload/files.

I need some guidance for:

1. If I were to run the backup(incremental) after excluding the content at public_html/upload/files, then manually copy the content of public_html/upload/files and then run rsync/incremental backup --> will this ensure that the backup will not take much time and load up the server?
2. I'm not clear on how the metadata file works with incremental backup. Since, I'm manually copying the contents public_html/upload/files, will all of that be excluded from the metadata file created during the incremental backup and hence, would it be copied again while running incremental backup?
3.Lastly, I know that incremental backup first creates a local copy (say 180G) in /backup and then does rync to remote server. In case there isn't any change in data between subsequent backups, will the local copy in /backup still be a complete copy (180G) and then it would rsync just the changes (0 GB) ?

 

[cPanelLauren]

1. If I were to run the backup(incremental) after excluding the content at public_html/upload/files, then manually copy the content of public_html/upload/files and then run rsync/incremental backup --> will this ensure that the backup will not take much time and load up the server?

The backup itself wouldn't take a lot of time but that wouldn't work for the way that incremental backups work for hardlinking. Your best bet with incremental backups would be to allow the first run to complete, which would be the longest/most intensive. Once that backup is complete the subsequent ones will only backup changes since the last backup and be much faster.

2. I'm not clear on how the metadata file works with incremental backup. Since, I'm manually copying the contents public_html/upload/files, will all of that be excluded from the metadata file created during the incremental backup and hence, would it be copied again while running incremental backup?

I believe the documentation here should be helpful for the questions on backup metadata: https://docsbeta.cpanel.net/knowledge-base/backup/how-to-manage-metadata-settings/76/

If you're excluding files in a backup, they will not be added to the backup metadata.

3.Lastly, I know that incremental backup first creates a local copy (say 180G) in /backup and then does rync to remote server. In case there isn't any change in data between subsequent backups, will the local copy in /backup still be a complete copy (180G) and then it would rsync just the changes (0 GB) ?

In the event there are no changes, the backup will still be present and hard linked to the previous one, it will not consume as much space (it's highly unlikely that there are ever 0 changes in a backup when you think about file changes, log entries etc.) but it will contain the changes/differences since the last run.

 

[reageo]

Thank you for your reply. I wanted to clarify one more thing: Is the local backup copy always necessary? I mean, is there any scenario that I can avoid the creation of the local backup copy for the incremental process? The reason I ask this is because the local copy means I need to have twice the space required. For eg: If my data is 10 GB, then I need to have more than 20 GB to ensure enough space for the local backup copy. I know that I can set the configuration so that the local copy gets deleted after the backup runs, but the fact remains that during the backup process, I will need enough space on my server for the original data as well as the local backup copy.

Also, as per retention policy - does pruning/removal happen before or after the backup happens. As in- say I have 10 GB to backup. Retention is set to 1. So, at the remote destination/target - should I have 20 GB space? Or does the last backup get removed from the remote server and then the new backup gets copied to it?

 

[cPanelLauren]

The way that the backup is taken, it is necessary to at least have the amount of space the backup will consume available. It doesn't need to remain there, but will need to be created on the server before being transported.

Also, as per retention policy - does pruning/removal happen before or after the backup happens.

It happens after a successful backup to replace the pruned backup occurs.

 

[reageo]

One more question regarding metadata - Suppose I have 10 domains to backup. The first domain is 90 GB and the remaining nine domains have a total of 10 GB. Taking a backup for 100 GB seems to drain my server resources. So, I started an incremental back for the first one alone. Then, once it got over, I ran another incremental backup on the same day for all the remaining nine together (as its considerably small).

My question is this - I thought I can do the first run separately for the 90 GB domain and then another backup on the same day for the pending together. Since both backups ran on the same day - I assumed that it would have all its metadata together in a single file. The next day, if I were to run all 10 domains together - would the incremental run work correctly?

I tried the the first round ie., one backup for the 90GB domain and a second backup for the remaining 10 GB - but I can see that the metadata file seems to have no mention for the 90 GB domain from the first run. It only seems to have info from the second backup for the remaining nine domains.

 

[cPanelLauren]

There's no way to perform backups this with the automated backup system as it is right now. I'd suggest reading the documentation here: Backup Configuration - Version 84 Documentation - cPanel Documentation

 

[lumina]

One more question regarding metadata - Suppose I have 10 domains to backup. The first domain is 90 GB and the remaining nine domains have a total of 10 GB. Taking a backup for 100 GB seems to drain my server resources. So, I started an incremental back for the first one alone. Then, once it got over, I ran another incremental backup on the same day for all the remaining nine together (as its considerably small).

My question is this - I thought I can do the first run separately for the 90 GB domain and then another backup on the same day for the pending together. Since both backups ran on the same day - I assumed that it would have all its metadata together in a single file. The next day, if I were to run all 10 domains together - would the incremental run work correctly?

I tried the the first round ie., one backup for the 90GB domain and a second backup for the remaining 10 GB - but I can see that the metadata file seems to have no mention for the 90 GB domain from the first run. It only seems to have info from the second backup for the remaining nine domains.

Lot of people would like to have a more flexible backup system, for example here, you can vote for the feature, it would be great for your case or to be discussed, improved and completed...

Ability to Schedule different backup destination for daily, weekly and monthly

I would like to have in the next updates of cPanel a way to control and schedule different destination of backups based on the daily / weekly / montly basi

features.cpanel.net

Another possibility is to separate your tasks:
- use the classic backup system compressed option for all accounts
- for the big account, you add in a file at user root : cpbackup-exclude.conf, inside you add the path of the big directory you would like to remove from classic backup, example "public_html/uploads/"
- and you add a custom script with rsync to backup the big directory directly on a remote server via ssh