New backup system: disk space

[mindnetcombr]

Hello,

First, let me tell the behavior from legacy backup:

- Server have 100Gb disk space
- Data usage is 40Gb
- Legacy backup run and generate a daily backup with 40Gb
- Data 40Gb + Backup 40Gb = 80Gb, and left 20Gb of space on server
- Next day: legacy backup run, overwrite the previous backup, 20Gb of space free, loop.

Now with the new backup system:

- Server have 100Gb disk space
- Data usage is 40Gb
- NEW backup run and generate a daily backup with 40Gb
- Data 40Gb + Backup 40Gb = 80Gb, and left 20Gb of space on server
- Next day: NEW backup run, and only will delete the previous backup (previous day) if the today backup finish without errors. We have 20Gb of space, new backup will fail because server dont have 40Gb free.

So in the new backup system, the "1 day backup retention" need twice backup disk space than your data. If you have 1Tb of data you need at least 2Tb of disk space to save 1 backup:

- "Retention behavior - When a complete backup finishes, the system deletes the oldest backup. When the system backup partially completes (fails), the system will not delete the oldest backup, and it will continue to retain the oldest backups. This procedure ensures that the system retains at least one retained complete backup." - source 11.64 manual.

Good security measure, but now all servers will need more backup space.

This is a problem on some servers, ex: VPS servers with one disk, need reserve 2/3 of disk space to save 1 backup. You have a VPS with 100Gb? Good, but if you want generate ONE backup you can only use 33Gb of your 100Gb disk space.

Any way to the new backup system "overwrite" the previous backup, like the legacy do?

 

[Infopro]

The docs should be of some use to you:
Backup Configuration - Version 64 Documentation - cPanel Documentation

Each Strictly enforce retention, regardless of backup success setting overrides the standard behavior for that backup type. If you select this option, the system will always retain only the desired number of backups with at least one complete backup.

cPanelTV has a new video explaining things as well:

 

[mindnetcombr]

I try this before open this thread.

I enabled the option "Strictly enforce retention, regardless of backup success." but the new backup system only delete the previous backup after run the today backup.

In any case the new backup system will keep the yesterday + today backup at same time until finish the today backup, because the option "Strictly enforce retention, regardless of backup success." only tell to backup system to delete previous day even if the today backup have errors.

In any case the space used by previous backup will be used until the today backup finish.

The new backup system dont delete the previous backup to start a new one, even with "Strictly enforce retention, regardless of backup success." enabled.

So the server must have twice disk space to start a new backup job.

Any way to force new backup system delete the previous backup before start a new one, like the legacy do?

 

[cPanelMichael]

Hello @mindnetcombr,

Would you mind posting the contents of the /var/cpanel/backups/config file on your system so we can verify how backups are configured?

Thank you.

 

[mindnetcombr]

Hello @mindnetcombr,

Would you mind posting the contents of the /var/cpanel/backups/config file on your system so we can verify how backups are configured?

Thank you.

Sure, this is the file:

---
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 0,1,2,3,4,5,6
BACKUPDIR: /backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'no'
BACKUPLOGS: 'no'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'yes'
BACKUPTYPE: compressed
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 1
BACKUP_MONTHLY_DATES: 1
BACKUP_MONTHLY_ENABLE: 'no'
BACKUP_MONTHLY_RETENTION: 1
BACKUP_WEEKLY_DAY: 0
BACKUP_WEEKLY_ENABLE: 'no'
BACKUP_WEEKLY_RETENTION: 4
CHECK_MIN_FREE_SPACE: 1
ERRORTHRESHHOLD: 3
FORCE_PRUNE_DAILY: 1
FORCE_PRUNE_MONTHLY: 0
FORCE_PRUNE_WEEKLY: 0
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 7200
MIN_FREE_SPACE: 5
MIN_FREE_SPACE_UNIT: percent
MYSQLBACKUP: accounts
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'


The behavior of backup is:

1) generate backup for "day 1", using 40Gb to save backup.
2) on next day, generate a new backup for "day 2", using 40Gb to save backup (at this point backup use 80Gb)
3) at the end of "day 2" backup the "day 1" is deleted (and now only 40Gb is used by backup)

With or without the option "Strictly enforce retention, regardless of backup success." enabled ALWAYS the behavior of backup is this way described on these 3 steps above.

So no matter what I do, I need 80Gb of disk space to save only 1 backup (retention = 1) of 40GB data size.

I believe cpanel can make a option like "delete oldest backup on retention before start a new one" - DISABLED by default and with all warning about the risk.

Thank

 

[cPanelMichael]

Hello,

I believe the incremental backup type would better suite your needs if disk space is a concern. Here's a description of this backup type on our Backup Configuration document:

Incremental — Select this setting to store only one uncompressed backup file. The system saves only updated account information to the existing backup file. The system will not delete the backup file due to age (also known as "pruning"), even if the account is terminated. This setting limits your restoration settings, but runs faster and uses less disk space than other backup types.

Since it uses hard links, it won't make a full copy of all files under the accounts during the backup generation, leading to it taking up less space.

Thank you.

 

[Wabun]

Hello,

I believe the incremental backup type would better suite your needs if disk space is a concern. Here's a description of this backup type on our Backup Configuration document:



Since it uses hard links, it won't make a full copy of all files under the accounts during the backup generation, leading to it taking up less space.

Thank you.

This is no good option for me either, I have one huge account and many small ones, VPS is only 50GB, using the legacy with FTP backup.
I can't use for same space reason the new backup system, I wish you guys would not be so sure in deleting the legacy system when it is still used by many small VPS owners, in which I am sure I am not the only one!!!

 

[cPanelMichael]

This is no good option for me either, I have one huge account and many small ones, VPS is only 50GB, using the legacy with FTP backup.
I can't use for same space reason the new backup system, I wish you guys would not be so sure in deleting the legacy system when it is still used by many small VPS owners, in which I am sure I am not the only one!!!

Hello,

Could you provide some more details about what in-particular uses more space with the new backup system? Here's a thread that details how hard links work:

How do the backups work exactly?

Thank you.

 

[Wabun]

Sure, it is very simple, I use the legacy with FTP, this works as follow, it copies one account to /backup, tars it and use FTP to send to off-site FTP server, but it WAITS, till the FTP session is done then it starts with the next account.

The new system, doesn't care about what FTP is doing, it backups one after the other account in /backup, now if you have enough space, fine, but if your space is limited your server crashes as it runs out of space...

This VPS is 50 GB, it has 36 accounts, and backing it up is no problem at all with the legacy ftp system, if we use the new system the server crashes!!!

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 47G 22G 23G 49% /
tmpfs 939M 0 939M 0% /dev/shm
/dev/sda1 239M 101M 126M 45% /boot
/dev/sda2 2.0G 3.3M 1.9G 1% /tmp

 

[cPanelMichael]

Sure, it is very simple, I use the legacy with FTP, this works as follow, it copies one account to /backup, tars it and use FTP to send to off-site FTP server, but it WAITS, till the FTP session is done then it starts with the next account.

Hello,

Thank you for clarifying. This functionality is not currently supported, but I encourage you to vote and add feedback to the existing feature request for this at:

Remote FTP in the new backup system should pack and upload one file at a time

Thank you.