Incremental Backups taking up too much space

[adam987123]

Hello

I am a bit confused about Incremental backups. I am using them on my server which is 100GB. The backup is about 14GB in size. I have incremental backups set up daily with retention for 30 days.

I find that after 9 days I am now running out of space (backups are using 50GB of space) - How is this possible?

My only thoughts are that the incremental backups create a full backup each week? Is this correct? If so can I disable this?

Or if not what could be happening?

Any help or advice will be appreciated.

Many Thanks
Adam

 

[24x7server]

Hi,

If you carefully check what you configured you will see what is the cause of this: You have set daily backup with retention for 30 days, so every day backup will be retained. Now if you have a 14GB of backup, then each day it will do backup and store it separately as you configured for 30 days. It will keep 30 days backup, so 14GB x 30 days should be the disk space it will occupy.

You may got confused youself with the backup. Retention will retain all the days backup and not overwrite any backup. it will overwrite only after 30 days..

 

[adam987123]

Hi,

If you carefully check what you configured you will see what is the cause of this: You have set daily backup with retention for 30 days, so every day backup will be retained. Now if you have a 14GB of backup, then each day it will do backup and store it separately as you configured for 30 days. It will keep 30 days backup, so 14GB x 30 days should be the disk space it will occupy.

You may got confused youself with the backup. Retention will retain all the days backup and not overwrite any backup. it will overwrite only after 30 days..

Hi

I get this but it is incremental backups. It is only suppose to save the files that have changed each day.

This is the first time i have used incremental backups so so i am not sure if this is correct.

I don't think the documentation is very good for this and i am a little confused with how it works?

 

[cPanelMichael]

Hello,

The disk space will appear to be the same, but it's not actually duplicating the space. It's simply the nature of how hard links work. This is described in more detail on the following thread:

How do the backups work exactly?

Let us know if this helps.

Thanks!

 

[adam987123]

Hello,

The disk space will appear to be the same, but it's not actually duplicating the space. It's simply the nature of how hard links work. This is described in more detail on the following thread:

How do the backups work exactly?

Let us know if this helps.

Thanks!

Thak you for the reply.

So are you saying that when I go to Show Current Disk Usage on my control panel this is not the correct figure? I am also getting DISKWARN email from Cpanel should this be ignored?

Does this also mean if I use the new feature in Cpanel 66 where I can save incremental backups to a remote location (in my case AWS S3) this will use the full disk space or will this just save the changed files?

When I delete all backups the total disk usage is 23GB after 9 days the 100GB disk is almost full.

 

[uk01]

Same issue here, incremental is using all our backup space and doing full backup every day...
Used 1tb bandwidth in 2 days!

 

[24x7server]

Hi

I get this but it is incremental backups. It is only suppose to save the files that have changed each day.

This is the first time i have used incremental backups so so i am not sure if this is correct.

I don't think the documentation is very good for this and i am a little confused with how it works?

You did not seem to be having the concept of the increatment and retention of backup understood completely.

Suppose you have a daily backup configured with 1 retention, so on Monday a backup is performed of 14 GB, on Tuesday agai backup will start, but since you have 1 retention, it will stat overwriting the previous backup and add the files those ar new and only increase it to suppose 1 GB, so making it 15 GB and this goes for a week and at the end of the week you only have 1 daily backup configured (Incremental), so the data all over the week go on syncing into the same backup making it 14+7 21 GB.

Now, what you have done is you create more retention, take for example you made 7 retentions of daily backup, so a backup will done on Monday of 14 GB, on tuesday it will take fresh backup and so on and at the end of the week, you will have 7 retention configured and that will have monday to sunday backup 14x7 = 98 GB of data. and when the next week comes on Monday backup will again start, but since you only had 7 days backup retention and 7-day backup is complete, it will overwrite the first backup i.e, of Monday, then next day , it will change the tuesday backup, so at the end of each week you will have 7 backups separte...

And now back to your scenario, you have configured 30 days retention, now you can calculate what will happen..
14x30 = 420 GB..

I hope your concept about retention is clear now..

 

[cPanelMichael]

Hello,

Internal case CPANEL-15493 is open to address an issue where the removal of the local backup directory fails after a successful transport to a remote destination when incremental backups are enabled in cPanel version 66. I'll monitor this case and update this thread with more information as it becomes available.

Thank you.

 

[tui]

Hello,

Internal case CPANEL-15493 is open to address an issue where the removal of the local backup directory fails after a successful transport to a remote destination when incremental backups are enabled in cPanel version 66. I'll monitor this case and update this thread with more information as it becomes available.

Thank you.

Hello, any update on this case?

 

[cPanelMichael]

Hello, any update on this case?

A resolution is in testing, but there's currently no time frame available for the resolution's publication. I'll continue to monitor this case and update this thread with new information as it becomes available.

Thank you.

 

[tui]

A resolution is in testing, but there's currently no time frame available for the resolution's publication. I'll continue to monitor this case and update this thread with new information as it becomes available.

Thank you.

I see that cPanel V66 is now in Stable tier... I do not think this version should have been put on this tier with this big flaw ... all people expecting this release to use incremental remote backups will fill their local and backup hard drives in a few days and all their servers will crash... this is not responsible for your part

 

[brt]

The problem I am experiencing isn't that backups are using too much space, but my remote backup directories (daily, etc.) aren't being completely removed. They're mostly empty, but almost all have 1-3 accounts remaining in them -- usually large accounts with a ton of files.

I presume it's timing out?

 

[cPanelMichael]

The problem I am experiencing isn't that backups are using too much space, but my remote backup directories (daily, etc.) aren't being completely removed. They're mostly empty, but almost all have 1-3 accounts remaining in them -- usually large accounts with a ton of files.

There are separate cases open referenced on the following thread that might relate to this specific issue:

Remote incremental backups - timeouts

Thank you.

 

[tui]

Any update on this?

 

[cPanelMichael]

Any update on this?

There's no new information to remote on internal case CPANEL-15493 at this time. I'll continue to monitor the status of this case and update this thread with more information as it becomes available.

Thank you.

Update: The resolution associated with internal case CPANEL-15493 is scheduled for inclusion with cPanel version 68.

 

[cPanelMichael]

Hello,

To update, internal case CPANEL-15493 is included with cPanel version 66.0.26:

Fixed case CPANEL-15493: Make sure incremental dirs are removed when asked.

Thank you.

 

[tui]

Excellent, let me test it!

 

[tui]

Hello again,

Now that issues are resolved and a temporary workaround is available for CPANEL-15398 i enabled the incremental remote backups for my servers, and it seems that everything is working as expected, i just have a few questions:

When i go to my backup server, i could see that backups are there and space is incremental as you can see in follow lines of the backup folder of one server, all other servers backups folders has the same structure and same pattern of folder size (first backup folder is bigger than then following days folders):

[email protected]:/home/serverX# du -csh *
6.5G    2017-10-24
305M    2017-10-25
530M    2017-10-26
510M    2017-10-27
8G     total

The first folder is the first full backup (its needed for incremental, obviously) as you can see is the largest folder and is the same size as the sum of the sizes of the accounts on the server, so the size is correct and its full, then the folders of following days are smaller because only files with changes are backed up; i have 6 days of retention, it's two days before this is accomplished, so my question, What will happen on day 7 with the largest folder? this has to be removed, but what about all the original data that has not been modified on other days? Will the data of day one be copied to the folder of day two and so on? Or how does this will work?

Thanks in advice

 

[cPanelMichael]

Hi @tui,

The original files are moved to the existing backup directories once their associated backup directory is pruned. You can read more about how this works at:

How do the backups work exactly?

Thank you.