incremental backups creates dated directory

JTZEnterprise

Registered
Mar 24, 2009
3
1
51
We have changed to incremental backups on two different servers. One of them still creates a new subdirectory in the /backups directory named by the date. The other creates a subdirectory named, "incremental". We have noticed that the one creating a dated subdirectory is doing a full backup every day and not an incremental backup. How can we get this server to store the backups in a subdirectory named, "/incremental" and not a dated file like "/2017-04-18"?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Hello,

We added support for multiple incremental backups in cPanel 64. Since the backup directory structure changed as part of this new functionality, the most recent incremental backups before the update to cPanel 64 are automatically preserved to avoid data loss. As of cPanel 64.0.11, the "/backup/$weekly/incremental" directory is renamed to "/backup/$weekly/$date" when updating to cPanel 64. At this point, the backup system should follow your current retention values (e.g. if you configure the server to retain 1 weekly backup, then it should automatically remove the converted "/backup/$weekly/$date" directory once the new weekly incremental backup is generated.

Thank you.
 

JTZEnterprise

Registered
Mar 24, 2009
3
1
51
I am referring specifically to the daily backup. We would prefer the daily backups not be stored in a $date subdirectory. We have a separate function that will rsync the daily backups to another server but because the directory date changes with each backup from cpanel, the rsync is not able to incrementally move the files to the other location.
 
  • Like
Reactions: Optimizr

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Hello,

The use of the date in the directory name is part of the new functionality designed to offer multiple incremental backups. You would need to update your script to have rsync consider the directory name based on the date. Here are a couple of StackOverflow threads you may find helpful:

How to rsync a directory to a new directory with different name?
Using rsync to maintain a copy of a directory with a name that changes

Or, you may want to setup a hook that runs after each backup generation that runs a custom bash script to create a symbolic link to the previous incremental directory name. Note that support for remote incremental backups is planned for cPanel 66:

Support for remote incremental backups

Thank you.
 

cesarlwh

Well-Known Member
Sep 23, 2008
63
0
56
I am having the same problem! THe backup system are creating a new backup every day and not updating the current! Then is not a incremental and I need a incremental. My HDD of backup are full because of this!

See:
/backup/2017-04-11 - OK
/backup/2017-04-12 - OK
/backup/2017-04-13 - OK
/backup/2017-04-14 - Error because the HDD are full
/backup/2017-04-15 - Error because the HDD are full
/backup/2017-04-16 - Error because the HDD are full

The correct is move the old date dir to the new date and then make a incremental backup.

I need a solution to this ASAP because my servers are without backups.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Hello @cesarlwh,

You may want to review the backup logs in the /usr/local/cpanel/logs/cpbackup/ directory to see if the backups are partially failing (you can search for error or failed messages in these logs with the "grep" command). If so, you may want to adjust the retention settings for failed backups at:

Backup Configuration - Version 64 Documentation - cPanel Documentation

Or, free to open a support ticket using the link in my signature so we can take a closer look.

Thank you.
 

north captiva

Registered
May 4, 2017
2
3
3
Florida
cPanel Access Level
Root Administrator
Hello,

The use of the date in the directory name is part of the new functionality designed to offer multiple incremental backups. You would need to update your script to have rsync consider the directory name based on the date. Here are a couple of StackOverflow threads you may find helpful:

How to rsync a directory to a new directory with different name?
Using rsync to maintain a copy of a directory with a name that changes

Or, you may want to setup a hook that runs after each backup generation that runs a custom bash script to create a symbolic link to the previous incremental directory name. Note that support for remote incremental backups is planned for cPanel 66:

Support for remote incremental backups

Thank you.
With notice that the legacy backup will soon be retired, it would be a really nice option if cpanel would include an option to automatically create a /daily or /latest symlink for us to the latest dated 2017-xx-yy daily backup directory when the new backup script runs. This would make it much easier for those of us who have other backup routines rsyncing the /daily directory to remote backup locations seamless. Please... please...
 
  • Like
Reactions: cPanelNick

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
With notice that the legacy backup will soon be retired, it would be a really nice option if cpanel would include an option to automatically create a /daily or /latest symlink for us to the latest dated 2017-xx-yy daily backup directory when the new backup script runs. This would make it much easier for those of us who have other backup routines rsyncing the /daily directory to remote backup locations seamless. Please... please...
I encourage you to submit a feature request using the Submit A Feature Request link if you'd like to see this feature added.

Thank you.
 

mahinder

Well-Known Member
Jun 12, 2003
69
0
156
matrix
How can we configure cPanel "not" to create different date directories with incremental backups every day. Some of the systems I manage, backup can only be taken for 3-4 hours in a day on particular days and it has to be monitored for server performance reasons. Some times backup has to be killed in middle and restarted when it pushes server load very high.

In new backup system, backups are splitting in multiple date folders and I have to rename yesterday's backups folder before starting backup to prevent backup script from starting backup all over again.

Old incremental backups system was simple and straight forward. /incremental/accounts folder was just updated with new files and there was no multiple date folders.

In my opinion cPanel should provide a check box which gives user control to keep backups in just one folder instead of creating multiple folders every day.
 

InterServed

Well-Known Member
Jul 10, 2007
268
14
68
cPanel Access Level
DataCenter Provider
Hello,

I did "suffer" quite a lot by this change as well. Here is how i managed to work it out so i can still take rsync copies of the local generated incrementals (just 1 day retention locally as i take daily rsync of this and keep copies offsite):

/usr/local/cpanel/scripts/postcpbackup (adjust based on your requirements, like BKP_PATH and BKP_LINK)
Code:
#!/bin/bash
BKP_PATH=/backup
BKP_LINK=/lastbackup

BKP_DIR=$(find ${BKP_PATH} -maxdepth 1 -type d -not -name "*lost*" | sort -r | head -1)
if [ "${BKP_DIR}" == "${BKP_PATH}" ]; then
  exit 0
else
  BKP_DATE=$(basename ${BKP_DIR})
  [ ! -f ${BKP_DIR}/${BKP_DATE} ] && touch ${BKP_DIR}/${BKP_DATE}
  [ -L ${BKP_LINK} ] && rm -rf ${BKP_LINK}
  ln -s ${BKP_DIR} ${BKP_LINK} || exit 1
fi
1) .Make sure that the script has execution right
Code:
chmod +x /usr/local/cpanel/scripts/postcpbackup
2) Make sure you have enabled the use of the post script (else edit the file and set it to yes) :
Code:
[email protected] [~]# grep -i POSTBACKUP /var/cpanel/backups/config
POSTBACKUP: 'yes'
Using this , i can sync the /lastbackup/ in my case as it will always be a symlink to the latest cpanel backup. I recommend to read the script and to understand it before using it. I'm not responsible for any damage it may cause if you do not understand it.
 
Last edited:
  • Like
Reactions: kdean

holodyn

Member
Feb 6, 2008
5
1
53
This ruins incremental backups for us. We pair backup drives with usage 1-1, and the previous incremental allowed for that. This change requires us to consume large amounts of cache space with every backup, as all backups are created before cleanup is performed.

Please restore the option to have the incremental update perform a sync instead of a whole new copy so that we aren't required to have TRIPLE the disk space for a backup drive and be forced to slam the system with an entire copy / delete cycle every day.

For anyone who wants to re-create the previous incremental operation you can use a script like the following.

```
#/usr/bin/bash
cd /var/cpanel/users
for i in `ls *`
do
if [ -d "/home/$i" ]; then
/usr/local/cpanel/bin/pkgacct --incremental "$i" /backup/daily-incremental/accounts backup
fi
done
```