Help me understand if {cp}backup can do what I want

pla

Member
Oct 26, 2012
14
0
1
cPanel Access Level
Root Administrator
Hi

About 8 months ago I took over two cpanel machines - my first experience of cpanel. Backups seemed to be happening as my predecessor intended them to - destination is a ReadyNAS sshfs mount - so I left things alone. Now I want to change the way our backups work but it looks lke {cp}backup can't be made to do exactly what I want. Or maybe it can but reading lots of threads about how it works has left me too confused to figure things out.

The first thing I want to do is to use rsync to get data to the NAS. Should be simple enough. Set target to a directory on the local drive and install a postcpbackup script to rsync between local drive and NAS. This will vastly cut the traffic between the two boxen. The second thing I want to do is use rsync's --backup and --backup-dir so each day I get the equivalent of a full backup and a directory of differences between that and the previous backup. All standard rsync stuff. Except the way cpanel backs things up seems to be inimical to an rsync solution.

The daily backups are incremental. So apparently I can't just rsync a daily backup over and get anything useful. The weekly backups are full backups but use tarball each account. I can make the gzip use --rsyncable or disable gzip completely (except for system configuration which uses tarballs anyway) and rsync will then transfer data efficiently but weekly backups still tar each account, which means a one-byte change in a small file of an account can cause a large tarball to be put into the --backup-dir.

It's not clear to me if daily backups are actually full backups if incremental is disabled. In fact, from contradictory information I've encountered it's not clear to me that daily backups are not full backups if incremental is enabled. Certainly the presence of cpbackup/weekly/daily implies they're always full backups, and the fact that we have incremental enabled and the daily inside weekly has the structure of incremental (no tarballs). If daily incrementals are actually full backups then I can happily rsync those and get what I want.

Confusing the issue is that our weekly backup is about twice the size of a daily backup. But in an older thread in the forums I've seen it said that nothing gets deleted from a weekly backup, even if something is deleted from a user account.

Another problem I have is MySQL. It would be nice to have account databases in with the appropriate account (which can be enabled) so that everything needed to restore an account is in one place and then the rest of the databases (particularly mysql) in the system configuration backup. But if I want the convenience of account database in the account backup and the safety net of mysql db in system backup, I have to have all the mysql dbs in the system backup too.

Oh, and it looks like you're using mysqldump for account dbs but when you backup all dbs to system area you're using mysqlhotcopy. Except perhaps you're doing something naughty because other threads here state that if a db is heavily accessed whilst being backed up it can be corrupted.

Note: I know the way cpanel backups work is designed to facilitate restoring accounts and moving accounts to another server. I understand that what I'd like to do would make restoring an account a little harder. I can live with that.

Note: I'm not making any type of feature request here. I want a solution that satisfies me fairly quickly. A feature request is likely to take to long and not fully satisfy me.

So can somebody explain how these things work and/or correct the errors in my understanding? That way I can figure out the best way to move forward.

--
Paul
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,261
463
Hello :)

Are you using the new backup system introduced in cPanel version 11.38, or are you using the legacy backup system? I wanted to clarify this before answering any of the questions so I know which backup system you are using.

Thank you.
 

pla

Member
Oct 26, 2012
14
0
1
cPanel Access Level
Root Administrator
I'm willing to try either way.

Our current production servers are using legacy/incremental/no hard linking. I'm happy enough with legacy except for the worry that one day it will no longer be there. Right now I'm commissioning a new server so have the freedom to experiment for a while.

--
Paul
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,261
463
The primary difference with incremental backups is that only changed information is saved. Incremental backups are not stored in an archive like standard backups are. However, the same set of data should be preserved no matter which type is selected. The new backup system does not include a weekly backup option. Instead, you can retain multiple daily backups and/or choose to enable monthly backups on the 1st and 15th of the month. It's not currently possible to select specific databases when you select an option that backs up the entire MySQL directory. I noticed you mentioned that feature requests are not an option for you, but if you change your mind, you can open one for this feature at:

Submit A Feature Request

It's true that a database may not be backed up if the database files are being accessed at the time of the backup. The backup logs (/usr/local/cpanel/logs/cpbackup/) will indicate if a database was not included in the archive.

You may need to consider a third-party backup service such as R1Soft if the existing backup system does not meet all of your requirements.

Thank you.
 

pla

Member
Oct 26, 2012
14
0
1
cPanel Access Level
Root Administrator
Hi Michael

Another verbose reply from me so you know why I'm asking more questions. To let you easily spot what I'd like answered, the questions are in red.

The primary difference with incremental backups is that only changed information is saved. Incremental backups are not stored in an archive like standard backups are. However, the same set of data should be preserved no matter which type is selected. The new backup system does not include a weekly backup option. Instead, you can retain multiple daily backups and/or choose to enable monthly backups on the 1st and 15th of the month.
Having looked more closely at what is being generated it would seem that putting * in /etc/cpbackup-exclude.conf gets me close enough to what I need that I can take the rest from there. That appears to give me a backup of the metadata about an account I would need to restore it without generating a wasteful copy of /home/account in a format that is not useful to me - I can do a straight rsync of /home/account instead. Please let me know if excluding /home/account/* would lose me any metadata needed to recreate an account.

Come to that, do I need that per-account metadata in or is it all duplicated in the system backup? I understand that if I needed to restore an individual account it might take a lot of manual effort to restore that account's settings from the system stuff, but I can live with that.

I noticed you mentioned that feature requests are not an option for you, but if you change your mind, you can open one for this feature at:
Submit A Feature Request
It would take too much time with no guarantee that at the end of the process I'd get something capable of doing exactly what I wanted. Much faster and much more likely to result in something that makes me happy is to do it myself, leveraging as much of your backup system as I can. The only feature request I might put in, depending on your answer to a later question, would be about disabling backups of databases both per-account and "directory."

It's true that a database may not be backed up if the database files are being accessed at the time of the backup. The backup logs (/usr/local/cpanel/logs/cpbackup/) will indicate if a database was not included in the archive.
Let me see if I understand this. For client backup of databases, if the client ever bothers to do one (and most do not) you use mysqldump (or mysqlhotcopy or something like that) to get a dump that is almost certainly going to be usable. For system backup of databases, which on most servers will happen daily) you use cp -r (or rsync or something like that) which will be more likely to be unusable. Which is not nice, but at least I have the option of pulling a working copy from the client backup (if I happened to turn on the duplicative, wasteful option of dumping dbs both ways). Except I can't do that for the mysql database itself. So I have to trawl the logs every day to see if the mysql db was backed up OK.

Even if every other aspect of your backups satisfied my every need, the way you handle mysql backups means I'd have to write something to do that particular job properly. Does the MYSQLBACKUP option in /etc/cpbackup.conf have a setting that means "don't back up any databases in any way"? Because after I write something to do the job properly neither per-account dumps nor all-db breakable copies would do anything for me except waste disk space and processor time. If there isn't currently such an option I'll raise a feature request and live with the waste for now.

You may need to consider a third-party backup service such as R1Soft is the existing backup system does not meet all of your requirements.
It may do things differently from your way but is it close enough to what I want to let me do it my way? Probably not. And after looking at their site I'm not inclined to waste time on trying to find out.

--
Paul
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,261
463
Please let me know if excluding /home/account/* would lose me any metadata needed to recreate an account. Come to that, do I need that per-account metadata in or is it all duplicated in the system backup?
Excluding the data in the account's home directory by using the wildcard character in /etc/cpbackup-exclude.conf does not exclude the data necessary to re-create the account. The cPanel userdata and all additional configuration files related to the account are still backed up. Thus, restoring the account and then transferring the account's home directory back into place is a viable solution. You do still have to backup the individual accounts because this data is not included as part of the configuration file backup data, and utilizing it manually from the file system would involve extensive modifications to several configuration files.

Does the MYSQLBACKUP option in /etc/cpbackup.conf have a setting that means "don't back up any databases in any way"?
No, it's not possible to exclude databases completely. At least “Per Account” is needed to use the restore feature.

Thank you.