SOLVED Incremental vs Uncompressed and remote Rsync

[noox]

Two weeks ago I realized that Legacy Backup does not work any more. I used a special configuration where I fetched the backup on a remote server via rsync from my CPanel server. I heard from attacks where attackers deleted the remote backups first, so I thought it's better if the Cpanel server has no access to the backup server - just the other way around.

With the new backup system I first used incremental backup. But I had two issues:

• The backups varied in size (on some days e.g. 40 instead of 44 GB). When I looked some user accounts where missing on these days. Yesterday was very strange - only one account was in the backup directory. I looked up the backup log file but there was no error - just the log entries for this one account
• I have a post backup script which I changed to use the hook system. It adds read rights to the backup files for the backup user I use for the remote rsync login. On some days it worked, on some days it did not.

Seems like the incremental backup stops prematurely? But not every day. The incremental backup two days ago (with most likely all accounts) lasted for about 20 minutes. So that should not be an issue. When it backed up only one account it did this in less than a minute. (Legacy backup lasted 3 to 5 minutes.)


So yesterday I switched to uncompressed backup. But today I realized that uncompressed backup creates a .tar per account. Which is not ideal for remote rsync backup. With legacy backup my remote rsync transfer lasted for about 20 to 30 minutes. Last night with the .tar files it lasted almost two hours.


Do you have any suggestions for my problem:

• Does anybody else have problems with incremental backups not finishing properly?
• I had set the retention for incremental backup to three per day, week and month. I now try it with one each.
• Is it possible to disable creation of the .tar files for uncompressed backup so that it just copies the files and keeps it in the file tree
• Or should I untar the files in a post backup script.

Thanks!

Hint: The local backups are stored on a second pair of hard drives with more than enough space (1.2 TB available). On the remote backup site I have a retention logic with multiple daily, weekly and monthly backups.

 

[cPanelLauren]

• Does anybody else have problems with incremental backups not finishing properly?

Generally, I've not seen many issues with the incremental backups but if there is some issue, it will be noted in the backup logs at /usr/local/cpanel/logs/cpbackup/

• Is it possible to disable creation of the .tar files for uncompressed backup so that it just copies the files and keeps it in the file tree
  Or should I untar the files in a post backup script.

I'm not sure I understand what the issue with rsync with a .tar file. is exactly. Can you elaborate on how this is problematic?


Lastly, and I want to stress, I feel like the focus here should be on identifying the issue you're having with incremental backups. But if you're not wanting to perform an rsync with the .tar file you would have to implement a postbackup script to untar the file as there's no way to set this in the configuration.

 

[noox]

Thank you for your response!

I'm not sure I understand what the issue with rsync with a .tar file. is exactly. Can you elaborate on how this is problematic?

Because there are a lot of unchanged files in the accounts. E.g. some GBs of images. If the accounts are within a tar archive, alle data has to be transferred to the remote backup location. If I have the plain files, only the changed files have to be transferred which is a huge difference.


I now found out that something weird is going on and I have to investigate that further. I never had an issue with the legacy backup. It ran at 3 am. The new at 2 am. I now noticed that at the time when it stopped (without any error in the log) there are a lot of messages in /var/log/messages about starting services. But these messages startet even before the last backup log entry. I also ping this server from my smarthome hub every minute and I noticed one failed ping each day at the time the backup stopped. But I did not see this pattern before.

One or two weeks ago I switched to uncompressed backup and untared the files in a post backup hook script. Mostly it stopped while executing this post backup script. But sometimes during the backup itself. For two days now I have changed the backup time and made an extra cron job for the untar process. So far it worked.

One one hand it does not seem the backup itself is the problem. But on the other hand I did not see the random ping errors while the backup is running before I switched to the new backup process.

 

[noox]

Just to close this issue:

I finally found out that I had random reboots when the backup was running. Maybe two to three times a week. The backup type did not matter. So it also happend with uncompressed backup. I also tried to change the scheduled backup time. I found no log entries which indicated the cause of the reboot. I read that this may indicate a hardware defect.

But it seems like a kernel update resolved the issue. No backup issues for 1.5 to 2 weeks now.

 

[cPRex]

@noox - that's great news! Well, at least about the part of cPanel and the backup system working well, but not about possible hardware issues. Although if updating the kernel took care of things and it has been stable for two weeks now, it sounds like things are working well!