Too many backups retained

[Andrew Fraser]

/etc/redhat-release:CloudLinux Server release 6.7 (Aleksei Gubarev)
/usr/local/cpanel/version:11.56.0.24
/var/cpanel/envtype:kvm
CPANEL=release
Server version: Apache/2.2.31 (Unix)
Server built:   Feb  6 2016 15:41:26
Cpanel::Easy::Apache v3.32.9 rev9999 +cloudlinux
PHP 5.4.45 (cli) (built: Feb  6 2016 15:47:42)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
    with the ionCube PHP Loader v4.7.5, Copyright (c) 2002-2014, by ionCube Ltd., and
    with Zend Guard Loader v3.3, Copyright (c) 1998-2013, by Zend Technologies
mysql  Ver 15.1 Distrib 10.0.24-MariaDB, for Linux (x86_64) using readline 5.1

I am using backups through WHM with the following settings:

-- Legacy backups disabled
-- Backups enabled, compressed, max destination timeout 2700 seconds, max restore timeout 21600 seconds
-- Backup Weekly Saturday Retain 1 Weekly backup. Daily and Monthly backups disabled.
-- Backup Accounts. Disable suspended and access logs. Bandwidth data enabled. Local DNS disabled.
-- Backup system files.
-- Databases per account and entire MySQL directory.
-- Backup directory /backup.
-- Retain backups in the default backup directory.
-- Mount bacup drive as needed: disabled
-- Additional destinations: Amazon s3 (works fine)

The problem is, the system is retaining more backups than I am telling it to, and I don't understand why. Was cleaning up the server a few weeks back and noticed that there were 3 weekly backups in the /backup directory, when WHM was configured to have 2. I deleted one and told WHM to retain only one weekly backup instead, hoping that it meant how many "old" backups, not how many backups total (we want to have 2 backups total).

The next week I checked after the backup and it had 3 backups again. We only want 2! I have it set to 1, which is as low as it will let me set it (I can't set it to 0). What am I doing wrong here and why is it keeping 3 backups no matter what I tell it to do?

Not sure if this is relevant, but after some Googling, I also noticed that there is no "postcpbackup" file in the /scripts directory. Is there supposed to be and might this be the reason my backups are not being cleared?

Thanks in advance for any help.

 

[cPanelMichael]

Hello,

1. You mentioned enabling the following options:

-- Retain backups in the default backup directory.
-- Additional destinations: Amazon s3 (works fine)

This option retains backups in the default local backup directory, even after the system transfers the backups to another destination. Could you disable this option, remove the extra backup directory, and then force a new backup generation to see if the issue persists?

2. Backup hooks are configured using the standardized hook system:

Guide to Standardized Hooks - System Functions - Software Development Kit - cPanel Documentation

There's a thread on this at:

rsync after backup ends

Thank you.

 

[Andrew Fraser]

Sorry, could you just clarify what you mean by "remove the extra backup directory?" As far as I know, I don't have an extra backup directory.

 

[cPanelMichael]

You mentioned that "Retain backups in the default backup directory" is enabled in "WHM >> Backup Configuration". This option retains backups in the default local backup directory, even after the system transfers the backups to another destination. Disabling this option will ensure the backup is not retained on the local cPanel server after it's copied to the remote destination.

Thank you.

 

[Andrew Fraser]

Maybe I am not understanding you, but I don't think that will solve my problem. We would like to retain two local backups in addition to our Amazon S3 backups. The problem is that we do not want to retain any more than two local backups because it takes up too much disk space. Regardless, setting the number of backups to retain in WHM does not seem to affect the number of backups that are actually retained.

Please excuse me if I am misunderstanding something that you are saying, but it sounds to me like you are suggesting that we don't retain any local backups at all. Retaining backups on the local cPanel server is something that we want to do. The end goal is that, in addition to our S3 backups and our other backup processes, we would like to retain 2 local backups on the server and no more than that. The problem is that whatever I enter into WHM, it is keeping 3 weekly backups of all accounts anyway.

 

[cPanelMichael]

Hello,

I replicated your backup configuration on a test system, and here are the results:

• Two backup copies were retained in /backup/weekly

• One backup was retained in /backup/2016-06-29

The additional backup in /backup/2016-06-29 stems from the use of the following option:

"Retain backups in the default backup directory"

Keep in mind that backups are still stored on the remote destination when this option is disabled. Disabling the option doesn't disable remote backups, but rather it disables the copying of that backup to /backup/ on the cPanel server.

That said, if I understand correctly, you have three backups stored in /backup/weekly. Could you verify if that's the case? If so, remember that backups for weekly storage are only pruned on the days weekly backups are configured to run.

Thank you.

 

[Andrew Fraser]

Hm. Now I am confused. The backup runs on Saturday. Last Monday when I came in to the office, I checked the backups and there were three in /backup/weekly even though I had it set to retain two. I manually deleted one of the backups from command line to free up disk space. Then I changed the number in WHM to one in the hopes that it would then retain only two backups. Then, this Monday I came into the office and checked again and there were three backups in /backup/weekly, even though I had only told WHM to retain one. Again, I deleted the oldest backup directory manually from command line to free up space, leaving two backups.

I have not experienced any problems with having an extra backup being retained in /backup/2016-XX-XX. They have only ever been in /backup/weekly.

Now, because you asked, I just went to /backup/weekly to verify that there were the same number of backups there, and interestingly enough, no there aren't. There is only one backup there, which is what I have told it to do in WHM. The last time I checked (on Monday morning) there were definitely two (three before I deleted one of them). I am not sure what is going on. Perhaps it is because the backup process had not fully completed when I came in on the second Monday? It is a dedicated server with close to 300 accounts (something like 700GB total data being backed up), so it is possible that the process could take a very long time, depending on connection speeds, etc. I suppose it is possible that the first week when I checked it had completed, and the second week it hadn't? We also run another nightly backup process to another server using rsync, so there is a lot of bandwidth being used at once when the backup goes to S3 and that other server.

Regarding this:

Keep in mind that backups are still stored on the remote destination when this option is disabled. Disabling the option doesn't disable remote backups, but rather it disables the copying of that backup to /backup/ on the cPanel server.

If I disable this option, will I still retain the copies of the backups in /backup/weekly? When you say "the remote destination," I understand that you mean, in this case, S3. We would like to have backups on S3 for redundancy and archival purposes, and the two weekly local backups that don't require us to transfer in order to restore, in addition to our other nightly backups. Is that what should happen if I disable the option you're talking about?

 

[cPanelMichael]

If I disable this option, will I still retain the copies of the backups in /backup/weekly? When you say "the remote destination," I understand that you mean, in this case, S3. We would like to have backups on S3 for redundancy and archival purposes, and the two weekly local backups that don't require us to transfer in order to restore, in addition to our other nightly backups. Is that what should happen if I disable the option you're talking about?

Hello,

It does not affect the transfer or retention of backups on the S3 instance. The option affects the local storage of an additional backup archive on the cPanel server. This is explained in more detail in the "Retain backups in the default backup directory" section at:

Backup Configuration - Documentation - cPanel Documentation

Thank you.

 

[Andrew Fraser]

I appreciate your time in trying to help me out here. Unfortunately, I think that you are not understanding the nature of the problem that I am having. (I am still having the same problem).

Even though it was clear to me that you were misunderstanding my problem, I did what you suggested so that you wouldn't think I was being stubborn or not listening to you. I left it that way for a few weeks so that I could post the results here and explain to you how the results are different from what I would like to do. Unfortunately what happens if I do what you suggest is, no backups at all are retained locally. That is exactly what I thought would happen, and it is not at all what I want to do.

Ignore for a moment that I am sending backups to s3. The problem doesn't have anything to do with s3. I want to keep 2 weeks of local backups on our server, but it is keeping more than that.

To make it clear the problem I'm having, here is a screen cap:

i.imgur.com/0IOhSU1.png

(Mod note: Please attach images to your posts)

No matter what I do -- and I have tried basically everything I can think of -- CPanel / WHM backups are not honouring the "retain ___ weekly backups" option that is in WHM's backup configuration. No matter what number is in that box, it doesn't matter. Only one of two things happens:

Case A: If I check the box "Retain backups in the default backup directory" then it will retain all backups indefinitely, without ever deleting any of them. So, for example, if I set the box to "2," then 2 weeks pass, and I have 2 weekly backups on the server. Then the third week passes, and I have 3 backups on the server. On the fourth week, I have four backups on the server, etc. This continues on forever, and the only way to remove the local backups is to delete them myself.

The same is true no matter what number I put in the "retain x backups weekly" box. It is as if that box just doesn't do anything whatsoever.

Case B: If I follow your suggestion and uncheck the box "Retain backups in the default backup directory" then it retains no local backups. I do not want this. I want to retain the number of backups that I told WHM to retain.

To reiterate, I would like you to ignore anything I have said about s3 because that is not the issue. s3 is working fine, and I am not having any problems with the s3 functionality. It is the local backups that are an issue.

I can write my own script and set it to run on a cron job to delete the old backups, but that seems to me like it shouldn't be necessary, since clearly you are supposed to be able to tell WHM how many backups you would like it to retain. It would be nice if WHM just took care of it, since it seems to me like that is what it is supposed to do. I cannot believe that this checkbox is supposed to do nothing, and that WHM is doing what it is supposed to do here. There must be something wrong with my configuration or my server setup, or something else that I can look into to figure out why this isn't working.

Any ideas what I could try here before writing my own script and dealing with it myself? It's disappointing to say the least, and I'm still hoping you can maybe point me in the right direction as to what may be causing this problem.

 

[Infopro]

Case A: If I check the box "Retain backups in the default backup directory" then it will retain all backups indefinitely, without ever deleting any of them.

Backups are saved by date and rotated out. I backup daily, and retain 4 Daily backups. Those backups are saved locally and transported to a remote server as well. Local and remote backups show me the same sets of files as expected.

 

[Andrew Fraser]

Ok, sounds like maybe you are seeing what my problem is here. What you are describing is pretty close to what I would like to achieve -- in my case, I would like to retain 2 weekly backups locally. My problem is, unlike you, my backups are not being "rotated out" or deleted. They are just being retained forever unless I delete them manually. What I am trying to understand is *why??*

I am not worried about s3 -- it is there for redundancy. Doesn't matter if it is synced with the server or not, because the buckets expire after time anyway. If it is, then great. But all I care about is rotating out, or deleting the local backups on our server, so that we don't get notices about running out of space, and I don't have to delete the old backups manually or set up my own cron job. Any idea what might be the problem here?

 

[Infopro]

Your settings are using Compressed backups, correct?
When you look in your backup directory, are directories dated as mine are?
Your image shows you do not do any Daily backups, add one day, Saturday. Leave Weekly set as is.

Force a backup:

/usr/local/cpanel/bin/backup --force

 

[Andrew Fraser]

Yes, backup type is compressed.

So You are suggesting that I should leave the backup weekly to Saturday, and *also* set backup daily to Saturday. Okay. I will try this. I will force a backup before I head home at 5pm tonight. (backups can take a long time to complete as we have over 200 accounts on this server). Thanks.

 

[Andrew Fraser]

When you look in your backup directory, are directories dated as mine are?

And in answer to this question, yes, directories are dated.

 

[Infopro]

And in answer to this question, yes, directories are dated.

They should be rotating out. Keep us posted.

 

[Andrew Fraser]

FYI, we are still having this issue and urgently need to resolve it. I was under the impression that support for CPanel and WHM were included in the price we are paying for our hosting. However, perhaps I am pursuing the wrong avenue for support -- I will open ticket with our host as well.

None of the suggestions that have been made in this thread worked to resolve my problem, and in fact this suggestion made thnigs worse:

Your image shows you do not do any Daily backups, add one day, Saturday. Leave Weekly set as is.

What happens when I implement this suggestion is, two sets of backups are taken on the same day. Over the weekend this caused our server to run out of space and crash, leaving our 200+ clients without websites or email for several hours. I appreciate that you were trying to help, but I am wishing that I had listened to my gut and not followed this suggestion.

It is fairly obvious to me now that this is not a configuration problem. Something else is going wrong here -- we had about a year of backups being taken successfully with this configuration with no problem, and then earlier this year, the backups stopped being rotated out. If you have any idea what might be causing this, please let me know. I will also be chasing the issue down with my host, because I believe that there should be support for this native WHM feature, and I should be able to take backups as advertised.

 

[Infopro]

Please feel free to open a ticket to cPanel Technical Support so they can take a closer look at your issue for you.

 

[Andrew Fraser]

Thank you. I may do that if hosting support is unable to resolve the issue. What exactly is the process for opening a ticket with technical support?

 

[Infopro]

From within your WHM, top right corner is an icon that looks like a life preserver, on that menu is a link to Contact cPanel.

 

[Dhaupin]

Thank you. I may do that if hosting support is unable to resolve the issue.

Did you ever figure this out? We are encountering the same issue on out Cloudlinux 6.8 KVM VPS. That server crashed over the weekend due to disk space running out when WHM stopped obeying the 3 backup retention limit. It has been running without issue for years, we didn't change anything, but it seems sometime after v60 update last week more than 3 backups are now kept. I was working on the house all weekend in "amish offline mode" but luckily the host provider caught it before massive/extended outage or data loss.

We have another server that is standard CentOS 6.8 on virtuozzo [OpenVZ]. Same settings, also updated to v60 the same day, but the backup retention still works correctly.

Both servers are set as daily backups, compression on, 2700 timeout, retain 3 in total, store in /backup folder, legacy disabled (no settings were changed). The permissions look correct in /backup as well as the dated sub folders.

Does it have to do with Cloudlinux/Kernelcare perhaps? That's the only thing i see different besides the virtualization type of the instance.

EDIT: Looking in the most recent backup log, there were a mysql dump couple errors:
- There were 2 databases deleted via PhpMyAdmin, but were still "there" in the cPanel account, which backup mysqldump tried to look for
- The process finished with Final state is Backup::PartialFailure (0) im assuming due to that failed dump? Would that "abort" before deleting the old folder?

Thanks