Incremental backup takes longer every day

stormy

Well-Known Member
Nov 22, 2003
167
19
168
Spain
cPanel Access Level
Root Administrator
I think there's a bug in cPanel 60 that causes the incremental backups to take longer every day, until they take more than 10 hours. A server reboot cures this and restarts the cycle.

This doesn't happen to my servers on Cloudlinux 5/Cpanel 58 (which I'm in the process of migrating).

My experience:

We migrated a machine from CL5/cPanel 58 to CL7/cPanel 60. We also upgraded the hardware massively, including a move to full SSD RAIDs.

The first nightly backup took 1 hour, where the previous system was taking 6-7 hours. I was amazed at the difference the new hardware made. Little by little, the incremental backup started to take longer and longer until it was taking more than any of our old servers on non SSD hardware, with the same amount of accounts and data.

After a server reboot, backup time went back to little more than 1 hour. This cycle happened a couple times, each taking about 1-2 weeks.

Now here's the clincher. We migrated this same server to another datacenter, and left the old machine running, with all the accounts suspended, and no traffic in the server. Still, each night, the backups take longer, even with no new data being generated in the server at all!

Sadly, we are experiencing the same problem with the new hardware as well. These machines are very very powerful and their load is minimal.

How can we troubleshoot this?

Here's my backup info:

Code:
cat /var/cpanel/backups/config
---
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 0,1,2,3,4,5,6
BACKUPDIR: /backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'yes'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'yes'
BACKUPTYPE: incremental
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 1
BACKUP_MONTHLY_DATES: 1
BACKUP_MONTHLY_ENABLE: 'no'
BACKUP_MONTHLY_RETENTION: 1
BACKUP_WEEKLY_DAY: 0
BACKUP_WEEKLY_ENABLE: 'no'
BACKUP_WEEKLY_RETENTION: 4
ERRORTHRESHHOLD: 3
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 7200
MYSQLBACKUP: accounts
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'
The Tweak Settings for the backups have been changed to the cPanel defaults. "Extra CPUs for server load" was previously set to 5, and "I/O priority level at which nightly backups are run" to 3. Is it possible that somehow these settings haven't been applied?
 

stormy

Well-Known Member
Nov 22, 2003
167
19
168
Spain
cPanel Access Level
Root Administrator
The last backup of the machine that has no new data being generated, most services stopped, and load close to zero took more than 14 hours.

It would be great if this could be troubleshooted before I decomission the server.

Anyone else having backups that get slower every day?
 

ChrisI

Technical Analyst II
Mar 24, 2014
67
12
83
cPanel Access Level
Root Administrator
Hello!

The best way to begin troubleshooting that would be to check what is occurring in the backup logs at /usr/local/cpanel/logs/cpbackup/. You should be able to use the information and the times there to help pinpoint what is taking the most time during the backup.
 

stormy

Well-Known Member
Nov 22, 2003
167
19
168
Spain
cPanel Access Level
Root Administrator
Unfortunately, the logs don't solve the mistery. Some accounts, randomly, will simply start taking longer and displaying dots in the logs. It's not always the same account.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,226
463
Hello,

Feel free to open a support ticket using the link in my signature so we can take a closer look and see what's happening. You can post the ticket number here so we can update this thread with the outcome.

Thank you.
 

stormy

Well-Known Member
Nov 22, 2003
167
19
168
Spain
cPanel Access Level
Root Administrator
My datacenter has already opened a ticket. It's probably a memory leak related to Cloudlinux 7, and we've had it happen on more than one server already. I don' thave the ticket number though.
 

budway

Well-Known Member
Apr 16, 2003
189
0
166
Hi,

I noticed this too, on a server not running cloudlinux.

I thought it was client uploading more data and didn't care to dig in, but it has increased about 1/6 to 1/8 the time, recently.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,226
463
I thought it was client uploading more data and didn't care to dig in, but it has increased about 1/6 to 1/8 the time, recently.
Would you mind opening a support ticket using the link in my signature so we can take a closer look and see what's happening? You can post the ticket number here so we can update this thread with the outcome.

Thank you.