The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Incremental backup takes longer every day

Discussion in 'Data Protection' started by stormy, Dec 24, 2016.

Tags:
  1. stormy

    stormy Well-Known Member

    Joined:
    Nov 22, 2003
    Messages:
    122
    Likes Received:
    6
    Trophy Points:
    168
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    I think there's a bug in cPanel 60 that causes the incremental backups to take longer every day, until they take more than 10 hours. A server reboot cures this and restarts the cycle.

    This doesn't happen to my servers on Cloudlinux 5/Cpanel 58 (which I'm in the process of migrating).

    My experience:

    We migrated a machine from CL5/cPanel 58 to CL7/cPanel 60. We also upgraded the hardware massively, including a move to full SSD RAIDs.

    The first nightly backup took 1 hour, where the previous system was taking 6-7 hours. I was amazed at the difference the new hardware made. Little by little, the incremental backup started to take longer and longer until it was taking more than any of our old servers on non SSD hardware, with the same amount of accounts and data.

    After a server reboot, backup time went back to little more than 1 hour. This cycle happened a couple times, each taking about 1-2 weeks.

    Now here's the clincher. We migrated this same server to another datacenter, and left the old machine running, with all the accounts suspended, and no traffic in the server. Still, each night, the backups take longer, even with no new data being generated in the server at all!

    Sadly, we are experiencing the same problem with the new hardware as well. These machines are very very powerful and their load is minimal.

    How can we troubleshoot this?

    Here's my backup info:

    Code:
    cat /var/cpanel/backups/config
    ---
    BACKUPACCTS: 'yes'
    BACKUPBWDATA: 'yes'
    BACKUPDAYS: 0,1,2,3,4,5,6
    BACKUPDIR: /backup
    BACKUPENABLE: 'yes'
    BACKUPFILES: 'yes'
    BACKUPLOGS: 'yes'
    BACKUPMOUNT: 'no'
    BACKUPSUSPENDEDACCTS: 'yes'
    BACKUPTYPE: incremental
    BACKUP_DAILY_ENABLE: 'yes'
    BACKUP_DAILY_RETENTION: 1
    BACKUP_MONTHLY_DATES: 1
    BACKUP_MONTHLY_ENABLE: 'no'
    BACKUP_MONTHLY_RETENTION: 1
    BACKUP_WEEKLY_DAY: 0
    BACKUP_WEEKLY_ENABLE: 'no'
    BACKUP_WEEKLY_RETENTION: 4
    ERRORTHRESHHOLD: 3
    GZIPRSYNCOPTS: --rsyncable
    KEEPLOCAL: 1
    LINKDEST: 0
    LOCALZONESONLY: 'no'
    MAXIMUM_RESTORE_TIMEOUT: 21600
    MAXIMUM_TIMEOUT: 7200
    MYSQLBACKUP: accounts
    POSTBACKUP: 'no'
    PREBACKUP: -1
    PSQLBACKUP: 'no'
    The Tweak Settings for the backups have been changed to the cPanel defaults. "Extra CPUs for server load" was previously set to 5, and "I/O priority level at which nightly backups are run" to 3. Is it possible that somehow these settings haven't been applied?
     
  2. stormy

    stormy Well-Known Member

    Joined:
    Nov 22, 2003
    Messages:
    122
    Likes Received:
    6
    Trophy Points:
    168
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    The last backup of the machine that has no new data being generated, most services stopped, and load close to zero took more than 14 hours.

    It would be great if this could be troubleshooted before I decomission the server.

    Anyone else having backups that get slower every day?
     
  3. cPanelChrisI

    cPanelChrisI Technical Analyst II
    Staff Member

    Joined:
    Mar 24, 2014
    Messages:
    67
    Likes Received:
    11
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    Hello!

    The best way to begin troubleshooting that would be to check what is occurring in the backup logs at /usr/local/cpanel/logs/cpbackup/. You should be able to use the information and the times there to help pinpoint what is taking the most time during the backup.
     
  4. stormy

    stormy Well-Known Member

    Joined:
    Nov 22, 2003
    Messages:
    122
    Likes Received:
    6
    Trophy Points:
    168
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    Unfortunately, the logs don't solve the mistery. Some accounts, randomly, will simply start taking longer and displaying dots in the logs. It's not always the same account.
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    37,064
    Likes Received:
    1,288
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    Feel free to open a support ticket using the link in my signature so we can take a closer look and see what's happening. You can post the ticket number here so we can update this thread with the outcome.

    Thank you.
     
  6. stormy

    stormy Well-Known Member

    Joined:
    Nov 22, 2003
    Messages:
    122
    Likes Received:
    6
    Trophy Points:
    168
    Location:
    Spain
    cPanel Access Level:
    Root Administrator
    My datacenter has already opened a ticket. It's probably a memory leak related to Cloudlinux 7, and we've had it happen on more than one server already. I don' thave the ticket number though.
     
  7. budway

    budway Well-Known Member

    Joined:
    Apr 16, 2003
    Messages:
    189
    Likes Received:
    0
    Trophy Points:
    166
    Hi,

    I noticed this too, on a server not running cloudlinux.

    I thought it was client uploading more data and didn't care to dig in, but it has increased about 1/6 to 1/8 the time, recently.
     
  8. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    37,064
    Likes Received:
    1,288
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Would you mind opening a support ticket using the link in my signature so we can take a closer look and see what's happening? You can post the ticket number here so we can update this thread with the outcome.

    Thank you.
     
Loading...

Share This Page