Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Remote incremental backups issue

Discussion in 'Data Protection' started by uk01, Aug 24, 2017.

  1. uk01

    uk01 Well-Known Member

    Joined:
    Dec 31, 2009
    Messages:
    71
    Likes Received:
    5
    Trophy Points:
    58
    Hi there,

    Quick question for you guys

    New incremental backup started this week with v66, however we realised after the first night that we needed to switch change remote backups to rsync for them to work with incremental, so we did that yesterday.

    backup server 1 (mounted)
    tuesday night - backup folder size 56GB
    last night - backup folder size 28GB

    backup server 2 (remote)
    tuesday night - backup folder size 0GB
    last night - backup folder size 28GB

    So all is working fine, my only question is that if it's rsync'd last nights backup folder over to the remote server, what happens to the hard links to the other 28GB of files which are not in the tuesday night folder on the remote server?

    If we retain 5 days incremental I'm guessing this sorts itself out after 5 days when tuesdays folder is ditched by the first backup server. Am I right?

    Cheers

    EDIT
    I didn't realise after 14 hours the backups were still running, they normally only take 4 hours, so I expected the second night incremental to be much less.

    Its now
    backup server 1 (mounted)
    tuesday night - backup folder size 56GB
    last night - backup folder size 56GB <<< the same size?!

    backup server 2 (remote)
    tuesday night - backup folder size 0GB
    last night - backup folder size 37GB (probably still rsync'ing)

    I'll see what happens tonight, I'm not sure why it's 56GB again, maybe it restarted the incremental full copy when I added the remote.
     
    #1 uk01, Aug 24, 2017
    Last edited: Aug 24, 2017
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,080
    Likes Received:
    1,364
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    The disk space will appear to be the same, but it's not actually duplicating the space. It's simply the nature of how hard links work. This is described in more detail on the following thread:

    How do the backups work exactly?

    Let us know if this helps.

    Thanks!
     
  3. uk01

    uk01 Well-Known Member

    Joined:
    Dec 31, 2009
    Messages:
    71
    Likes Received:
    5
    Trophy Points:
    58
    ahh, right, even when i run du -msh /foldername, I'll take a read of the thread.

    Could you also clarify this bit if you don't mind, many thanks
    "So all is working fine, my only question is that if it's rsync'd last nights backup folder over to the remote server, what happens to the hard links to the other 28GB of files which are not in the tuesday night folder on the remote server?

    If we retain 5 days incremental I'm guessing this sorts itself out after 5 days when tuesdays folder is ditched by the first backup server. Am I right?"
     
  4. uk01

    uk01 Well-Known Member

    Joined:
    Dec 31, 2009
    Messages:
    71
    Likes Received:
    5
    Trophy Points:
    58
    ps strange how it's taking all day to run, I'll see what happens tonight but it's filling the backup server up. Hopefully sort itself out
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,080
    Likes Received:
    1,364
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    The remote incremental backup process (with rsync) is designed to check if the files associated with the account backup on the cPanel server also exist on the remote backup destination. If the files do not exist on the remote backup destination (or if the files have changed), then it copies the actual files. Otherwise, it makes use of hard links when the files already exist on the remote backup destination. The following document is available if you'd like to read more about how account backup information is stored:

    Metadata for Backups - Version 66 Documentation - cPanel Documentation

    Thank you.
     
  6. uk01

    uk01 Well-Known Member

    Joined:
    Dec 31, 2009
    Messages:
    71
    Likes Received:
    5
    Trophy Points:
    58
    Very well thought out! I love cpanel, it gets better and better.
     
    cPanelMichael likes this.
  7. uk01

    uk01 Well-Known Member

    Joined:
    Dec 31, 2009
    Messages:
    71
    Likes Received:
    5
    Trophy Points:
    58
    Unfortunately, as predicted yesterday (18 hours to backup instead of 4), this is not working properly.

    Last night it started backing up the whole thing again, our backup drives are 99% full and we've used nearly 1TB bandwidth in 24hrs.
    This puts our monthly predicted bandwidth 5Tb over our limit so I'm now going to have to stop backups until the issue is resolved.

    Back to compressed.
     
  8. InterServed

    InterServed Well-Known Member

    Joined:
    Jul 10, 2007
    Messages:
    258
    Likes Received:
    3
    Trophy Points:
    68
    cPanel Access Level:
    DataCenter Provider
    Hi,

    Same boat here , incremental backup before took about 2 hours to finish , with v66 it took even 8 hours. Everything started the get - - up ever since they implemented date for the folders (had a few talks about this with them and they never wanted to support the previous incremental method). In the past i simply backup /localbackup/incremental and rsync this offsite (worked awesome). Since the date implementation in incremental backups (da heck with it) all is working 10 times slower. For me all they did was to ruin the product instead of improving it.

    I will pay 10 times more attention for cPanel changes from now on. So many years of cPanel and yet not a decent backup system (shame).

    For large servers , all the crap about meta and hard-links (way to much overhead i assume) -> BIG BIG BIG NO THANK YOU ! Wish they would at least still support the previous incremental method (optional) or if they had conducted any kind of feedback about the implications of this changes.
     
    #8 InterServed, Aug 27, 2017
    Last edited by a moderator: Aug 27, 2017
  9. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    38,080
    Likes Received:
    1,364
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Hello,

    Internal case CPANEL-15493 is open to address an issue where the removal of the local backup directory fails after a successful transport to a remote destination when incremental backups are enabled in cPanel version 66. I'll monitor this case and update this thread with more information as it becomes available.

    As far as the performance of remote incremental backups, could you open a support ticket using the link in my signature so we can take a closer look? You can post the ticket number here so we can update this thread with the outcome.

    Thank you.
     
  10. graham_w

    graham_w Well-Known Member

    Joined:
    May 25, 2004
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    156
    Not sure if it's related, but I am using the rsync additional destination in WHM Backup for uncompressed backups and it seems to open two rsync processes and they both try and rsync the same file. I can also see on the remote server that its trying to save 2 copies of the same file. Once I kill one of the processes, it seems to complete the rsync as expected.

    Have opened ticket 8829847 to allow you guys to take a look.
     
  11. uk01

    uk01 Well-Known Member

    Joined:
    Dec 31, 2009
    Messages:
    71
    Likes Received:
    5
    Trophy Points:
    58
    sorry, only just looked back at this thread. We didn't open a ticket, we went back to compressed backups for now as every night cpanel was backing up the full accounts from 5 servers and using all our bandwidth, then storing the full backups multiple times and using all backup space.
    I know the folders "look" like they are using space, which is how hard links work but we KNOW they were as the backup space is gone and all our bandwidth was used. Incremental should use much less!

    I'm wondering that incremental does not play well with an SSHFS mounted backup space, because effectively cpanel is saving to a folder called /backup, which has nothing in it, it's a remote sshfs folder, so it saved everything every night because it looks in the /backup folder and see anything, so saves everything again, instead of being able to know its a remote backup server it's linked to.

    We'll come back to this when I have more time, but for now I can confirm incremental does not work with mounted backup drives.
     
  12. uk01

    uk01 Well-Known Member

    Joined:
    Dec 31, 2009
    Messages:
    71
    Likes Received:
    5
    Trophy Points:
    58
    ps we have mounted drives via sshfs because a/ we don't want the backups thrashing our top spec ssds and b/ we want to make use of the space on each server, not have it 50% empty for backups. Our hourly backups backup directly to the backup server through an agent.
     
  13. uk01

    uk01 Well-Known Member

    Joined:
    Dec 31, 2009
    Messages:
    71
    Likes Received:
    5
    Trophy Points:
    58
    I've now found out this issue relates to hardlinking.
    Sshfs has supported hardlinking for some time now but cpanel doesn't allow it.
    See the discussion on the feature request here
    Provide SSHFS support for Backups

    Until this is implemented sshfs will not work with incremental.
     
    cPanelMichael likes this.
Loading...

Share This Page