The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HOW TO: make full backups use --rsyncable w/ gzip

Discussion in 'General Discussion' started by Lyttek, Nov 30, 2005.

  1. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    HOW TO: make full backups work with rsync

    Latest edit: rsyncable option is now included in cpbackup by cPanel, so all the instructions below are moot for version 11 and up.

    Edits: The simplified version after much testing and trial is provided below. Thanks to Chirpy for finding the exact method that works so simply! Also added a simple rotation script.

    Modify the root crontab changing the line shown as follows:

    That's it!

    Here's a simple script to rotate the backups:
    Code:
    #!/bin/sh
    
    # This script rotates the backup files daily, keeping 7 days worth.
    
    DATEFORMAT=$(date +%a)
    BACKUPSOURCE=/backup
    BACKUPDEST=/Data/backup/rotation/
    
    rm -Rf $BACKUPDEST/$DATEFORMAT
    mkdir $BACKUPDEST/$DATEFORMAT
    cp -R $BACKUPSOURCE/* $BACKUPDEST/$DATEFORMAT
    
    The original post is below.

    ******** Original Post ************


    Note: This post may become obsolete if it turns out that setting a permanent environment variable fixes the issue. If so, this will get updated to reflect that.

    Using the Full Backup feature built into WHM is nice, because it grabs pretty much everything and creates stand-alone backups. The problem with this backup, however, is that because it uses gzip compression, it isn't rsync-friendly.

    Rsync is a nifty file transfer tool that only transfers the changes in a file. So, once you have downloaded a backup once, rather than download the whole backup again to update it, it only sends the changes to the file and incorporates them into the existing file. In the case of files that have been compressed using the default gzip settings, almost the entire file is transferred again because of the adaptive compression algorithm.

    There is good news, though. I just saved a bunch of money on my car insurance by switching to Geico ;)

    Ok, seriously, the good news is that later versions of gzip have the capability of playing nice with rysnc by way of the '--rsyncable' switch. With an rsyncable gzip file, rsync can process it much more efficiently. (You can check your version of gzip to see if it will handle rsync by looking at the help: 'gzip --help')

    So how do we make that happen? The good news is it's a simple, one-line modification to the 'cpbackup' script located in the /scripts directory. The bad news is that cpanel will overwrite this file every time it runs the 'upcp' script. This means we have to protect it somehow. Another nifty script will take care of that.

    So, first, let's modify the 'cpbackup' script:

    1) make a copy of the script and call it 'cpbackupunedited'
    2) Open the 'cpbackup' script and add the following line just after the remarks at the top, but before the 'BEGIN' statement
    3) Save the script. Now, make another copy of the file, this time calling it 'cpbackupedited'
    Your backups are now going to work with rsync rather nicely! For instance, on the first test I made using a 1.5gig set of gzip files, 35 megs were transferred, in contrast to the 1.2gigs transferred without the --rsyncable switch. That speeds up transfer times quite a bit!

    Now, we need to protect our cpbackup file. While we could 'chattr' so that it can't be overwritten, there is another, more refined method that will let us know if there are changes in the script that need to be looked at. This is a modification of the 'watchwwwacct' script you can read more about elsewhere on the forum. I'll post just the modified script for sake of brevity:

    Lastly, we need to setup our crontab to run this watcher script *after* 'upcp' runs, but *before* the cpbackup script runs. Edit the crontab (as root, obviously):

    Look for the line with '/scripts/upcp' and change it:
    Save it, and you're done!


    If you want to automate the rsyncing of your files, take a peek at the following HOW-TO, which is clear and works great:

    http://www.jdmz.net/ssh/
     
    #1 Lyttek, Nov 30, 2005
    Last edited: May 12, 2007
    alwaysweb likes this.
  2. CoolMike

    CoolMike Well-Known Member

    Joined:
    Sep 6, 2001
    Messages:
    307
    Likes Received:
    0
    Trophy Points:
    16
    Hi

    Did you test the restore already? Because if this works, it would be a great solution for us. Because we copy every day all the backup files to a external server.

    Michael
     
  3. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    I've not had time to do an actual restore test, but I have extracted the things on the backup server and it all looks well.

    What I need to do is flesh this out a bit so that the md5 checksums on both systems are compared to determine if there were any errors.
     
  4. alwaysweb

    alwaysweb Well-Known Member

    Joined:
    Mar 8, 2002
    Messages:
    97
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Dallas, TX
    cPanel Access Level:
    Root Administrator
    Thanks for this cool post! (We previously had to adjust our rsyncing of our server's backups due to the bandwidth used.)

    I tried it out, but wasn't successful in actually seeing any rsync savings in the transfer... I installed as indicated, and ran /scripts/cpbackup manually, and then rsync'd that to my offsite backup server. Easy.

    Then, I dropped a few 5mb files into a few domains' www folders... then I renamed /backup/cpbackup/daily to /backup/cpbackup/daily.old (so cpbackup would run again) and ran /scripts/cpbackup let it finish, and did the rsync again...

    It seems to have transmitted the full .tar.gz for each user in full all over again.

    Any ideas? Could you perhaps help me on this on forum or off? Is there another way you can think of to test this? Etc.

    (We would be glad to pay you for your time) I PM'd you my contact info, thanks.
     
  5. dgbaker

    dgbaker Well-Known Member
    PartnerNOC

    Joined:
    Sep 20, 2002
    Messages:
    2,578
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Toronto, Ontario Canada
    cPanel Access Level:
    DataCenter Provider
    Because you renamed the daily directory to something else, there is nothing for it to rsync with. The script only works to copy to an existing gz backup file. If no gz file exists the entire file is copied.

    Nice script though BTW.
     
  6. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    Here's what I think has happened:

    1) original cpbackup ran and created all the new backup files under /cpbackup/daily

    2) rsync'd, pulling down /cpbackup/daily

    3) renamed /cpbackup/daily to /cpbackup/daily.old

    4) ran cpbackup again, recreating /cpbackup/daily

    At this point you have /cpbackup/daily and /cpbackup/daily.old Assuming you used the same rsync script that I did:

    5) rsync'd again, pulling down minor changes from /cpbackup/daily and the full /cpbackup/daily.old directory, since it didn't exist the first time.

    The script pulls down changes recursively from /cpbackup.
     
  7. alwaysweb

    alwaysweb Well-Known Member

    Joined:
    Mar 8, 2002
    Messages:
    97
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Dallas, TX
    cPanel Access Level:
    Root Administrator
    HI, I did the renaming on the local machine (not the backup server) so that I could run the cpbackup "daily" again and rsync that...
     
  8. alwaysweb

    alwaysweb Well-Known Member

    Joined:
    Mar 8, 2002
    Messages:
    97
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Dallas, TX
    cPanel Access Level:
    Root Administrator
    Hi, I don't think I retransmitted the .old... as I used "rsync-exclude" to ignore the daily.old in my remote backup script... but I think the reason it did a full transfer was that the .tar.gz's on the backup server were not "gzip --rsyncable"... Apparently, the file on both ends needs to be rsync friendly right to make a difference? Since the one on the backup server were older and not rsync friendly, and even though the local daily backup .tar.gz for a site WAS rsync friendly, unless they both are then it does a full transmit...? That's at least what it seems to me! By letting the full rsync finish (so the backup server has rsync friendly .tar.gz's) and then with the next day's cpbackup running.. I then did my remote backup script and it only transmitted changes. If this is the case, it may not be a bad idea to mention in the original post... Thanks all !



    P.S. I explored the "global environment variables" you mentioned, and by editing:

    /etc/profile

    and above this line (approx line 58):

    export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC

    I added:

    GZIP="--rsyncable"

    And then added GZIP to the end of the export line to look like this:

    export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC GZIP


    And, to make it go into effect right away I did this from the command line with:

    # GZIP="--rsyncable"; export GZIP

    I believe it has accomplished the same thing. I took the line out of the /scripts/cpbackup and it appears to successfully do a rsync-friendly backup. Can anyone else test and confirm this?
     
    #8 alwaysweb, Dec 7, 2005
    Last edited: Dec 7, 2005
  9. CoolMike

    CoolMike Well-Known Member

    Joined:
    Sep 6, 2001
    Messages:
    307
    Likes Received:
    0
    Trophy Points:
    16
    Is the restore function in WHM still possible for this soluiton? Because if yes, maybe it would make sense, when Cpanel would implement this change globaly...

    Michael
     
  10. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    Yep.

    All this does is force gzip to make a change to the compression algorithm. Any version of gzip, even one that's old enough to not have the rsync capability, can decompress the file.

    Link that explains the --rsyncable switch
     
  11. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    I'm testing a similar route. I've also reverted back to the original cpbackup script. Rather than edit /etc/profile I dropped a new script in /etc/profile.d named gzipenv.sh that is as follows:

    I'll post back with results. How's your test going?
     
  12. CoolMike

    CoolMike Well-Known Member

    Joined:
    Sep 6, 2001
    Messages:
    307
    Likes Received:
    0
    Trophy Points:
    16
    This looks even more simple, please inform us, if this works, I will implement it imidatly and I'm sure I can save a lot of traffic in my network.

    Michael
     
  13. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    Well...

    The script ran, and the following shows up when I use the 'env' command:

    GZIP=--rsyncable

    However, the sync took 1.2 gigs, so something didn't work correctly. Not sure why.

    Here's something I don't understand yet, and it may have bearing:

    In the HOW TO section, the actual command to set the GZIP variable is
    How does this differ in results from
    In particular, does the . before the = in the former statement mean anything?
     
  14. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    Ok, did some more testing thusly:

    I left the latest script in /etc/profile.d, AND reverted back to the edited cpbackup script. When I try to manually run it, it errors out because now it's trying to use gzip with a switch of --rsyncable--rsyncable

    This tells me that both scripts are adding the requisite switch... so why didn't it work? Maybe a fluke, so I've gone back to the original, unedited version of cpbackup and the /etc/profile.d script and we'll see what happens tonight.
     
  15. alwaysweb

    alwaysweb Well-Known Member

    Joined:
    Mar 8, 2002
    Messages:
    97
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Dallas, TX
    cPanel Access Level:
    Root Administrator
    I think you're right, the env only doesn't seem to be cutting it --
    From a programming background, The ".=" versus "=" means append to... the previous settings of the variable. So if test = "abcd" and then you did test .= "efgh" then test would be "abcdefgh".

    The second one is obvious, a normal equal sign. It just overwrites the variable with the value.... Hope this helps!

    Perhaps its worth testing with a ".=" vs. "=" and see which behaves :)
     
  16. alwaysweb

    alwaysweb Well-Known Member

    Joined:
    Mar 8, 2002
    Messages:
    97
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Dallas, TX
    cPanel Access Level:
    Root Administrator
    I tried this myself as well and got the --rsyncable--rsyncable messages on the command line...

    Try this: If you change the quoted "--rsyncable" to " --rsyncable" with a space before the double dash (in both locations)... and if they BOTH apply... and its:

    " --rsyncable --rsyncable"

    that still is a valid argument to gzip. :)
     
  17. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    Ok, I see what caused the problem with the double switch, which makes sense. Thanks for that important bit of info! Still not sure why the env. variable didn't seem to work. Still waiting for test to finish.
     
  18. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    Sucess!!

    It must have been some type of fluke. Just finished the second round of tests, and it updated 1.8 gigs with only 39 megs of transfer!

    Guess I'll update the original post with the new instructions.
     
  19. CoolMike

    CoolMike Well-Known Member

    Joined:
    Sep 6, 2001
    Messages:
    307
    Likes Received:
    0
    Trophy Points:
    16
    Hi Lyttek

    Which is now the solution you used?

    Michael
     
  20. Lyttek

    Lyttek Well-Known Member

    Joined:
    Jan 2, 2004
    Messages:
    770
    Likes Received:
    3
    Trophy Points:
    18
    Dropping the script in /etc/profile.d

    Check the edited first post.
     
Loading...

Share This Page