The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

CPanel Backup using Amazon S3 Service

Discussion in 'Data Protection' started by bbercik, Oct 7, 2006.

  1. bbercik

    bbercik Registered

    Joined:
    Jan 11, 2006
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    Hey, has anyone been able to do remote backups with CPanel to Amazon's S3 backup service?

    It would be really nice to be able to do this.

    Thanks,
    Bill
     
  2. misterq

    misterq Member

    Joined:
    Jul 10, 2006
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    I'm working on this at the moment, currently testing the upload times for a daily backup of 9 gigs to see if it is anywhere near backing up to another server on the lan. I don't want backups to be running for more than 4 hours a night.

    Once I'm happy with this I'm going to make a new script modelled on the /scripts/cpbackup script but sending to the s3 service rather than an ftp account.
     
  3. wantmoore

    wantmoore Registered

    Joined:
    Jul 5, 2005
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    I'm extremely interested in this as well. I found a few scripts through Google, but most of them are poorly written, or even worse, poorly documented. Has anyone had any luck with this?
     
  4. jameshsi

    jameshsi Well-Known Member

    Joined:
    Oct 22, 2001
    Messages:
    347
    Likes Received:
    0
    Trophy Points:
    16
    Hey,
    This could be very interested, if you got testing result, please let us know.
     
  5. matthewdavis

    matthewdavis Well-Known Member

    Joined:
    Jun 26, 2003
    Messages:
    90
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    NC, USA
    The ideal solution would be to have a machine brought up in their ec2 service to be your remote ftp/backup server which, in turn, uses s3 as data storage. Thou I don't know the exact details (what happens when the machine is brought down, loses power, stability, etc). I've been investigating this.

    I've tried many of the pre-made scripts, but none of them are really fully functional and of much use. The only real solution is to write your own.
     
  6. jameshsi

    jameshsi Well-Known Member

    Joined:
    Oct 22, 2001
    Messages:
    347
    Likes Received:
    0
    Trophy Points:
    16
    If someone can make a whm module for remotely backup to S3 , I willing to pay for this.
     
  7. bigdessert

    bigdessert Active Member

    Joined:
    Jul 21, 2006
    Messages:
    35
    Likes Received:
    0
    Trophy Points:
    6
    i would also like to know if this is possible.....
     
  8. IbnKuldun

    IbnKuldun Member

    Joined:
    May 6, 2005
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    1
  9. freedman

    freedman Well-Known Member

    Joined:
    Feb 13, 2005
    Messages:
    312
    Likes Received:
    1
    Trophy Points:
    18
    this would be the ideal solution:
    http://code.google.com/p/s3fs-fuse/

    a fuse based interface... I think this prject is way early stage, and you have to compile from source.. but.. being able to simply mount the backup drive is perfect, since it will intergrate seemlessly with current cpanel.

    my concern would be that you might get a lot of "get/put" billing if you use the rsync backups, depending on how it checks for last udates and checksums...
     
  10. bigdessert

    bigdessert Active Member

    Joined:
    Jul 21, 2006
    Messages:
    35
    Likes Received:
    0
    Trophy Points:
    6
    I think i found a solution

    check out

    http://www.elasticdrive.com/

    allows s3 mounted as a drive, any filesystem you like. free for 40gb, but does not include charges for space used by amazon.
     
  11. phoenixdarkdirk

    Joined:
    Feb 23, 2003
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    We have been trialling the open source Duplicity, with moderate results. It integrates rsync + s3 transport + gpg encryption. Only the most recent versions have the necessary functionality for fully unattended backups, but the overall concept is a very good one.

    Still, while S3 may be cheaper, we've been leaning toward a service like rsync.net with normal rsync + ssh. The idea of encrypted backups is nice, but the integrity and ease of restoration with rsync is unparalleled. All of the rsync-esque solutions via S3 seem to be too unstable (they have to transfer via HTTP, after all).
     
  12. bweeb

    bweeb Registered

    Joined:
    Jun 1, 2005
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    hey did you ever get this working?
     
  13. dwykofka

    dwykofka Well-Known Member

    Joined:
    Aug 6, 2003
    Messages:
    394
    Likes Received:
    3
    Trophy Points:
    18
  14. Drew Nichols

    Drew Nichols Well-Known Member

    Joined:
    May 5, 2003
    Messages:
    96
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    SC
    It seems like Cpanel would be wise to integrate this into the backup options. Imagine the heartache it would save to have this written once and added to a cpanel build rather than have all of us try to hack things and get them working on our own.
     
  15. freedman

    freedman Well-Known Member

    Joined:
    Feb 13, 2005
    Messages:
    312
    Likes Received:
    1
    Trophy Points:
    18
    I would have to question whether or not it'd be "wise" for them to integrate this.

    First of all, they're at the mercy of a 3rd party system... one from which they make NO money. I would think it would be "wise" of AMAZON to create a cpanel plugin to use their service. they're the ones getting paid, it should be worth their time if this is a market they want to go after.

    Cpanel provides reasonable backup options. if you want to use S3, or anything else, simply find a piece of software which makes it available as a local mounted filesystem in which case, cpanel needn't edit a single line of code.

    S3 FUSE: http://sourceforge.net/projects/s3fuse/ (I believe you have to compile this one from the CVS as there isn't a downloadable package yet)

    GmailFS: http://richard.jones.name/google-hacks/gmail-filesystem/gmail-filesystem.html

    So there's 2 solutions for you.. you can easilly use S3 without cpanel writing a single line of code,
    OR you can use gmail space without cpanel writing a single line of code.

    There's also "fuse-sshfs" to mount a remote filesystem via ssh.
     
    #15 freedman, Mar 29, 2008
    Last edited: Mar 29, 2008
  16. smoge

    smoge Well-Known Member

    Joined:
    Jul 2, 2004
    Messages:
    52
    Likes Received:
    0
    Trophy Points:
    6
    We use this on several servers.

    We use fuse, and a copy of JungleDisk... along with a custom whm_to_s3.sh script that, for us, archives the /backup/cpbackup/daily to /backup/archive with a time stamp (we keep 5 days worth of archived complete backups), then sends the /backup/cpbackup/daily to /mnt/s3/whm (Amazon S3) cloud.

    Amazon S3 runs about 10% cpu usage on our Dual Quad Core Xeons, and uploads at about 80mbit/sec

    So in the end, we have:

    1) backup in /backup/cpbackup/daily
    2) archived daily backup in /backup/archive (5 days worth)
    3) archived daily backup to Amazon S3 (5 days worth)

    Smoge
     
  17. bornonline

    bornonline Well-Known Member

    Joined:
    Nov 19, 2004
    Messages:
    139
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Earth
    I spent some time messing with this over the weekend. It's currently rsyncing to Amazon S3.

    I installed fuse and the command line version of jungledisk.

    First I had to add the rpmforge repo. You may or may not have to do this. I could not install fuse on my system until I added this repo. You could install from source, but I wanted to yum it. I think all I installed was yum install fuse and yum install dkms-fuse. I will have to go back and look.

    After fuse installed I ran "modprobe fuse"

    Then you need to create a jungledisk INI file. To do this, I downloaded the USB Jungledisk version and set that up on my windows machine. I took the new INI file from Jungledisk USB and moved that to /etc/jd.ini on my CentOS server. You cannot use the full windows Jungledisk program to create the Linux INI file. You can use the USB windows version to create the INI file.

    I created a /mnt/s3 directory and then ran:
    ./jungledisk /mnt/s3 -o config=/etc/jd.ini

    Then tail -f /var/log/jungledisk.log to see what's going on. You should then be able to ls -l /mnt/s3 and see your S3 file system.

    Kick off an rsync and watch for problems..
    *Check the INSTALL file for details*
    rsync -a --inplace /src/* /mnt/s3

    I'm still messing with it, but this got me going. Not sure I'm going to use it, but I had to try.
     
    #17 bornonline, Apr 7, 2008
    Last edited: Apr 7, 2008
  18. rnldpj

    rnldpj Registered

    Joined:
    Mar 22, 2011
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    A step by step howto for configuring cpanel backup to amazon s3. Hope this might be useful:)

    /https://jackal777.wordpress.com/2011/03/22/cpanel-backup-to-amazon-s3/
     
  19. lenin

    lenin Member

    Joined:
    Sep 9, 2011
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Carson City, Nevada, United States
    cPanel Access Level:
    DataCenter Provider
  20. allpar

    allpar Active Member

    Joined:
    Sep 16, 2005
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    6
    Just be careful about this, because it might not actually be copying all the files -- I checked and found S3FS tended not to do a complete backup.

    I am annoyed that while CPanel was saying a new backup routine would be available for 11.34, it's not here yet... with no ETA at all. Lots of screen shots and cryptic messages. Secure off-site backup should not be an "oh, yeah, that would be nice" at this point. We should not even be running FTP, much less using it to send configuration files around. Backup on the same hard drive as the server is... well, fast, but not a great idea. (It does help if you have two hard drives in your server box, as I did, but that's not an option with many plans.)
     
Loading...

Share This Page