CPanel Backup using Amazon S3 Service

bbercik

Registered
Jan 11, 2006
1
0
151
Hey, has anyone been able to do remote backups with CPanel to Amazon's S3 backup service?

It would be really nice to be able to do this.

Thanks,
Bill
 

misterq

Member
Jul 10, 2006
7
0
151
I'm working on this at the moment, currently testing the upload times for a daily backup of 9 gigs to see if it is anywhere near backing up to another server on the lan. I don't want backups to be running for more than 4 hours a night.

Once I'm happy with this I'm going to make a new script modelled on the /scripts/cpbackup script but sending to the s3 service rather than an ftp account.
 

wantmoore

Registered
Jul 5, 2005
3
0
151
I'm extremely interested in this as well. I found a few scripts through Google, but most of them are poorly written, or even worse, poorly documented. Has anyone had any luck with this?
 

jameshsi

Well-Known Member
Oct 22, 2001
347
0
316
Hey,
This could be very interested, if you got testing result, please let us know.
 

matthewdavis

Well-Known Member
Jun 26, 2003
90
0
156
NC, USA
The ideal solution would be to have a machine brought up in their ec2 service to be your remote ftp/backup server which, in turn, uses s3 as data storage. Thou I don't know the exact details (what happens when the machine is brought down, loses power, stability, etc). I've been investigating this.

I've tried many of the pre-made scripts, but none of them are really fully functional and of much use. The only real solution is to write your own.
 

jameshsi

Well-Known Member
Oct 22, 2001
347
0
316
If someone can make a whm module for remotely backup to S3 , I willing to pay for this.
 

freedman

Well-Known Member
Feb 13, 2005
312
1
168
Just came across a very interesting article on the issue of doing backups to amazon s3.

http://paulstamatiou.com/2007/07/29/how-to-bulletproof-server-backups-with-amazon-s3/

For me s3 seems like outstanding value for money. If anyone has any updates on backing up onto s3 via cpanel please do let us know.
this would be the ideal solution:
http://code.google.com/p/s3fs-fuse/

a fuse based interface... I think this prject is way early stage, and you have to compile from source.. but.. being able to simply mount the backup drive is perfect, since it will intergrate seemlessly with current cpanel.

my concern would be that you might get a lot of "get/put" billing if you use the rsync backups, depending on how it checks for last udates and checksums...
 
Feb 23, 2003
7
0
151
We have been trialling the open source Duplicity, with moderate results. It integrates rsync + s3 transport + gpg encryption. Only the most recent versions have the necessary functionality for fully unattended backups, but the overall concept is a very good one.

Still, while S3 may be cheaper, we've been leaning toward a service like rsync.net with normal rsync + ssh. The idea of encrypted backups is nice, but the integrity and ease of restoration with rsync is unparalleled. All of the rsync-esque solutions via S3 seem to be too unstable (they have to transfer via HTTP, after all).
 

bweeb

Registered
Jun 1, 2005
4
0
151
I'm working on this at the moment, currently testing the upload times for a daily backup of 9 gigs to see if it is anywhere near backing up to another server on the lan. I don't want backups to be running for more than 4 hours a night.

Once I'm happy with this I'm going to make a new script modelled on the /scripts/cpbackup script but sending to the s3 service rather than an ftp account.
hey did you ever get this working?
 

Drew Nichols

Well-Known Member
May 5, 2003
96
0
156
SC
It seems like Cpanel would be wise to integrate this into the backup options. Imagine the heartache it would save to have this written once and added to a cpanel build rather than have all of us try to hack things and get them working on our own.
 

freedman

Well-Known Member
Feb 13, 2005
312
1
168
It seems like Cpanel would be wise to integrate this into the backup options. Imagine the heartache it would save to have this written once and added to a cpanel build rather than have all of us try to hack things and get them working on our own.
I would have to question whether or not it'd be "wise" for them to integrate this.

First of all, they're at the mercy of a 3rd party system... one from which they make NO money. I would think it would be "wise" of AMAZON to create a cpanel plugin to use their service. they're the ones getting paid, it should be worth their time if this is a market they want to go after.

Cpanel provides reasonable backup options. if you want to use S3, or anything else, simply find a piece of software which makes it available as a local mounted filesystem in which case, cpanel needn't edit a single line of code.

S3 FUSE: http://sourceforge.net/projects/s3fuse/ (I believe you have to compile this one from the CVS as there isn't a downloadable package yet)

GmailFS: http://richard.jones.name/google-hacks/gmail-filesystem/gmail-filesystem.html

So there's 2 solutions for you.. you can easilly use S3 without cpanel writing a single line of code,
OR you can use gmail space without cpanel writing a single line of code.

There's also "fuse-sshfs" to mount a remote filesystem via ssh.
 
Last edited:

smoge

Well-Known Member
Jul 2, 2004
52
0
156
We use this on several servers.

We use fuse, and a copy of JungleDisk... along with a custom whm_to_s3.sh script that, for us, archives the /backup/cpbackup/daily to /backup/archive with a time stamp (we keep 5 days worth of archived complete backups), then sends the /backup/cpbackup/daily to /mnt/s3/whm (Amazon S3) cloud.

Amazon S3 runs about 10% cpu usage on our Dual Quad Core Xeons, and uploads at about 80mbit/sec

So in the end, we have:

1) backup in /backup/cpbackup/daily
2) archived daily backup in /backup/archive (5 days worth)
3) archived daily backup to Amazon S3 (5 days worth)

Smoge
 

bornonline

Well-Known Member
Nov 19, 2004
139
0
166
Earth
I spent some time messing with this over the weekend. It's currently rsyncing to Amazon S3.

I installed fuse and the command line version of jungledisk.

First I had to add the rpmforge repo. You may or may not have to do this. I could not install fuse on my system until I added this repo. You could install from source, but I wanted to yum it. I think all I installed was yum install fuse and yum install dkms-fuse. I will have to go back and look.

After fuse installed I ran "modprobe fuse"

Then you need to create a jungledisk INI file. To do this, I downloaded the USB Jungledisk version and set that up on my windows machine. I took the new INI file from Jungledisk USB and moved that to /etc/jd.ini on my CentOS server. You cannot use the full windows Jungledisk program to create the Linux INI file. You can use the USB windows version to create the INI file.

I created a /mnt/s3 directory and then ran:
./jungledisk /mnt/s3 -o config=/etc/jd.ini

Then tail -f /var/log/jungledisk.log to see what's going on. You should then be able to ls -l /mnt/s3 and see your S3 file system.

Kick off an rsync and watch for problems..
*Check the INSTALL file for details*
rsync -a --inplace /src/* /mnt/s3

I'm still messing with it, but this got me going. Not sure I'm going to use it, but I had to try.
 
Last edited:

rnldpj

Registered
Mar 22, 2011
2
0
51
A step by step howto for configuring cpanel backup to amazon s3. Hope this might be useful:)

/https://jackal777.wordpress.com/2011/03/22/cpanel-backup-to-amazon-s3/
 

allpar

Well-Known Member
Sep 16, 2005
52
3
158
Just be careful about this, because it might not actually be copying all the files -- I checked and found S3FS tended not to do a complete backup.

I am annoyed that while CPanel was saying a new backup routine would be available for 11.34, it's not here yet... with no ETA at all. Lots of screen shots and cryptic messages. Secure off-site backup should not be an "oh, yeah, that would be nice" at this point. We should not even be running FTP, much less using it to send configuration files around. Backup on the same hard drive as the server is... well, fast, but not a great idea. (It does help if you have two hard drives in your server box, as I did, but that's not an option with many plans.)