I'm working on this at the moment, currently testing the upload times for a daily backup of 9 gigs to see if it is anywhere near backing up to another server on the lan. I don't want backups to be running for more than 4 hours a night.
Once I'm happy with this I'm going to make a new script modelled on the /scripts/cpbackup script but sending to the s3 service rather than an ftp account.
Once I'm happy with this I'm going to make a new script modelled on the /scripts/cpbackup script but sending to the s3 service rather than an ftp account.
The ideal solution would be to have a machine brought up in their ec2 service to be your remote ftp/backup server which, in turn, uses s3 as data storage. Thou I don't know the exact details (what happens when the machine is brought down, loses power, stability, etc). I've been investigating this.
I've tried many of the pre-made scripts, but none of them are really fully functional and of much use. The only real solution is to write your own.
I've tried many of the pre-made scripts, but none of them are really fully functional and of much use. The only real solution is to write your own.
Just came across a very interesting article on the issue of doing backups to amazon s3.
http://paulstamatiou.com/2007/07/29/how-to-bulletproof-server-backups-with-amazon-s3/
For me s3 seems like outstanding value for money. If anyone has any updates on backing up onto s3 via cpanel please do let us know.
http://paulstamatiou.com/2007/07/29/how-to-bulletproof-server-backups-with-amazon-s3/
For me s3 seems like outstanding value for money. If anyone has any updates on backing up onto s3 via cpanel please do let us know.
this would be the ideal solution:
http://code.google.com/p/s3fs-fuse/
a fuse based interface... I think this prject is way early stage, and you have to compile from source.. but.. being able to simply mount the backup drive is perfect, since it will intergrate seemlessly with current cpanel.
my concern would be that you might get a lot of "get/put" billing if you use the rsync backups, depending on how it checks for last udates and checksums...
I think i found a solution
check out
http://www.elasticdrive.com/
allows s3 mounted as a drive, any filesystem you like. free for 40gb, but does not include charges for space used by amazon.
check out
http://www.elasticdrive.com/
allows s3 mounted as a drive, any filesystem you like. free for 40gb, but does not include charges for space used by amazon.
We have been trialling the open source Duplicity, with moderate results. It integrates rsync + s3 transport + gpg encryption. Only the most recent versions have the necessary functionality for fully unattended backups, but the overall concept is a very good one.
Still, while S3 may be cheaper, we've been leaning toward a service like rsync.net with normal rsync + ssh. The idea of encrypted backups is nice, but the integrity and ease of restoration with rsync is unparalleled. All of the rsync-esque solutions via S3 seem to be too unstable (they have to transfer via HTTP, after all).
Still, while S3 may be cheaper, we've been leaning toward a service like rsync.net with normal rsync + ssh. The idea of encrypted backups is nice, but the integrity and ease of restoration with rsync is unparalleled. All of the rsync-esque solutions via S3 seem to be too unstable (they have to transfer via HTTP, after all).
It seems like Cpanel would be wise to integrate this into the backup options. Imagine the heartache it would save to have this written once and added to a cpanel build rather than have all of us try to hack things and get them working on our own.
I would have to question whether or not it'd be "wise" for them to integrate this.
First of all, they're at the mercy of a 3rd party system... one from which they make NO money. I would think it would be "wise" of AMAZON to create a cpanel plugin to use their service. they're the ones getting paid, it should be worth their time if this is a market they want to go after.
Cpanel provides reasonable backup options. if you want to use S3, or anything else, simply find a piece of software which makes it available as a local mounted filesystem in which case, cpanel needn't edit a single line of code.
S3 FUSE: http://sourceforge.net/projects/s3fuse/ (I believe you have to compile this one from the CVS as there isn't a downloadable package yet)
GmailFS: http://richard.jones.name/google-hacks/gmail-filesystem/gmail-filesystem.html
So there's 2 solutions for you.. you can easilly use S3 without cpanel writing a single line of code,
OR you can use gmail space without cpanel writing a single line of code.
There's also "fuse-sshfs" to mount a remote filesystem via ssh.
Last edited:
We use this on several servers.
We use fuse, and a copy of JungleDisk... along with a custom whm_to_s3.sh script that, for us, archives the /backup/cpbackup/daily to /backup/archive with a time stamp (we keep 5 days worth of archived complete backups), then sends the /backup/cpbackup/daily to /mnt/s3/whm (Amazon S3) cloud.
Amazon S3 runs about 10% cpu usage on our Dual Quad Core Xeons, and uploads at about 80mbit/sec
So in the end, we have:
1) backup in /backup/cpbackup/daily
2) archived daily backup in /backup/archive (5 days worth)
3) archived daily backup to Amazon S3 (5 days worth)
Smoge
We use fuse, and a copy of JungleDisk... along with a custom whm_to_s3.sh script that, for us, archives the /backup/cpbackup/daily to /backup/archive with a time stamp (we keep 5 days worth of archived complete backups), then sends the /backup/cpbackup/daily to /mnt/s3/whm (Amazon S3) cloud.
Amazon S3 runs about 10% cpu usage on our Dual Quad Core Xeons, and uploads at about 80mbit/sec
So in the end, we have:
1) backup in /backup/cpbackup/daily
2) archived daily backup in /backup/archive (5 days worth)
3) archived daily backup to Amazon S3 (5 days worth)
Smoge
I spent some time messing with this over the weekend. It's currently rsyncing to Amazon S3.
I installed fuse and the command line version of jungledisk.
First I had to add the rpmforge repo. You may or may not have to do this. I could not install fuse on my system until I added this repo. You could install from source, but I wanted to yum it. I think all I installed was yum install fuse and yum install dkms-fuse. I will have to go back and look.
After fuse installed I ran "modprobe fuse"
Then you need to create a jungledisk INI file. To do this, I downloaded the USB Jungledisk version and set that up on my windows machine. I took the new INI file from Jungledisk USB and moved that to /etc/jd.ini on my CentOS server. You cannot use the full windows Jungledisk program to create the Linux INI file. You can use the USB windows version to create the INI file.
I created a /mnt/s3 directory and then ran:
./jungledisk /mnt/s3 -o config=/etc/jd.ini
Then tail -f /var/log/jungledisk.log to see what's going on. You should then be able to ls -l /mnt/s3 and see your S3 file system.
Kick off an rsync and watch for problems..
*Check the INSTALL file for details*
rsync -a --inplace /src/* /mnt/s3
I'm still messing with it, but this got me going. Not sure I'm going to use it, but I had to try.
I installed fuse and the command line version of jungledisk.
First I had to add the rpmforge repo. You may or may not have to do this. I could not install fuse on my system until I added this repo. You could install from source, but I wanted to yum it. I think all I installed was yum install fuse and yum install dkms-fuse. I will have to go back and look.
After fuse installed I ran "modprobe fuse"
Then you need to create a jungledisk INI file. To do this, I downloaded the USB Jungledisk version and set that up on my windows machine. I took the new INI file from Jungledisk USB and moved that to /etc/jd.ini on my CentOS server. You cannot use the full windows Jungledisk program to create the Linux INI file. You can use the USB windows version to create the INI file.
I created a /mnt/s3 directory and then ran:
./jungledisk /mnt/s3 -o config=/etc/jd.ini
Then tail -f /var/log/jungledisk.log to see what's going on. You should then be able to ls -l /mnt/s3 and see your S3 file system.
Kick off an rsync and watch for problems..
*Check the INSTALL file for details*
rsync -a --inplace /src/* /mnt/s3
I'm still messing with it, but this got me going. Not sure I'm going to use it, but I had to try.
Last edited:
You can try Vembu StoreGrid. It takes granular backup of cPanel domains and settings and streams it to S3. More details on StoreGrid Server Backup – Granular Domain Backup and Recovery « Application Catalog
Just be careful about this, because it might not actually be copying all the files -- I checked and found S3FS tended not to do a complete backup.
I am annoyed that while CPanel was saying a new backup routine would be available for 11.34, it's not here yet... with no ETA at all. Lots of screen shots and cryptic messages. Secure off-site backup should not be an "oh, yeah, that would be nice" at this point. We should not even be running FTP, much less using it to send configuration files around. Backup on the same hard drive as the server is... well, fast, but not a great idea. (It does help if you have two hard drives in your server box, as I did, but that's not an option with many plans.)
I am annoyed that while CPanel was saying a new backup routine would be available for 11.34, it's not here yet... with no ETA at all. Lots of screen shots and cryptic messages. Secure off-site backup should not be an "oh, yeah, that would be nice" at this point. We should not even be running FTP, much less using it to send configuration files around. Backup on the same hard drive as the server is... well, fast, but not a great idea. (It does help if you have two hard drives in your server box, as I did, but that's not an option with many plans.)
