rsync backups

#1

I have searched around the forums for this, but I can't seem to find what I'm looking for.

For those of you using rsync -- how are you running backups? We want to use rsync to sync up only the changed data, but still be able to use the cPanel restore feature.

Currently we have a custom script that iterates through each user, uses pkgacct without compression to make a .tar, and then rsync the tar. This transfers only changed bits (in theory).

The rsync process is working great, and is fast -- but the tar process, not so much. Last night our custom script took 22 hours on a server that is normal for our environment -- not overloaded at all. Obviously, this isn't going to work.

So, how are you doing it?

-Chris

B

budlight

Member

#2

disk space is cheap prob less hassle to directly copy the backups to another hd on server then download them weekly to a local portable hd also rsyncing a tar kind of defeats the point in alot of ways. That's sorta like when people rar a torrent into 50 files

E

ee99ee

Well-Known Member

#3

budlight said:
disk space is cheap prob less hassle to directly copy the backups to another hd on server then download them weekly to a local portable hd also rsyncing a tar kind of defeats the point in alot of ways. That's sorta like when people rar a torrent into 50 files

This isn't really reasonable at our scale. Plus, our rsync cluster is off-site.

We put some logging in our script. The rsync transfers are fast, it's the tar process that is slow.

-Chris

H

hostultra

Well-Known Member

#4

We use rsync backups, and dont use the cpanel backups feature at all.

If a server fails, you can restore cpanel by just restoring the folders like /home /var/db etc..

M

madaboutlinux

Well-Known Member

Jul 9, 2010

#5

hostultra said:
If a server fails, you can restore cpanel by just restoring the folders like /home /var/db etc..

In this case, you will have to add the domain, create email accounts, databases, addon domains etc etc.. and what if you have hundreds of domains on the server?

If you are just backing up /home and /var/db using rsync method, you will be in trouble if the hard drive crash and you have to restore everything back. Instead, backup all the cPanel related files/directories as well along with home and mysql directories, so restoring these will bring your server back to the original state.

L

lthiagol

Member

Jul 9, 2010

#6

Hi, I'm working with ee99ee. We were using the pkgacct to avoid issues when restoring backups, if we just rsync the /home and the db folder the restore task will be a mess!
Do anyone has a tip to turn the backups faster ? Or can describe how the rsync backup is done on your hosts ?

H

hostultra

Well-Known Member

Jul 9, 2010

#7

I said /var/db etc..

The etc.. means all the folders cpanel uses.
A restore can be done by just putting those folders back in the proper places, and I have done it many times, without loosing any databases or configurations, or having to re-add domains or email accounts.

Actually the way I do it is I rsync the entire / directory, with an exclude list containing what I dont want to rsync, like /dev and large log folders.

Miraenda

Well-Known Member

Jul 10, 2010

#8

Since the actual list of files and folders to rsync hasn't been noted, I'll provide a list of the files and folders I've used personally for cPanel rsyncs in the past:

/etc/passwd
/etc/shadow
/etc/sudoers
/etc/quota.conf
/etc/group
/etc/domainips
/etc/exim/
/etc/exim.key
/etc/exim.crt
/etc/exim.pl.local
/etc/exim.conf
/etc/ips
/etc/localdomains
/etc/proftpd/
/etc/my.cnf
/etc/rc.local
/etc/resolv.conf
/etc/trueuserdomains
/etc/userdomains
/etc/valiases/
/etc/vfilters/
/etc/vdomainaliases/
/etc/vmail/
/etc/wwwacct.conf
/root/.my.cnf
/usr/java/
/usr/local/apache/conf/
/usr/local/apache/htdocs/
/usr/local/cpanel/3rdparty/mailman/archives/
/usr/local/cpanel/3rdparty/mailman/lists/
/usr/local/cpanel/base/frontend/lp/
/usr/share/ssl/
/var/lib/pgsql/
/var/spool/cpcpan/
/var/cpanel/
/var/spool/cron/
/var/lib/mysql/
/var/named/
/home

After rsyncing, if your machines are on different OS or architectures, please remove the following file on the new server:

Code:

/var/cpanel/sysinfo.config

That file contains data on the old server's OS and architecture and is not interchangeable between systems.

If the old server IPs will be used for the new server, you might well want these files (be careful copying these otherwise as they'll change your main server IP to the old server's IP and should only be copied if you intend to switch to the new server shortly after a reboot with the old IP becoming the main IP on that new server):

Code:

/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-eth0

After you copy over these files and folders, then you would want to rebuild Apache. The above files should include all the user data you need.

Normally, I would then create a file in /tmp on the new server called movelist and include the files and folders, then I would run the following command on the new server:

Code:

for file in `cat /tmp/movelist` ;do rsync --archive --verbose --rsh=ssh oldserverIP:$file $file --exclude '/var/cpanel/sysinfo.config' ;done

Replacing oldserverIP with the IP for the old server.

Please note that you cannot use cpmove or cPanel restore files when using this method as the two methods are not interchangeable. It is either cPanel backups and restores or rsyncs. This is because the cPanel backups when being restored on a system presume the account doesn't have files or folders already existing for the data. If the account exists in any fashion (the username in /etc/passwd, /home/user/ exists, and so on), then the cPanel backup cannot restore via WHM on that system, although you can force a restore in root SSH using the --force flag.

I also would highly suggest using dumps for MySQL data instead of this method if you have InnoDB MySQL database tables. If MySQL is running when you do an rsync like this for backups, the /var/lib/mysql/ibdata1 file might corrupt. A corrupted InnoDB table or tables can cause MySQL to not start, so you are risking MySQL not working by using an rsync. It's better to get a dump of the databases on top of the rsync for MySQL.

Last edited: Jul 10, 2010

Reactions: Geison Porfirio

F

furquan

Well-Known Member

#9

Hello Miraenda :

I was looking for such a script to sync 2 servers, since we have migrated a server from an OLD hardware to New hardware.

Since i have copied the data from the old to the new via cpanel copy feature i still need to sync the old with the new to cover up the changed data, Can i use the above script to achieve the same ?

Thank you

cPanelTristan

Quality Assurance Analyst

Staff member

#10

furquan, you can use the same to re-sync the data from the old machine to the new machine for the changed files and folders. The post was made by my non-staff account at the time, so I'm the same person who did the prior post anyway.

F

furquan

Well-Known Member

#11

Thank you very much Tristan

, highly appreciate that..i'd like to tell you, your post on the forums have been very helpful.

Thank you.

cPanelTristan

Quality Assurance Analyst

Staff member

#12

Certainly, you are very welcome and thank you very much for the kind words

F

furquan

Well-Known Member

#13

Tristan :

please pardon me for bumping this thread once again

but i have a small query.

I am about to migrate a server "abc" within the same DC from Physical TO Virtual., Now the issue is,

Since the accounts will be copied / transfered over to the virtual server on a temporary ip address during the copy process , BUT
The sites will still need to use the same OLD ip after the migration, Hence my question,

How can i change the A record of all the sites in one go on the virtual machine after i have copied/synced over my data from the physical machine and give all my accounts the actual IP address they were actually on. ?

Please assist ..

cPanelTristan

Quality Assurance Analyst

Staff member

#14

Why do the accounts need to be transferred onto a temporary IP address during the copy process? If you are rsyncing the data, then you are not copying and moving them to a new IP address space in those files themselves. You can simply keep them on the old IP in the files, then use the process to copy the IP over that I indicated for those IP files. Upon reboot, the machine will take the IP from the old machine and, since all the files for the rsync data already have that old IP, there isn't any new IP or transfer of the IP that would take place for the accounts.

If you are using the cPanel backup and transfer instead, then yes you would need to change the IPs and have to do it using WHM > Change Site's IP Address area or use the multiple IP change area instead.

F

furquan

Well-Known Member

#15

Why do the accounts need to be transferred onto a temporary IP address during the copy process?

Since i cannot afford down time, i will copy the data on the temporary IP address using "Copy multiple accounts/packages from another server" from WHM and then rsync the data using the script above and then in the end change IP address of the VM, But since only one ip can be live during all this while, i will first copy the accounts to a temporary account and then switch them back to their OLD ip address ...

cPanelTristan

Quality Assurance Analyst

Staff member

#16

You will need to use the IP migration wizard and select all sites to go to the old IP after you add the old IP to the machine. Since you are going to use the cPanel process for transfer, you have no choice but to use the WHM > IP Migration Wizard to change the IPs using that method afterward.

F

furquan

Well-Known Member

#17

Tristan :

I think i found my answer :D, "multiple IP change " does exist in WHM, i had completely overlooked it, and hence this confusion.

I can copy the accounts over to this temporary ip and after i sync the data, i can swith the IP's of all the sites at once using the above whm feature

Thank you for pointing that out to me !!

F

furquan

Well-Known Member