The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

rsync backups

Discussion in 'General Discussion' started by ee99ee, Jul 8, 2010.

  1. ee99ee

    ee99ee Well-Known Member

    Joined:
    Aug 15, 2003
    Messages:
    70
    Likes Received:
    0
    Trophy Points:
    6
    I have searched around the forums for this, but I can't seem to find what I'm looking for.

    For those of you using rsync -- how are you running backups? We want to use rsync to sync up only the changed data, but still be able to use the cPanel restore feature.

    Currently we have a custom script that iterates through each user, uses pkgacct without compression to make a .tar, and then rsync the tar. This transfers only changed bits (in theory).

    The rsync process is working great, and is fast -- but the tar process, not so much. Last night our custom script took 22 hours on a server that is normal for our environment -- not overloaded at all. Obviously, this isn't going to work.

    So, how are you doing it?

    -Chris
     
  2. budlight

    budlight Member

    Joined:
    Jan 11, 2007
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
    disk space is cheap prob less hassle to directly copy the backups to another hd on server then download them weekly to a local portable hd also rsyncing a tar kind of defeats the point in alot of ways. That's sorta like when people rar a torrent into 50 files
     
  3. ee99ee

    ee99ee Well-Known Member

    Joined:
    Aug 15, 2003
    Messages:
    70
    Likes Received:
    0
    Trophy Points:
    6
    This isn't really reasonable at our scale. Plus, our rsync cluster is off-site.

    We put some logging in our script. The rsync transfers are fast, it's the tar process that is slow.

    -Chris
     
  4. hostultra

    hostultra Well-Known Member

    Joined:
    Aug 21, 2002
    Messages:
    167
    Likes Received:
    0
    Trophy Points:
    16
    We use rsync backups, and dont use the cpanel backups feature at all.

    If a server fails, you can restore cpanel by just restoring the folders like /home /var/db etc..
     
  5. madaboutlinux

    madaboutlinux Well-Known Member

    Joined:
    Jan 24, 2005
    Messages:
    1,052
    Likes Received:
    2
    Trophy Points:
    38
    Location:
    Earth
    In this case, you will have to add the domain, create email accounts, databases, addon domains etc etc.. and what if you have hundreds of domains on the server?

    If you are just backing up /home and /var/db using rsync method, you will be in trouble if the hard drive crash and you have to restore everything back. Instead, backup all the cPanel related files/directories as well along with home and mysql directories, so restoring these will bring your server back to the original state.
     
  6. lthiagol

    lthiagol Member

    Joined:
    Mar 7, 2010
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Hi, I'm working with ee99ee. We were using the pkgacct to avoid issues when restoring backups, if we just rsync the /home and the db folder the restore task will be a mess!
    Do anyone has a tip to turn the backups faster ? Or can describe how the rsync backup is done on your hosts ?

    :)
     
  7. hostultra

    hostultra Well-Known Member

    Joined:
    Aug 21, 2002
    Messages:
    167
    Likes Received:
    0
    Trophy Points:
    16
    I said /var/db etc..

    The etc.. means all the folders cpanel uses.
    A restore can be done by just putting those folders back in the proper places, and I have done it many times, without loosing any databases or configurations, or having to re-add domains or email accounts.

    Actually the way I do it is I rsync the entire / directory, with an exclude list containing what I dont want to rsync, like /dev and large log folders.
     
  8. Miraenda

    Miraenda Well-Known Member

    Joined:
    Jul 28, 2004
    Messages:
    242
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Coralville, Iowa USA
    Since the actual list of files and folders to rsync hasn't been noted, I'll provide a list of the files and folders I've used personally for cPanel rsyncs in the past:

    After rsyncing, if your machines are on different OS or architectures, please remove the following file on the new server:

    Code:
    /var/cpanel/sysinfo.config
    That file contains data on the old server's OS and architecture and is not interchangeable between systems.

    If the old server IPs will be used for the new server, you might well want these files (be careful copying these otherwise as they'll change your main server IP to the old server's IP and should only be copied if you intend to switch to the new server shortly after a reboot with the old IP becoming the main IP on that new server):

    Code:
    /etc/sysconfig/network
    /etc/sysconfig/network-scripts/ifcfg-eth0
    After you copy over these files and folders, then you would want to rebuild Apache. The above files should include all the user data you need.

    Normally, I would then create a file in /tmp on the new server called movelist and include the files and folders, then I would run the following command on the new server:

    Code:
    for file in `cat /tmp/movelist` ;do rsync --archive --verbose --rsh=ssh oldserverIP:$file $file --exclude '/var/cpanel/sysinfo.config' ;done
    Replacing oldserverIP with the IP for the old server.

    Please note that you cannot use cpmove or cPanel restore files when using this method as the two methods are not interchangeable. It is either cPanel backups and restores or rsyncs. This is because the cPanel backups when being restored on a system presume the account doesn't have files or folders already existing for the data. If the account exists in any fashion (the username in /etc/passwd, /home/user/ exists, and so on), then the cPanel backup cannot restore via WHM on that system, although you can force a restore in root SSH using the --force flag.

    I also would highly suggest using dumps for MySQL data instead of this method if you have InnoDB MySQL database tables. If MySQL is running when you do an rsync like this for backups, the /var/lib/mysql/ibdata1 file might corrupt. A corrupted InnoDB table or tables can cause MySQL to not start, so you are risking MySQL not working by using an rsync. It's better to get a dump of the databases on top of the rsync for MySQL.
     
    #8 Miraenda, Jul 10, 2010
    Last edited: Jul 10, 2010
  9. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    425
    Likes Received:
    0
    Trophy Points:
    16
    Hello Miraenda :

    I was looking for such a script to sync 2 servers, since we have migrated a server from an OLD hardware to New hardware.

    Since i have copied the data from the old to the new via cpanel copy feature i still need to sync the old with the new to cover up the changed data, Can i use the above script to achieve the same ?

    Thank you
     
  10. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    furquan, you can use the same to re-sync the data from the old machine to the new machine for the changed files and folders. The post was made by my non-staff account at the time, so I'm the same person who did the prior post anyway.
     
  11. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    425
    Likes Received:
    0
    Trophy Points:
    16
    Thank you very much Tristan :), highly appreciate that..i'd like to tell you, your post on the forums have been very helpful.

    Thank you.
     
  12. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Certainly, you are very welcome and thank you very much for the kind words :)
     
  13. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    425
    Likes Received:
    0
    Trophy Points:
    16
    Tristan :

    please pardon me for bumping this thread once again :( but i have a small query.

    I am about to migrate a server "abc" within the same DC from Physical TO Virtual., Now the issue is,

    Since the accounts will be copied / transfered over to the virtual server on a temporary ip address during the copy process , BUT
    The sites will still need to use the same OLD ip after the migration, Hence my question,

    How can i change the A record of all the sites in one go on the virtual machine after i have copied/synced over my data from the physical machine and give all my accounts the actual IP address they were actually on. ?

    Please assist ..
     
  14. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Why do the accounts need to be transferred onto a temporary IP address during the copy process? If you are rsyncing the data, then you are not copying and moving them to a new IP address space in those files themselves. You can simply keep them on the old IP in the files, then use the process to copy the IP over that I indicated for those IP files. Upon reboot, the machine will take the IP from the old machine and, since all the files for the rsync data already have that old IP, there isn't any new IP or transfer of the IP that would take place for the accounts.

    If you are using the cPanel backup and transfer instead, then yes you would need to change the IPs and have to do it using WHM > Change Site's IP Address area or use the multiple IP change area instead.
     
  15. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    425
    Likes Received:
    0
    Trophy Points:
    16
    Since i cannot afford down time, i will copy the data on the temporary IP address using "Copy multiple accounts/packages from another server" from WHM and then rsync the data using the script above and then in the end change IP address of the VM, But since only one ip can be live during all this while, i will first copy the accounts to a temporary account and then switch them back to their OLD ip address ...
     
  16. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    You will need to use the IP migration wizard and select all sites to go to the old IP after you add the old IP to the machine. Since you are going to use the cPanel process for transfer, you have no choice but to use the WHM > IP Migration Wizard to change the IPs using that method afterward.
     
  17. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    425
    Likes Received:
    0
    Trophy Points:
    16
    Tristan :

    I think i found my answer :D, "multiple IP change " does exist in WHM, i had completely overlooked it, and hence this confusion.

    I can copy the accounts over to this temporary ip and after i sync the data, i can swith the IP's of all the sites at once using the above whm feature :)

    Thank you for pointing that out to me !!
     
  18. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    425
    Likes Received:
    0
    Trophy Points:
    16
    "multiple IP change " is a tedious process for changing the ip's of all the sites on the server, I guess "P Migration Wizard" is what i will have to use.
     
  19. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    425
    Likes Received:
    0
    Trophy Points:
    16
    Tristan :

    Need some assistance here, when i try to use the rsync script above and try to log on to the OLD server, I get a message again and again asking for password and also :-
    Can you please shed some light on this ?

    Thank you
     
  20. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Can you SSH from the one server to the other with the login credentials you are trying to use? It seems it isn't able to authenticate to run the rsync from the one machine to the other. It requires regular SSH login ability to do any SSH commands such as rsync as it would for anything else in SSH.
     
Loading...

Share This Page