Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Legacy restore, why copy twice?

Discussion in 'Data Protection' started by mindnetcombr, Aug 15, 2017.

Tags:
  1. mindnetcombr

    mindnetcombr Well-Known Member

    Joined:
    Sep 16, 2004
    Messages:
    65
    Likes Received:
    3
    Trophy Points:
    158
    I know legacy backup/restore is dead, but today I need restore a cpanel server with legacy restore, 1Tb of data.

    All data stored on a backup hdd, without compression, without .tar or .tar.gz - my backup is incremental, file by file, folder by folder.

    This is how legacy restore work:

    1) copy the account backup to a temporary folder inside the backup disk;
    2) copy the data copied to temporary folder, to the final destination (/home);

    Repeat this 600 times, with 1Tb of data.

    A big waste of time and resources, why legacy copy to a temporary location to after this restore the account? Why not restore direct from /backup/cpbackup/daily/account_login/ ????

    This temporary copy double the time necessary to restore, and stress the backup disk with a lot of writes.

    I dont know how the new restore system work, I hope not in the same way. This dont make sense, just lock backups while do restore.
     
  2. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,888
    Likes Received:
    90
    Trophy Points:
    78
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hi,

    The simplest way for you would be that you just create an archive from those user backup folder and restore it directly..

    # tar -zcvf user.tar.gz user

    This will create a tar and then you can restore it directly..
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. mindnetcombr

    mindnetcombr Well-Known Member

    Joined:
    Sep 16, 2004
    Messages:
    65
    Likes Received:
    3
    Trophy Points:
    158

    Yes but I will lose more time, to compact the backup to .tar.gz, and when use /restorepkg /backup/cpbackup/daily/user.tar.gz this script will create temporary files again on /backup/cpabackup/daily/cpanelpkgrestore.TMP.work.SOMETHING/

    Compact to .tar.gz to restore will not avoid cpanel to use the temporary folder.

    This issue is similar to this one:

    restorepkg can't restore from a read-only file system

    cpanel legacy restore ALWAYS copy the backup to a temporary folder before restore. This step is not necessary since in a incremental backup all files are uncompressed.
     
  4. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,264
    Likes Received:
    1,940
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    You can browse to "WHM >> Backup Configuration", configure the "Incremental" backup type, configure the local backup directory to the partition where the incremental backups are stored, and then move the existing incremental backups to the "/$configured-backup-dir/date/accounts" location. Once you do this, the backups should appear in WHM's Backup Restoration interface under the appropriate account and date.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. mindnetcombr

    mindnetcombr Well-Known Member

    Joined:
    Sep 16, 2004
    Messages:
    65
    Likes Received:
    3
    Trophy Points:
    158
    Michael, in this case the "new" backup system will restore without create a temporary copy? Restore will go direct from date/accounts to user account?

    thank
     
  6. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,888
    Likes Received:
    90
    Trophy Points:
    78
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hi,

    Restoration will create a temporary session for restoration. What exact is the issue you are facing with temporary sessions/copy?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. mindnetcombr

    mindnetcombr Well-Known Member

    Joined:
    Sep 16, 2004
    Messages:
    65
    Likes Received:
    3
    Trophy Points:
    158

    the issue is the time.

    for every account restored I need wait cpanel create a temporary copy. So one account with 80Gb, 200.000 inodes, this take almost 1 hour.

    I have 600 accounts, fresh hardware, and more than 30 hours of restore.

    If dont have the temporary copy, this will take half time.

    This is the issue.

    thank
     
  8. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,888
    Likes Received:
    90
    Trophy Points:
    78
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hi,

    I would suggest you do as suggested by the Michael for one account and see what occurs and how much time it takes for it to restore..
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. mindnetcombr

    mindnetcombr Well-Known Member

    Joined:
    Sep 16, 2004
    Messages:
    65
    Likes Received:
    3
    Trophy Points:
    158
    Yes I will! thank everyone!
     
  10. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,264
    Likes Received:
    1,940
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Temporarily files are utilized, however the restore feature runs a check before starting the restoration process to ensure enough disk space is available. EX:

    Code:
    Target “/backup/2017-08-17/accounts” on host “123.hostname.tld” has 11.16 GB free and requires at least 67.4 MB free, which includes space for temporary files. 
    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. mindnetcombr

    mindnetcombr Well-Known Member

    Joined:
    Sep 16, 2004
    Messages:
    65
    Likes Received:
    3
    Trophy Points:
    158
    Hello,

    Today I run a test: using the new backup system, I restored one account with 14Gb.

    Unfortunately the backup system first copie the entire backup (14Gb) to a temporary directory /backup/date/accounts/cpanelpkgrestore.TMP.work.RuFrBXh5

    Well... I have the incremental backup, so all files on my backup are ready to be restored (no .tar or .tar.gz).

    The restore system copy all data to a temporary folder, and only after this restore the account - dont make any sense with a incremental backup setup.

    This "step" (copy to temporary folder) only add time to restore and unnecessarily generates writing on the backup disk, which may be undesirable, because when restoring a backup is a moment that can be critical and the last thing we want is to stress the backup disk.

    I understand when the backup have the format .tar or .tar.gz is necessary copy to a temporary folder to extract the files, but with "incremental" option this temporary copy is useless.

    Maybe someone from cpanel reads this and agrees with my argument, if any, thank you bring the matter to the developers. There could be an automatic scan if the backup is incremental so do not use temporary folder.

    To finish: 1Tb restore of 600 accounts, in a new hardware (Xeon 1270 V6 + 32Gb RAM + Adaptec RAID1 7.2K) took 30 hours. Half this spent with cpanel copying backup to temporary folders unnecessarily.

    Thank
     
  12. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,264
    Likes Received:
    1,940
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    The following feature would likely help to some extent if you did not need to restore the entire account:

    cPanel user Granular backup/restore

    However, I encourage you to open a separate feature request if you'd like to see new functionality that doesn't make use of a temporary directory for restoration purposes:

    Submit A Feature Request

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. mindnetcombr

    mindnetcombr Well-Known Member

    Joined:
    Sep 16, 2004
    Messages:
    65
    Likes Received:
    3
    Trophy Points:
    158


    Thank you.

    I will do this, and I will pray that other users will need to do incremental backup restore faster, so maybe - with many votes - cpanel.net decides to make this small change.
     
  14. Nguger Tarhely

    Nguger Tarhely Registered

    Joined:
    Dec 22, 2017
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Budapest
    cPanel Access Level:
    DataCenter Provider
    We are struggling with the same problem. We have a file level backup from users accounts on a remote server. After we mount the remote server's disk, and try to restore the backup with /scripts/restorepkg /mnt/something/$USERNAME, the script make a copy from the given user's backup, which make a very big - and unnecessary - overhead.

    Do you have any update about this topic? If you submitted a feature request, can you share the request's link, please?
     
  15. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,264
    Likes Received:
    1,940
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hi @Nguger Tarhely,

    I believe this is the feature request you are looking for:

    Restore backup from FTP storage

    In the meantime, how large are the individual accounts you are attempting to restore? You may want to consider restoring the accounts without the home directory, and then manually moving the home directory in place after the account itself is restored.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. Nguger Tarhely

    Nguger Tarhely Registered

    Joined:
    Dec 22, 2017
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Budapest
    cPanel Access Level:
    DataCenter Provider
    Our problem is a little bit different.
    We have uncompressed backup on remote server. We attached the backup server's file system with NFS (we tried with ISCSI also, but the problem is the same).
    We can restore the user from backup with /scripts/restorepkg /mnt/something/randomusername, where "randomusername" folder contains the uncompressed backup:

    randomusername
    randomusername/cron
    randomusername/bandwidth
    randomusername/autossl.json
    randomusername/mma
    randomusername/logs
    randomusername/psql
    randomusername/cp
    randomusername/ssl
    randomusername/va
    randomusername/sslcerts
    randomusername/has_sslstorage
    randomusername/suspendinfo
    randomusername/mm
    randomusername/sslkeys
    randomusername/bandwidth_db.json
    randomusername/resellerfeatures
    randomusername/resellerconfig
    randomusername/psql_grants.sql
    randomusername/shadow
    randomusername/shell
    randomusername/homedir
    randomusername/version
    randomusername/ips
    randomusername/bandwidth_db
    randomusername/digestshadow
    randomusername/meta
    randomusername/sds2
    randomusername/vf
    randomusername/mms
    randomusername/ssldomain
    randomusername/userconfig
    randomusername/locale
    randomusername/mysql-timestamps
    randomusername/addons
    randomusername/resellerpackages
    randomusername/domainkeys
    randomusername/apache_tls
    randomusername/quota
    randomusername/bandwidth_db_data.json
    randomusername/counters
    randomusername/httpfiles
    randomusername/mysql.sql
    randomusername/proftpdpasswd
    randomusername/dnszones
    randomusername/sds
    randomusername/homedir_paths
    randomusername/vad
    randomusername/userdata
    randomusername/suspended
    randomusername/psql_users.sql
    randomusername/mysql
    randomusername/pds

    When we run the /scripts/restorepkg /mnt/something/randomusername the restorepkg make a full copy from /mnt/something/randomusername to /mnt/something/cpanelpkgrestoreXXXXX, see below:

    [ 17059][RESTORE:1 ][A:randomusername ]: Target “/mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk/unsafe_to_read_archive” on host “server.example.com” has 806.22 GB free and requires at least 491.08 MB free, which includes space for temporary files.
    [ 17059][RESTORE:1 ][A:randomusername ]: Target “/mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk/unsafe_to_read_archive” on host “server.example.com” has 60,503,178 inodes free and requires at least 1,472 inodes free, which includes space for temporary files.

    During the restorepkg, I ran du -sh /mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk which returned with nearly the same size as mentoined on the restorepkg's output (494 Mb).

    After the copy, the /scripts/restorepkg restore the user, but this copy is clearly unneccessary and make big overhead, because the data copied twice. Why is it important to copy the uncompressed backup to another folder? Can the restorepkg skip this step?

    Also, I checked the /scripts/restorepkg help, but it does not have any option to skip home directory:
    /usr/local/cpanel/bin/restorepkg [--unrestricted] [--restricted] [--force] [--newuser <username>] [--allow_reseller] [--skipaccount] [--ip=(y|n|custom IP)] -- [cpuser|/path/to/cpuser-file|/path/to/extracted-cpuser-file|/path/to/incremental_backup]

    --allow_reseller Restore reseller privileges if the archive contains them.
    NOTE: This option is only available in unrestricted mode.

    --force Force the package to restore and overwrite all account settings
    and databases. If the account already exists on the system
    this implies --skipaccount.

    --ip Assign a dedicated IP address. You may specify a particular
    IP address, or “y” to have one randomly assigned.

    --newuser Specify the username to use for the restored account instead
    of the one contained within the archive. Databases and DB
    users will retain their names unless a conflict is detected.
    This option cannot be used with --skipaccount.

    --skipaccount Restore the package into the existing account with the same
    username. This option cannot be used in conjunction with
    --newuser.

    --restricted Use the Restricted Restore feature to restore the package.

    --unrestricted Restore the package in unrestricted mode. This flag is on by
    default and overridden by the --restricted flag.

    If we can skip the home directory, it would be nice.

    If you need more details, please let me know.
     
  17. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,264
    Likes Received:
    1,940
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    The temporary directory is for staging purposes during the restoration process. It's how the integrity of the backup archive used for the restore is validated. You could open a separate feature request if you have a specific request to change this behavior. As far as restoring the account without the home directory in the situation you described, you'd need to move the account's public_html directory from the directory you are restoring it from to another location (outside of the account).

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice