Legacy restore, why copy twice?

mindnetcombr

Well-Known Member
Sep 16, 2004
70
5
158
I know legacy backup/restore is dead, but today I need restore a cpanel server with legacy restore, 1Tb of data.

All data stored on a backup hdd, without compression, without .tar or .tar.gz - my backup is incremental, file by file, folder by folder.

This is how legacy restore work:

1) copy the account backup to a temporary folder inside the backup disk;
2) copy the data copied to temporary folder, to the final destination (/home);

Repeat this 600 times, with 1Tb of data.

A big waste of time and resources, why legacy copy to a temporary location to after this restore the account? Why not restore direct from /backup/cpbackup/daily/account_login/ ????

This temporary copy double the time necessary to restore, and stress the backup disk with a lot of writes.

I dont know how the new restore system work, I hope not in the same way. This dont make sense, just lock backups while do restore.
 

24x7server

Well-Known Member
Apr 17, 2013
1,911
96
78
India
cPanel Access Level
Root Administrator
Twitter
Hi,

The simplest way for you would be that you just create an archive from those user backup folder and restore it directly..

# tar -zcvf user.tar.gz user

This will create a tar and then you can restore it directly..
 

mindnetcombr

Well-Known Member
Sep 16, 2004
70
5
158
Hi,

The simplest way for you would be that you just create an archive from those user backup folder and restore it directly..

# tar -zcvf user.tar.gz user

This will create a tar and then you can restore it directly..

Yes but I will lose more time, to compact the backup to .tar.gz, and when use /restorepkg /backup/cpbackup/daily/user.tar.gz this script will create temporary files again on /backup/cpabackup/daily/cpanelpkgrestore.TMP.work.SOMETHING/

Compact to .tar.gz to restore will not avoid cpanel to use the temporary folder.

This issue is similar to this one:

restorepkg can't restore from a read-only file system

cpanel legacy restore ALWAYS copy the backup to a temporary folder before restore. This step is not necessary since in a incremental backup all files are uncompressed.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello,

You can browse to "WHM >> Backup Configuration", configure the "Incremental" backup type, configure the local backup directory to the partition where the incremental backups are stored, and then move the existing incremental backups to the "/$configured-backup-dir/date/accounts" location. Once you do this, the backups should appear in WHM's Backup Restoration interface under the appropriate account and date.

Thank you.
 

mindnetcombr

Well-Known Member
Sep 16, 2004
70
5
158
Hello,

You can browse to "WHM >> Backup Configuration", configure the "Incremental" backup type, configure the local backup directory to the partition where the incremental backups are stored, and then move the existing incremental backups to the "/$configured-backup-dir/date/accounts" location. Once you do this, the backups should appear in WHM's Backup Restoration interface under the appropriate account and date.

Thank you.
Michael, in this case the "new" backup system will restore without create a temporary copy? Restore will go direct from date/accounts to user account?

thank
 

mindnetcombr

Well-Known Member
Sep 16, 2004
70
5
158
Hi,

Restoration will create a temporary session for restoration. What exact is the issue you are facing with temporary sessions/copy?

the issue is the time.

for every account restored I need wait cpanel create a temporary copy. So one account with 80Gb, 200.000 inodes, this take almost 1 hour.

I have 600 accounts, fresh hardware, and more than 30 hours of restore.

If dont have the temporary copy, this will take half time.

This is the issue.

thank
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Michael, in this case the "new" backup system will restore without create a temporary copy? Restore will go direct from date/accounts to user account?
Temporarily files are utilized, however the restore feature runs a check before starting the restoration process to ensure enough disk space is available. EX:

Code:
Target “/backup/2017-08-17/accounts” on host “123.hostname.tld” has 11.16 GB free and requires at least 67.4 MB free, which includes space for temporary files.
Thank you.
 

mindnetcombr

Well-Known Member
Sep 16, 2004
70
5
158
Hello,

Today I run a test: using the new backup system, I restored one account with 14Gb.

Unfortunately the backup system first copie the entire backup (14Gb) to a temporary directory /backup/date/accounts/cpanelpkgrestore.TMP.work.RuFrBXh5

Well... I have the incremental backup, so all files on my backup are ready to be restored (no .tar or .tar.gz).

The restore system copy all data to a temporary folder, and only after this restore the account - dont make any sense with a incremental backup setup.

This "step" (copy to temporary folder) only add time to restore and unnecessarily generates writing on the backup disk, which may be undesirable, because when restoring a backup is a moment that can be critical and the last thing we want is to stress the backup disk.

I understand when the backup have the format .tar or .tar.gz is necessary copy to a temporary folder to extract the files, but with "incremental" option this temporary copy is useless.

Maybe someone from cpanel reads this and agrees with my argument, if any, thank you bring the matter to the developers. There could be an automatic scan if the backup is incremental so do not use temporary folder.

To finish: 1Tb restore of 600 accounts, in a new hardware (Xeon 1270 V6 + 32Gb RAM + Adaptec RAID1 7.2K) took 30 hours. Half this spent with cpanel copying backup to temporary folders unnecessarily.

Thank
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello,

The following feature would likely help to some extent if you did not need to restore the entire account:

cPanel user Granular backup/restore

However, I encourage you to open a separate feature request if you'd like to see new functionality that doesn't make use of a temporary directory for restoration purposes:

Submit A Feature Request

Thank you.
 

mindnetcombr

Well-Known Member
Sep 16, 2004
70
5
158
Hello,

The following feature would likely help to some extent if you did not need to restore the entire account:

cPanel user Granular backup/restore

However, I encourage you to open a separate feature request if you'd like to see new functionality that doesn't make use of a temporary directory for restoration purposes:

Submit A Feature Request

Thank you.


Thank you.

I will do this, and I will pray that other users will need to do incremental backup restore faster, so maybe - with many votes - cpanel.net decides to make this small change.
 

Nguger Tarhely

Registered
Dec 22, 2017
3
0
1
Budapest
cPanel Access Level
DataCenter Provider
We are struggling with the same problem. We have a file level backup from users accounts on a remote server. After we mount the remote server's disk, and try to restore the backup with /scripts/restorepkg /mnt/something/$USERNAME, the script make a copy from the given user's backup, which make a very big - and unnecessary - overhead.

Thank you.

I will do this, and I will pray that other users will need to do incremental backup restore faster, so maybe - with many votes - cpanel.net decides to make this small change.
Do you have any update about this topic? If you submitted a feature request, can you share the request's link, please?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hi @Nguger Tarhely,

I believe this is the feature request you are looking for:

Restore backup from FTP storage

In the meantime, how large are the individual accounts you are attempting to restore? You may want to consider restoring the accounts without the home directory, and then manually moving the home directory in place after the account itself is restored.

Thank you.
 

Nguger Tarhely

Registered
Dec 22, 2017
3
0
1
Budapest
cPanel Access Level
DataCenter Provider
Our problem is a little bit different.
We have uncompressed backup on remote server. We attached the backup server's file system with NFS (we tried with ISCSI also, but the problem is the same).
We can restore the user from backup with /scripts/restorepkg /mnt/something/randomusername, where "randomusername" folder contains the uncompressed backup:

randomusername
randomusername/cron
randomusername/bandwidth
randomusername/autossl.json
randomusername/mma
randomusername/logs
randomusername/psql
randomusername/cp
randomusername/ssl
randomusername/va
randomusername/sslcerts
randomusername/has_sslstorage
randomusername/suspendinfo
randomusername/mm
randomusername/sslkeys
randomusername/bandwidth_db.json
randomusername/resellerfeatures
randomusername/resellerconfig
randomusername/psql_grants.sql
randomusername/shadow
randomusername/shell
randomusername/homedir
randomusername/version
randomusername/ips
randomusername/bandwidth_db
randomusername/digestshadow
randomusername/meta
randomusername/sds2
randomusername/vf
randomusername/mms
randomusername/ssldomain
randomusername/userconfig
randomusername/locale
randomusername/mysql-timestamps
randomusername/addons
randomusername/resellerpackages
randomusername/domainkeys
randomusername/apache_tls
randomusername/quota
randomusername/bandwidth_db_data.json
randomusername/counters
randomusername/httpfiles
randomusername/mysql.sql
randomusername/proftpdpasswd
randomusername/dnszones
randomusername/sds
randomusername/homedir_paths
randomusername/vad
randomusername/userdata
randomusername/suspended
randomusername/psql_users.sql
randomusername/mysql
randomusername/pds

When we run the /scripts/restorepkg /mnt/something/randomusername the restorepkg make a full copy from /mnt/something/randomusername to /mnt/something/cpanelpkgrestoreXXXXX, see below:

[ 17059][RESTORE:1 ][A:randomusername ]: Target “/mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk/unsafe_to_read_archive” on host “server.example.com” has 806.22 GB free and requires at least 491.08 MB free, which includes space for temporary files.
[ 17059][RESTORE:1 ][A:randomusername ]: Target “/mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk/unsafe_to_read_archive” on host “server.example.com” has 60,503,178 inodes free and requires at least 1,472 inodes free, which includes space for temporary files.

During the restorepkg, I ran du -sh /mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk which returned with nearly the same size as mentoined on the restorepkg's output (494 Mb).

After the copy, the /scripts/restorepkg restore the user, but this copy is clearly unneccessary and make big overhead, because the data copied twice. Why is it important to copy the uncompressed backup to another folder? Can the restorepkg skip this step?

Also, I checked the /scripts/restorepkg help, but it does not have any option to skip home directory:
/usr/local/cpanel/bin/restorepkg [--unrestricted] [--restricted] [--force] [--newuser <username>] [--allow_reseller] [--skipaccount] [--ip=(y|n|custom IP)] -- [cpuser|/path/to/cpuser-file|/path/to/extracted-cpuser-file|/path/to/incremental_backup]

--allow_reseller Restore reseller privileges if the archive contains them.
NOTE: This option is only available in unrestricted mode.

--force Force the package to restore and overwrite all account settings
and databases. If the account already exists on the system
this implies --skipaccount.

--ip Assign a dedicated IP address. You may specify a particular
IP address, or “y” to have one randomly assigned.

--newuser Specify the username to use for the restored account instead
of the one contained within the archive. Databases and DB
users will retain their names unless a conflict is detected.
This option cannot be used with --skipaccount.

--skipaccount Restore the package into the existing account with the same
username. This option cannot be used in conjunction with
--newuser.

--restricted Use the Restricted Restore feature to restore the package.

--unrestricted Restore the package in unrestricted mode. This flag is on by
default and overridden by the --restricted flag.

If we can skip the home directory, it would be nice.

If you need more details, please let me know.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello,

The temporary directory is for staging purposes during the restoration process. It's how the integrity of the backup archive used for the restore is validated. You could open a separate feature request if you have a specific request to change this behavior. As far as restoring the account without the home directory in the situation you described, you'd need to move the account's public_html directory from the directory you are restoring it from to another location (outside of the account).

Thank you.