Backup restoration causing high load averages?

RicardoFC

Member
Oct 21, 2017
20
0
1
Spain
cPanel Access Level
Root Administrator
Hi,

My load averages are always something like 3.02 4.23 3.89, however once I restore a site it goes to 30.04 29.93 25.78, as site is large server load average is quite high during a lot of time, is there a way to limit backup restoration to use this much CPU?
 

Suede Y

Registered
Nov 12, 2017
2
0
1
Houston
cPanel Access Level
Root Administrator
Hey there Ricardo,

You may be able to do it using CPU watch through SSH like this:

/usr/local/cpanel/bin/cpuwatch 4 /scripts/restorepkg username

This should have the CPU watch keep an eye on the process and pause it when the load goes over 4 and if it hits that load it pauses the process until the load is under that specific number again. For your server you may want to do a higher number but that should be a good way for that.
 

RicardoFC

Member
Oct 21, 2017
20
0
1
Spain
cPanel Access Level
Root Administrator
Hey there Ricardo,

You may be able to do it using CPU watch through SSH like this:

/usr/local/cpanel/bin/cpuwatch 4 /scripts/restorepkg username

This should have the CPU watch keep an eye on the process and pause it when the load goes over 4 and if it hits that load it pauses the process until the load is under that specific number again. For your server you may want to do a higher number but that should be a good way for that.
Tried, still using that amount of CPU, if the server starts unzipping files it stops the transference process but it keeps unzipping, once it reaches below 4 it justs says the restoration is done :(
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,911
2,234
363
Hello,

Could you open a support ticket using the link in my signature so we can take a closer look and verify if it's mostly related to your server's hardware, or if there's anything in the restoration process that's leading to the increased load?

Thank you.
 

Suede Y

Registered
Nov 12, 2017
2
0
1
Houston
cPanel Access Level
Root Administrator
Tried, still using that amount of CPU, if the server starts unzipping files it stops the transference process but it keeps unzipping, once it reaches below 4 it justs says the restoration is done :(
I’m sorry to hear that. I did see that Michael recommended to make a ticket. Can you update the thread when you can to see what happened? I’m pretty curious on the outcome of the ticket.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,911
2,234
363
Hello,

Feel free to post the support ticket number here should you decide to open a ticket and we can update this thread with the outcome.

Thanks!
 

Bidi

Well-Known Member
Oct 3, 2012
84
2
8
Romania, Transilvania
cPanel Access Level
DataCenter Provider
Hello,

I got the same problem, is there a way to do the limit as default ?
When i restore account it dose high load, even when i restore it from cpmove file (WHM), even when some customer restore hes website when i was using jetbackup (we uninstaled because the server was getting crazy) high load 30 40 % and usualy the load is 2, 3

Thank you
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,911
2,234
363
Hello @Bidi,

I encourage you to open a feature request if you'd like to see an option that allows you to configure a default cpuwatch value when using the Restorepkg script:

Submit A Feature Request

Note that generally a backup restoration shouldn't lead to an excessive server load. You may want to look into upgrading your server hardware or checking to ensure there are no hardware issues with your disk.

Thank you.
 

JetAppsClark

Registered
Aug 16, 2017
2
1
3
California
cPanel Access Level
Root Administrator
Hello rarod,

Can you please update us if you were able to make some progress with this issue?
At a glance it looks like an IO issue, usually caused when backup folder and data folder are on the same mount, or when the server is over crowded with huge IOwait. If you're still encountering these issues, please let us know via our Helpdesk so our support techs can assist you!
 
  • Like
Reactions: cPanelMichael

rarod

Active Member
Apr 20, 2017
36
1
8
Spain
cPanel Access Level
Root Administrator
Can you please update us if you were able to make some progress with this issue?
I can add some details about this: I have high IO during restores, in theory JetBackup has options to use CloudLinux LVE for limit IO. But this is not working in restores because some restore processess run with root user instead of regular user.

At a glance it looks like an IO issue, usually caused when backup folder and data folder are on the same mount, or when the server is over crowded with huge IOwait.
Backup destination is a remote server.

f you're still encountering these issues, please let us know via our Helpdesk so our support techs can assist you!
I already have opened ticket: #433207. This was closed with:
"This is not an issue. You are requesting new feature, and we will consider it."
 

elialum

Active Member
Sep 10, 2008
29
1
53
Israel
cPanel Access Level
DataCenter Provider
Rarod,

Can you tell if it's a MySQL issue, ?

Thanks,
Eli.
 

rarod

Active Member
Apr 20, 2017
36
1
8
Spain
cPanel Access Level
Root Administrator
Rarod,

Can you tell if it's a MySQL issue, ?

Thanks,
Eli.
I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs.

Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.
 
Last edited:

sys_alex

Member
Apr 23, 2018
18
6
3
Spain, Galicia
cPanel Access Level
Root Administrator
I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs.

Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.
I agree with this. I've faced similar problems when restoring accounts using Jetbackup. If the account have a large number of files the process can get a spike in cpu load and high io wait. Adding ionice, renice or lve don't seem to work at all
 
  • Like
Reactions: rarod

elialum

Active Member
Sep 10, 2008
29
1
53
Israel
cPanel Access Level
DataCenter Provider
Hi,

I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs.

Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.
I asked because you mentioned you opened a ticket with Oracle, so it made sense that it's a MySQL issue.
Does your IOwait usually high ? Can you post some averages ?

As for the ticket you have with us (#433207), can you re-open it and grant me access to the server with permission to restore an account and test it ?
If so, please provide as much information as possible (test account name), and a link to this thread.

Thanks :)

Eli.
 

rarod

Active Member
Apr 20, 2017
36
1
8
Spain
cPanel Access Level
Root Administrator
I asked because you mentioned you opened a ticket with Oracle, so it made sense that it's a MySQL issue.
This was not me.

Does your IOwait usually high ? Can you post some averages ?
Here you can see clearly the restore:

i.imgur.com/El8nJVa.png

As for the ticket you have with us (#433207), can you re-open it and grant me access to the server with permission to restore an account and test it ?
If so, please provide as much information as possible (test account name), and a link to this thread.
I will do it right now

Thank you.
 
Last edited by a moderator:

rarod

Active Member
Apr 20, 2017
36
1
8
Spain
cPanel Access Level
Root Administrator
JetBackup tested the restore issue in one of my servers. The answer:

according the the logs I've collected the major reason for the high IO was "gtar" process in which cPanel "restorepkg" is using when restoring the account.

As you are using JetBackup 3.1, we believe that upgrading to the latest 3.2.15 version could help as we changed the entire concept and shouldn't use gtar while restoring.
Furthermore, with the latest JetBackup 3.2 there is much more debug info which will help us understand better what is going on behind the scenes.
I will wait for 3.2 and test if the issue is fixed.
 

elialum

Active Member
Sep 10, 2008
29
1
53
Israel
cPanel Access Level
DataCenter Provider
Rarod,

Thank you for bringing this issue back :)

From the information we gathered from your server, seems like the server load is caused indeed from cPanel's internal /script/restorepkg script.

I believe that this is a great example for "it's not a bug, it's a feature" case. I'll try to explain -

cPanel's internal /script/restorepkg system should accept "path/to/incremental_backup" as per their help -

Code:
[email protected] home2]# /scripts/restorepkg --help
/usr/local/cpanel/bin/restorepkg [--unrestricted] [--restricted] [--force] [--newuser <username>] [--allow_reseller] [--skipaccount] [--ip=(y|n|custom IP)] [--disable=<module>[,<module>]] -- [cpuser|/path/to/cpuser-file|/path/to/extracted-cpuser-file|/path/to/incremental_backup]
Regardless of JetBackup, let's make a test with a small account -


Create an incremental backup -

/scripts/pkgacct --incremental frooty /home2/

Restore the incremental backup -

/scripts/restorepkg --force /home2/cpmove-frooty


Although we used "incremental" backup to restore the backup, seems like cPanel will create an archive, and then will extract it again -

Code:
 8683 frooty    20   0  121708   1244    960 S  14.0  0.0   0:00.42 /usr/bin/gtar --extract --no-same-owner --preserve-permissions --file - --no-wildcards-m+
 8682 root      20   0  121860   1068    884 D  11.0  0.0   0:00.33 /usr/bin/gtar --create --file - --no-wildcards-match-slash --anchored --sparse --blockin+
This is also documented in the restore log -

Code:
[  6581][RESTORE:1 ][A:frooty          ]: ArchiveManager
[  6581][RESTORE:1 ][A:frooty          ]:               Preparing archive for restoration …
[  6581][RESTORE:1 ][A:frooty          ]:               This archive’s payload appears to be in the archive’s “cpmove-frooty” directory.
[  6581][RESTORE:1 ][A:frooty          ]:       ArchiveManager
[  6581][RESTORE:1 ][A:frooty          ]:       The system successfully prepared the archive for restoration.
When doing this with 18G account, you should expect high load :)

Going back to JetBackup, we are working on a workaround for this where you should be able to restore files directly to the homedir (reverse rsync), allowing much faster restores and using less CPU.
 
Last edited: