Backup restoration causing high load averages?

[RicardoFC]

Hi,

My load averages are always something like 3.02 4.23 3.89, however once I restore a site it goes to 30.04 29.93 25.78, as site is large server load average is quite high during a lot of time, is there a way to limit backup restoration to use this much CPU?

 

[Suede Y]

Hey there Ricardo,

You may be able to do it using CPU watch through SSH like this:

/usr/local/cpanel/bin/cpuwatch 4 /scripts/restorepkg username

This should have the CPU watch keep an eye on the process and pause it when the load goes over 4 and if it hits that load it pauses the process until the load is under that specific number again. For your server you may want to do a higher number but that should be a good way for that.

 

[RicardoFC]

Hey there Ricardo,

You may be able to do it using CPU watch through SSH like this:

/usr/local/cpanel/bin/cpuwatch 4 /scripts/restorepkg username

This should have the CPU watch keep an eye on the process and pause it when the load goes over 4 and if it hits that load it pauses the process until the load is under that specific number again. For your server you may want to do a higher number but that should be a good way for that.

Tried, still using that amount of CPU, if the server starts unzipping files it stops the transference process but it keeps unzipping, once it reaches below 4 it justs says the restoration is done

 

[cPanelMichael]

Hello,

Could you open a support ticket using the link in my signature so we can take a closer look and verify if it's mostly related to your server's hardware, or if there's anything in the restoration process that's leading to the increased load?

Thank you.

 

[Suede Y]

Tried, still using that amount of CPU, if the server starts unzipping files it stops the transference process but it keeps unzipping, once it reaches below 4 it justs says the restoration is done

I’m sorry to hear that. I did see that Michael recommended to make a ticket. Can you update the thread when you can to see what happened? I’m pretty curious on the outcome of the ticket.

 

[cPanelMichael]

Hello,

Feel free to post the support ticket number here should you decide to open a ticket and we can update this thread with the outcome.

Thanks!

 

[Bidi]

Hello,

I got the same problem, is there a way to do the limit as default ?
When i restore account it dose high load, even when i restore it from cpmove file (WHM), even when some customer restore hes website when i was using jetbackup (we uninstaled because the server was getting crazy) high load 30 40 % and usualy the load is 2, 3

Thank you

 

[cPanelMichael]

Hello @Bidi,

I encourage you to open a feature request if you'd like to see an option that allows you to configure a default cpuwatch value when using the Restorepkg script:

Submit A Feature Request

Note that generally a backup restoration shouldn't lead to an excessive server load. You may want to look into upgrading your server hardware or checking to ensure there are no hardware issues with your disk.

Thank you.

 

[rarod]

I opened a ticket to JetBackup about this restore issue. They answer:

Hello,

This is not an issue.
You are requesting new feature, and we will consider it.

Thank you,

So no luck.

 

[JetAppsClark]

Hello rarod,

Can you please update us if you were able to make some progress with this issue?
At a glance it looks like an IO issue, usually caused when backup folder and data folder are on the same mount, or when the server is over crowded with huge IOwait. If you're still encountering these issues, please let us know via our Helpdesk so our support techs can assist you!

 

[rarod]

Can you please update us if you were able to make some progress with this issue?

I can add some details about this: I have high IO during restores, in theory JetBackup has options to use CloudLinux LVE for limit IO. But this is not working in restores because some restore processess run with root user instead of regular user.

At a glance it looks like an IO issue, usually caused when backup folder and data folder are on the same mount, or when the server is over crowded with huge IOwait.

Backup destination is a remote server.

f you're still encountering these issues, please let us know via our Helpdesk so our support techs can assist you!

I already have opened ticket: #433207. This was closed with:

"This is not an issue. You are requesting new feature, and we will consider it."

 

[elialum]

Rarod,

Can you tell if it's a MySQL issue, ?

Thanks,
Eli.

 

[rarod]

Rarod,

Can you tell if it's a MySQL issue, ?

Thanks,
Eli.

I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs.

Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.

 

[sys_alex]

I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs.

Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.

I agree with this. I've faced similar problems when restoring accounts using Jetbackup. If the account have a large number of files the process can get a spike in cpu load and high io wait. Adding ionice, renice or lve don't seem to work at all

 

[elialum]

Hi,

I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs.

Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.

I asked because you mentioned you opened a ticket with Oracle, so it made sense that it's a MySQL issue.
Does your IOwait usually high ? Can you post some averages ?

As for the ticket you have with us (#433207), can you re-open it and grant me access to the server with permission to restore an account and test it ?
If so, please provide as much information as possible (test account name), and a link to this thread.

Thanks

Eli.

 

[rarod]

I asked because you mentioned you opened a ticket with Oracle, so it made sense that it's a MySQL issue.

This was not me.

Does your IOwait usually high ? Can you post some averages ?

Here you can see clearly the restore:

i.imgur.com/El8nJVa.png

As for the ticket you have with us (#433207), can you re-open it and grant me access to the server with permission to restore an account and test it ?
If so, please provide as much information as possible (test account name), and a link to this thread.

I will do it right now

Thank you.

 

[rarod]

JetBackup tested the restore issue in one of my servers. The answer:

according the the logs I've collected the major reason for the high IO was "gtar" process in which cPanel "restorepkg" is using when restoring the account.

As you are using JetBackup 3.1, we believe that upgrading to the latest 3.2.15 version could help as we changed the entire concept and shouldn't use gtar while restoring.
Furthermore, with the latest JetBackup 3.2 there is much more debug info which will help us understand better what is going on behind the scenes.

I will wait for 3.2 and test if the issue is fixed.

 

[rarod]

The issue is not fixed.

Now we are in Jetbackup 4.0.6 and I had to disable restores in some some of my non-ssd servers to avoid the issue.

 

[elialum]

Rarod,

Thank you for bringing this issue back

From the information we gathered from your server, seems like the server load is caused indeed from cPanel's internal /script/restorepkg script.

I believe that this is a great example for "it's not a bug, it's a feature" case. I'll try to explain -

cPanel's internal /script/restorepkg system should accept "path/to/incremental_backup" as per their help -

[email protected] home2]# /scripts/restorepkg --help
/usr/local/cpanel/bin/restorepkg [--unrestricted] [--restricted] [--force] [--newuser <username>] [--allow_reseller] [--skipaccount] [--ip=(y|n|custom IP)] [--disable=<module>[,<module>]] -- [cpuser|/path/to/cpuser-file|/path/to/extracted-cpuser-file|/path/to/incremental_backup]

Regardless of JetBackup, let's make a test with a small account -


Create an incremental backup -

/scripts/pkgacct --incremental frooty /home2/

Restore the incremental backup -

/scripts/restorepkg --force /home2/cpmove-frooty


Although we used "incremental" backup to restore the backup, seems like cPanel will create an archive, and then will extract it again -

 8683 frooty    20   0  121708   1244    960 S  14.0  0.0   0:00.42 /usr/bin/gtar --extract --no-same-owner --preserve-permissions --file - --no-wildcards-m+
 8682 root      20   0  121860   1068    884 D  11.0  0.0   0:00.33 /usr/bin/gtar --create --file - --no-wildcards-match-slash --anchored --sparse --blockin+

This is also documented in the restore log -

[  6581][RESTORE:1 ][A:frooty          ]: ArchiveManager
[  6581][RESTORE:1 ][A:frooty          ]:               Preparing archive for restoration …
[  6581][RESTORE:1 ][A:frooty          ]:               This archive’s payload appears to be in the archive’s “cpmove-frooty” directory.
[  6581][RESTORE:1 ][A:frooty          ]:       ArchiveManager
[  6581][RESTORE:1 ][A:frooty          ]:       The system successfully prepared the archive for restoration.

When doing this with 18G account, you should expect high load

Going back to JetBackup, we are working on a workaround for this where you should be able to restore files directly to the homedir (reverse rsync), allowing much faster restores and using less CPU.

 

[rarod]

Hello,

Thanks, I am glad that you are working on a workaround for the feature. Any ETA?