Backup restoration causing high load averages?

Hey there Ricardo,

You may be able to do it using CPU watch through SSH like this:

/usr/local/cpanel/bin/cpuwatch 4 /scripts/restorepkg username

This should have the CPU watch keep an eye on the process and pause it when the load goes over 4 and if it hits that load it pauses the process until the load is under that specific number again. For your server you may want to do a higher number but that should be a good way for that.

 

I’m sorry to hear that. I did see that Michael recommended to make a ticket. Can you update the thread when you can to see what happened? I’m pretty curious on the outcome of the ticket.

 

I have opened five Oracle service requests for multiple symptoms.would hang, the agent would hang, backup jobs would hang,

Great Thinking your feedback is awesome

 

Hello,

I got the same problem, is there a way to do the limit as default ?
When i restore account it dose high load, even when i restore it from cpmove file (WHM), even when some customer restore hes website when i was using jetbackup (we uninstaled because the server was getting crazy) high load 30 40 % and usualy the load is 2, 3

Thank you

 

I opened a ticket to JetBackup about this restore issue. They answer:

So no luck.

 

Hello rarod,

Can you please update us if you were able to make some progress with this issue?
At a glance it looks like an IO issue, usually caused when backup folder and data folder are on the same mount, or when the server is over crowded with huge IOwait. If you're still encountering these issues, please let us know via our Helpdesk so our support techs can assist you!

 

I can add some details about this: I have high IO during restores, in theory JetBackup has options to use CloudLinux LVE for limit IO. But this is not working in restores because some restore processess run with root user instead of regular user.

Backup destination is a remote server.

I already have opened ticket: #433207. This was closed with:

 

Rarod,

Can you tell if it's a MySQL issue, ?

Thanks,
Eli.

 

I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs.

Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.

 

I agree with this. I've faced similar problems when restoring accounts using Jetbackup. If the account have a large number of files the process can get a spike in cpu load and high io wait. Adding ionice, renice or lve don't seem to work at all

 

Hi,

I asked because you mentioned you opened a ticket with Oracle, so it made sense that it's a MySQL issue.
Does your IOwait usually high ? Can you post some averages ?

As for the ticket you have with us (#433207), can you re-open it and grant me access to the server with permission to restore an account and test it ?
If so, please provide as much information as possible (test account name), and a link to this thread.

Thanks

Eli.

 

This was not me.

Here you can see clearly the restore:

i.imgur.com/El8nJVa.png

I will do it right now

Thank you.

 

JetBackup tested the restore issue in one of my servers. The answer:

I will wait for 3.2 and test if the issue is fixed.

 

The issue is not fixed.

Now we are in Jetbackup 4.0.6 and I had to disable restores in some some of my non-ssd servers to avoid the issue.

 

Rarod,

Thank you for bringing this issue back

From the information we gathered from your server, seems like the server load is caused indeed from cPanel's internal /script/restorepkg script.

I believe that this is a great example for "it's not a bug, it's a feature" case. I'll try to explain -

cPanel's internal /script/restorepkg system should accept "path/to/incremental_backup" as per their help -

Code:

root@cpanel-dev-cpanel home2]# /scripts/restorepkg --help
/usr/local/cpanel/bin/restorepkg [--unrestricted] [--restricted] [--force] [--newuser <username>] [--allow_reseller] [--skipaccount] [--ip=(y|n|custom IP)] [--disable=<module>[,<module>]] -- [cpuser|/path/to/cpuser-file|/path/to/extracted-cpuser-file|/path/to/incremental_backup]

Regardless of JetBackup, let's make a test with a small account -


Create an incremental backup -

/scripts/pkgacct --incremental frooty /home2/

Restore the incremental backup -

/scripts/restorepkg --force /home2/cpmove-frooty


Although we used "incremental" backup to restore the backup, seems like cPanel will create an archive, and then will extract it again -

Code:

 8683 frooty    20   0  121708   1244    960 S  14.0  0.0   0:00.42 /usr/bin/gtar --extract --no-same-owner --preserve-permissions --file - --no-wildcards-m+
 8682 root      20   0  121860   1068    884 D  11.0  0.0   0:00.33 /usr/bin/gtar --create --file - --no-wildcards-match-slash --anchored --sparse --blockin+

This is also documented in the restore log -

Code:

[  6581][RESTORE:1 ][A:frooty          ]: ArchiveManager
[  6581][RESTORE:1 ][A:frooty          ]:               Preparing archive for restoration …
[  6581][RESTORE:1 ][A:frooty          ]:               This archive’s payload appears to be in the archive’s “cpmove-frooty” directory.
[  6581][RESTORE:1 ][A:frooty          ]:       ArchiveManager
[  6581][RESTORE:1 ][A:frooty          ]:       The system successfully prepared the archive for restoration.

When doing this with 18G account, you should expect high load

Going back to JetBackup, we are working on a workaround for this where you should be able to restore files directly to the homedir (reverse rsync), allowing much faster restores and using less CPU.