reducing load and disk I/O for cpbackup with CloudLinux

[Ebridge]

I've been struggling with cpbackup putting too much stress on the server (disk latency peaks, high CPU load and http load time peaks of multiple seconds, sometimes more than 10 seconds for a very short period of time) during backups. Tried all kinds of optimizations but nothing really worked. Pigz (cPanel 11.38) reduced the time for the backups to complete a bit, but the peaks stayed just as high.

Judging from the Munin-graphs, the disk utilization (peaking regularly at 100% during backups, especially for extremely large accounts) seemed to be the evildoer.

As I'm running the CloudLinux kernel (which I really like anyway) I got the idea to use its command line LVE-wrapper (Lightweight Virtual Environment) to wrap the cpbackup root process into an LVE, and then try to use the I/O throttling option of CloudLinux to moderate the disk I/O to more friendly values.

It took a bit of experimenting, but it works really well!

Average CPU load during backups dropped from 2 to less than 1. Even better is the reduction of the peak CPU load: from 6 down to 1 (practically no CPU peaks during backups)


An important real life test of course is the http loadtime. Loadtime during backups was reduced from 0.5 seconds average and more than 3 seconds peak to 0.15 seconds average and 0.5 seconds peak.

First days in the graphs show the 'old' situation, then I started experimenting with the LVE-wrapper parameters (resulting in varying values), and last night seems to be the winning configuration. The peak values were read from the graphs, but actual peaks can be much higher (but too short duration for Munin to register)

Overall, the server felt almost as responsive during backups as during daytime, which is a big win.
I'm sure some greater gains can be achieved by finetuning the LVE wrapper parameters some more but for now I'm happy with the results.

For reference some info on this server:
4 cores, 4GB memory, cPanel 11.38, CloudLinux 5 hybrid. Hosting a lot of large (4-16GB) accounts. Daily full backups over FTP to backup server.

I do understand this is the cPanel forum and most people are not running the CloudLinux kernel, but I hope some CloudLinux users could benefit from my experiences (or even better, convert your CentOS installation to CloudLinux :D )

crontab for lve wrapping:

lve_suwrapper 50 /usr/local/cpanel/scripts/cpbackup

my current configuration

lvectl set 50 --ncpu=100 --cpu=75 --io=6000

this last line basically says: limit the disk I/O for the cpbackup process (50) to 6000kB/sec.

 

[Ebridge]

UPDATE
I was able to further reduce CPU load and increase overall server responsiveness by using multiple crontab entries to change the I/O throttling depending on the time.

I now have cpbackup starting much earlier than ever before (22:30 in this case), but with low I/O bandwidth so there is hardly any perceivable stress on the CPU and disk.

The different lvectl-entries in the crontab gradually increase the I/O bandwidth for cpbackup as traffic to/from the server decreases (nighttime).
In the morning they gradually decrease the I/O bandwith as server traffic increases. From 8:30 on it stays at a very low 1000kB/sec. This allows the backup to finish at little expense if it didn't complete during the night for some reason, without causing high disk/CPU loads.

30 22 * * * lve_suwrapper 50 /usr/local/cpanel/scripts/cpbackup
29 22 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=40 --io=2500
00 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
30 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3500
00 0 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=5000
00 1 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=7000
00 5 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=5000
00 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
30 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
30 7 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=2500
00 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=2000
30 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=50 --io=1000

Of course these values are completely dependent on your particular server hardware setup, application, traffic patterns etc.

Watching the graphs in Munin should give an indication what values and times to start with

 

[electric]

This is a great idea!

It would be nifty to see this "built in" to cpanel to manage using the WHM if cloudlinux is installed.

 

[techdruid]

I just wanted to chime in and to say thanks to Ebridge for posting this!

I've been using CloudLinux for a year and a half now and I love it. I've been having troubles with my backups slowing down servers and just implemented this fix on one of my servers as a test a few days ago. It seems to be working pretty well and I'm gearing up to implement on all my servers now. Yay for no more sluggish websites for my customers!

Richard

 

[bitronictech]

I just wanted to chime in and to say thanks to Ebridge for posting this!

I've been using CloudLinux for a year and a half now and I love it. I've been having troubles with my backups slowing down servers and just implemented this fix on one of my servers as a test a few days ago. It seems to be working pretty well and I'm gearing up to implement on all my servers now. Yay for no more sluggish websites for my customers!

Richard

I just implemented this on my servers, hopefully it reduces IO Wait considerably during backups. What would be a good backup timeout to set in seconds while running this?

 

[xanubi]

Does anyone use that solution with the new backup system and with the COMPRESSED backups option turned on?

 

[benito]

I will try this today.

 

[sahostking]

I will try this today.

Just make sure you give them different IDs. I tried this with cpbackup and it does slow down the backup process alot if you give the IO a low value. Keep that in mind

 

[Stuart Elliot]

Ive been confused over why my servers where having issues when backing up data , my support team all knew it was io wait, and told me to stop using compressed backups, i held on and kept firm with my self and after a few months of googling i found this post..

it helped loads,

i have a 8 core 16gig ram 2.5gig server and found using
" lvectl set 50 --ncpu=100 --cpu=75 --io=5000 "

worked perfectly for me, even when it comes to the big backup files, it stays solid and doesnt create massive load like before, when all my websites become slow and unusable,

now my backups are limited, i just need to fiqure out how to add it to the crontab now.


Thank you
Stuart Elliot
Buffalo Web Design

 

[splaquet]

I've recently started seeing some real slowdowns on our nightly backups. I can't say that I'm surprised, as I've slowly watched one site (in particular) grow at a rapid pace.

Any chance of these commands having "modern" equivalents? (as the current lines report as being associated with the Legacy Backup system.

 

[cPRex]

Could you just update the line to reference the current backup script at /usr/local/cpanel/bin/backup?

 

[splaquet]

Could you just update the line to reference the current backup script at /usr/local/cpanel/bin/backup?

Actually, I was able to:

30 22 * * * lve_suwrapper 50 /usr/local/cpanel/bin/backup
29 22 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=2500
00 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=3000
30 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=3500
00 0 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=5000
00 1 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=7000
00 5 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=5000
00 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=3000
30 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=3000
30 7 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=2500
00 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=2000
30 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=1000

BUT... I'm starting to question if it's really the backups that causing my recently IOWAIT spikes. Looking into the processes, it seems to be a combo of cpbackup & wp-toolkit.

is there any way to throttle down toolkit?

 

[RevivalTech]

Actually, I was able to:

30 22 * * * lve_suwrapper 50 /usr/local/cpanel/bin/backup
29 22 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=2500
00 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=3000
30 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=3500
00 0 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=5000
00 1 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=7000
00 5 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=5000
00 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=3000
30 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=3000
30 7 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=2500
00 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=2000
30 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --speed=100% --io=1000

BUT... I'm starting to question if it's really the backups that causing my recently IOWAIT spikes. Looking into the processes, it seems to be a combo of cpbackup & wp-toolkit.

is there any way to throttle down toolkit?

I've got some continued IO lve limit faults and I'm figuring it's cp backup but now you mention it, it could be toolkit too I suppose. Landed here looking for how to possibly whitelist the backups from io limits somehow, but also wonder if that'd be defeating the purpose at the same time... So another solution perhaps would be best. But would like to have lve limits in place for obvious reasons, but not trigger false positive by backups for obvious reason haha.

 

[cPRex]

Unless you have configured the customizations mentioned in this thread, the root backup wouldn't be under any LVE restrictions. A user-generated backup from inside the cPanel interface would.

 

[RevivalTech]

Unless you have configured the customizations mentioned in this thread, the root backup wouldn't be under any LVE restrictions. A user-generated backup from inside the cPanel interface would.

Hmmmm perhaps you help me understand. This is my confusion. This is the whm global high level backup system, but seems to be pushing the IO through each user and triggering limit. This is most I understand currently, as I don't get why the traffic would be under user if this is a highlevel administration backup system. So that's why I wonder if it's a "dumb" system on LVE limits, and it's grabbing all data moved at all including root, or the cpanel whm high level backup works as user and I don't understand that. That make sense?

 

[cPRex]

Sure, that makes sense. This is something you'd likely want to report to CloudLinux so they could exclude that process by default, as we don't have much control over that on our end.