reducing load and disk I/O for cpbackup with CloudLinux

Ebridge

Member
May 3, 2012
16
1
53
cPanel Access Level
Root Administrator
I've been struggling with cpbackup putting too much stress on the server (disk latency peaks, high CPU load and http load time peaks of multiple seconds, sometimes more than 10 seconds for a very short period of time) during backups. Tried all kinds of optimizations but nothing really worked. Pigz (cPanel 11.38) reduced the time for the backups to complete a bit, but the peaks stayed just as high.

Judging from the Munin-graphs, the disk utilization (peaking regularly at 100% during backups, especially for extremely large accounts) seemed to be the evildoer.

As I'm running the CloudLinux kernel (which I really like anyway) I got the idea to use its command line LVE-wrapper (Lightweight Virtual Environment) to wrap the cpbackup root process into an LVE, and then try to use the I/O throttling option of CloudLinux to moderate the disk I/O to more friendly values.

It took a bit of experimenting, but it works really well! :)

Average CPU load during backups dropped from 2 to less than 1. Even better is the reduction of the peak CPU load: from 6 down to 1 (practically no CPU peaks during backups)
cloudlinux load averages.png

An important real life test of course is the http loadtime. Loadtime during backups was reduced from 0.5 seconds average and more than 3 seconds peak to 0.15 seconds average and 0.5 seconds peak.
cloudlinux http load time.png
First days in the graphs show the 'old' situation, then I started experimenting with the LVE-wrapper parameters (resulting in varying values), and last night seems to be the winning configuration. The peak values were read from the graphs, but actual peaks can be much higher (but too short duration for Munin to register)

Overall, the server felt almost as responsive during backups as during daytime, which is a big win.
I'm sure some greater gains can be achieved by finetuning the LVE wrapper parameters some more but for now I'm happy with the results.

For reference some info on this server:
4 cores, 4GB memory, cPanel 11.38, CloudLinux 5 hybrid. Hosting a lot of large (4-16GB) accounts. Daily full backups over FTP to backup server.

I do understand this is the cPanel forum and most people are not running the CloudLinux kernel, but I hope some CloudLinux users could benefit from my experiences (or even better, convert your CentOS installation to CloudLinux :D )

crontab for lve wrapping:
Code:
lve_suwrapper 50 /usr/local/cpanel/scripts/cpbackup
my current configuration
Code:
lvectl set 50 --ncpu=100 --cpu=75 --io=6000
this last line basically says: limit the disk I/O for the cpbackup process (50) to 6000kB/sec.
 
Last edited:
  • Like
Reactions: eva2000

Ebridge

Member
May 3, 2012
16
1
53
cPanel Access Level
Root Administrator
UPDATE
I was able to further reduce CPU load and increase overall server responsiveness by using multiple crontab entries to change the I/O throttling depending on the time.

I now have cpbackup starting much earlier than ever before (22:30 in this case), but with low I/O bandwidth so there is hardly any perceivable stress on the CPU and disk.

The different lvectl-entries in the crontab gradually increase the I/O bandwidth for cpbackup as traffic to/from the server decreases (nighttime).
In the morning they gradually decrease the I/O bandwith as server traffic increases. From 8:30 on it stays at a very low 1000kB/sec. This allows the backup to finish at little expense if it didn't complete during the night for some reason, without causing high disk/CPU loads.

Code:
30 22 * * * lve_suwrapper 50 /usr/local/cpanel/scripts/cpbackup
29 22 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=40 --io=2500
00 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
30 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3500
00 0 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=5000
00 1 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=7000
00 5 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=5000
00 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
30 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
30 7 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=2500
00 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=2000
30 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=50 --io=1000
Of course these values are completely dependent on your particular server hardware setup, application, traffic patterns etc.

Watching the graphs in Munin should give an indication what values and times to start with
 

electric

Well-Known Member
Nov 5, 2001
742
5
318
This is a great idea!

It would be nifty to see this "built in" to cpanel to manage using the WHM if cloudlinux is installed.
 

techdruid

Member
Jan 16, 2012
9
0
51
cPanel Access Level
Root Administrator
I just wanted to chime in and to say thanks to Ebridge for posting this!

I've been using CloudLinux for a year and a half now and I love it. I've been having troubles with my backups slowing down servers and just implemented this fix on one of my servers as a test a few days ago. It seems to be working pretty well and I'm gearing up to implement on all my servers now. Yay for no more sluggish websites for my customers!

Richard
 

bitronictech

Registered
Jan 22, 2014
1
0
1
cPanel Access Level
Root Administrator
I just wanted to chime in and to say thanks to Ebridge for posting this!

I've been using CloudLinux for a year and a half now and I love it. I've been having troubles with my backups slowing down servers and just implemented this fix on one of my servers as a test a few days ago. It seems to be working pretty well and I'm gearing up to implement on all my servers now. Yay for no more sluggish websites for my customers!

Richard
I just implemented this on my servers, hopefully it reduces IO Wait considerably during backups. What would be a good backup timeout to set in seconds while running this?
 

xanubi

Well-Known Member
Jun 28, 2006
86
2
158
Does anyone use that solution with the new backup system and with the COMPRESSED backups option turned on?
 

Stuart Elliot

Member
Jan 3, 2016
11
2
3
United Kingdom
cPanel Access Level
Root Administrator
Ive been confused over why my servers where having issues when backing up data , my support team all knew it was io wait, and told me to stop using compressed backups, i held on and kept firm with my self and after a few months of googling i found this post..

it helped loads,

i have a 8 core 16gig ram 2.5gig server and found using
" lvectl set 50 --ncpu=100 --cpu=75 --io=5000 "

worked perfectly for me, even when it comes to the big backup files, it stays solid and doesnt create massive load like before, when all my websites become slow and unusable,

now my backups are limited, i just need to fiqure out how to add it to the crontab now.


Thank you
Stuart Elliot
Buffalo Web Design