The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

reducing load and disk I/O for cpbackup with CloudLinux

Discussion in 'CloudLinux' started by Ebridge, May 12, 2013.

  1. Ebridge

    Ebridge Member

    Joined:
    May 3, 2012
    Messages:
    16
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    I've been struggling with cpbackup putting too much stress on the server (disk latency peaks, high CPU load and http load time peaks of multiple seconds, sometimes more than 10 seconds for a very short period of time) during backups. Tried all kinds of optimizations but nothing really worked. Pigz (cPanel 11.38) reduced the time for the backups to complete a bit, but the peaks stayed just as high.

    Judging from the Munin-graphs, the disk utilization (peaking regularly at 100% during backups, especially for extremely large accounts) seemed to be the evildoer.

    As I'm running the CloudLinux kernel (which I really like anyway) I got the idea to use its command line LVE-wrapper (Lightweight Virtual Environment) to wrap the cpbackup root process into an LVE, and then try to use the I/O throttling option of CloudLinux to moderate the disk I/O to more friendly values.

    It took a bit of experimenting, but it works really well! :)

    Average CPU load during backups dropped from 2 to less than 1. Even better is the reduction of the peak CPU load: from 6 down to 1 (practically no CPU peaks during backups)
    cloudlinux load averages.png

    An important real life test of course is the http loadtime. Loadtime during backups was reduced from 0.5 seconds average and more than 3 seconds peak to 0.15 seconds average and 0.5 seconds peak.
    cloudlinux http load time.png
    First days in the graphs show the 'old' situation, then I started experimenting with the LVE-wrapper parameters (resulting in varying values), and last night seems to be the winning configuration. The peak values were read from the graphs, but actual peaks can be much higher (but too short duration for Munin to register)

    Overall, the server felt almost as responsive during backups as during daytime, which is a big win.
    I'm sure some greater gains can be achieved by finetuning the LVE wrapper parameters some more but for now I'm happy with the results.

    For reference some info on this server:
    4 cores, 4GB memory, cPanel 11.38, CloudLinux 5 hybrid. Hosting a lot of large (4-16GB) accounts. Daily full backups over FTP to backup server.

    I do understand this is the cPanel forum and most people are not running the CloudLinux kernel, but I hope some CloudLinux users could benefit from my experiences (or even better, convert your CentOS installation to CloudLinux :D )

    crontab for lve wrapping:
    Code:
    lve_suwrapper 50 /usr/local/cpanel/scripts/cpbackup
    my current configuration
    Code:
    lvectl set 50 --ncpu=100 --cpu=75 --io=6000
    this last line basically says: limit the disk I/O for the cpbackup process (50) to 6000kB/sec.
     
    #1 Ebridge, May 12, 2013
    Last edited: May 12, 2013
    eva2000 likes this.
  2. Ebridge

    Ebridge Member

    Joined:
    May 3, 2012
    Messages:
    16
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    UPDATE
    I was able to further reduce CPU load and increase overall server responsiveness by using multiple crontab entries to change the I/O throttling depending on the time.

    I now have cpbackup starting much earlier than ever before (22:30 in this case), but with low I/O bandwidth so there is hardly any perceivable stress on the CPU and disk.

    The different lvectl-entries in the crontab gradually increase the I/O bandwidth for cpbackup as traffic to/from the server decreases (nighttime).
    In the morning they gradually decrease the I/O bandwith as server traffic increases. From 8:30 on it stays at a very low 1000kB/sec. This allows the backup to finish at little expense if it didn't complete during the night for some reason, without causing high disk/CPU loads.

    Code:
    30 22 * * * lve_suwrapper 50 /usr/local/cpanel/scripts/cpbackup
    29 22 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=40 --io=2500
    00 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
    30 23 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3500
    00 0 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=5000
    00 1 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=7000
    00 5 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=5000
    00 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
    30 6 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=3000
    30 7 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=2500
    00 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=75 --io=2000
    30 8 * * * /usr/sbin/lvectl set 50 --ncpu=100 --cpu=50 --io=1000
    
    Of course these values are completely dependent on your particular server hardware setup, application, traffic patterns etc.

    Watching the graphs in Munin should give an indication what values and times to start with
     
  3. electric

    electric Well-Known Member

    Joined:
    Nov 5, 2001
    Messages:
    697
    Likes Received:
    1
    Trophy Points:
    18
    This is a great idea!

    It would be nifty to see this "built in" to cpanel to manage using the WHM if cloudlinux is installed.
     
  4. techdruid

    techdruid Member
    PartnerNOC

    Joined:
    Jan 16, 2012
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I just wanted to chime in and to say thanks to Ebridge for posting this!

    I've been using CloudLinux for a year and a half now and I love it. I've been having troubles with my backups slowing down servers and just implemented this fix on one of my servers as a test a few days ago. It seems to be working pretty well and I'm gearing up to implement on all my servers now. Yay for no more sluggish websites for my customers!

    Richard
     
  5. bitronictech

    bitronictech Registered

    Joined:
    Jan 22, 2014
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I just implemented this on my servers, hopefully it reduces IO Wait considerably during backups. What would be a good backup timeout to set in seconds while running this?
     
  6. xanubi

    xanubi Well-Known Member

    Joined:
    Jun 28, 2006
    Messages:
    86
    Likes Received:
    1
    Trophy Points:
    8
    Does anyone use that solution with the new backup system and with the COMPRESSED backups option turned on?
     
  7. benito

    benito Well-Known Member

    Joined:
    Jan 8, 2004
    Messages:
    296
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Mar del Plata - Argentina
    I will try this today.
     
  8. sahostking

    sahostking Well-Known Member

    Joined:
    May 15, 2012
    Messages:
    299
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Cape Town, South Africa
    cPanel Access Level:
    Root Administrator
    Just make sure you give them different IDs. I tried this with cpbackup and it does slow down the backup process alot if you give the IO a low value. Keep that in mind :)
     
  9. Stuart Elliot

    Stuart Elliot Member

    Joined:
    Jan 3, 2016
    Messages:
    6
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United Kingdom
    cPanel Access Level:
    Root Administrator
    Ive been confused over why my servers where having issues when backing up data , my support team all knew it was io wait, and told me to stop using compressed backups, i held on and kept firm with my self and after a few months of googling i found this post..

    it helped loads,

    i have a 8 core 16gig ram 2.5gig server and found using
    " lvectl set 50 --ncpu=100 --cpu=75 --io=5000 "

    worked perfectly for me, even when it comes to the big backup files, it stays solid and doesnt create massive load like before, when all my websites become slow and unusable,

    now my backups are limited, i just need to fiqure out how to add it to the crontab now.


    Thank you
    Stuart Elliot
    Buffalo Web Design
     
Loading...
Similar Threads - reducing load disk
  1. texas90
    Replies:
    2
    Views:
    257
  2. Naithen_tampa
    Replies:
    6
    Views:
    667

Share This Page