The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

High IOwait white

Discussion in 'CloudLinux' started by NixTree, Nov 13, 2012.

  1. NixTree

    NixTree Well-Known Member

    Joined:
    Aug 19, 2010
    Messages:
    386
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Gods Own Country
    cPanel Access Level:
    Root Administrator
    Hello,

    I have noticed highIO wait on the server intermittently. I suspect that the IO wait increased while the data in meory is writeback to the HDD. Please find the meminfo file details below while the IO wait increases.

    MemTotal: 16287364 kB
    MemFree: 92904 kB
    Buffers: 1211192 kB
    Cached: 7257116 kB
    SwapCached: 14400 kB
    Active: 10222852 kB
    Inactive: 4998312 kB
    Active(anon): 6006260 kB
    Inactive(anon): 763112 kB
    Active(file): 4216592 kB
    Inactive(file): 4235200 kB
    Unevictable: 16824 kB
    Mlocked: 16824 kB
    SwapTotal: 4194296 kB
    SwapFree: 4065288 kB
    Dirty: 23984 kB
    Writeback: 864 kB
    AnonPages: 6758816 kB
    Mapped: 58960 kB
    Shmem: 12936 kB
    Slab: 772904 kB
    SReclaimable: 658088 kB
    SUnreclaim: 114816 kB
    KernelStack: 5440 kB
    PageTables: 56080 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    WritebackTmp: 0 kB
    CommitLimit: 12337976 kB
    Committed_AS: 6621536 kB
    VmallocTotal: 34359738367 kB
    VmallocUsed: 326432 kB
    VmallocChunk: 34359368204 kB
    HardwareCorrupted: 0 kB
    AnonHugePages: 0 kB
    HugePages_Total: 0
    HugePages_Free: 0
    HugePages_Rsvd: 0

    Cloud Linux details:
    # cat /etc/redhat-release
    CloudLinux Server release 6.3
    Linux 2.6.32-379.9.1.lve1.1.9.7.2.el6.x86_64 #1 SMP

    I noticed that each time the IO goes high, kswapd0 is running and it will come back to nornal after a couple of service restarts ( mysql, http ).

    Please let me know your suggestions.

    Thank you.
     
  2. tiff2342

    tiff2342 Well-Known Member

    Joined:
    Apr 20, 2012
    Messages:
    140
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    we have had this issue as well ALOT. so you are not a lone. we revently updated to the latest kernel and still have random issues with it

    what is your
    sysctl.conf (vm. settings)
    php.conf
    and LVE settings?

    we found tweaking all of the values above has helped some but we still see spikes way up past 10 even into 15's
     
    #2 tiff2342, Nov 14, 2012
    Last edited: Nov 14, 2012
  3. JStevensons

    JStevensons Registered

    Joined:
    Nov 14, 2012
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Same problems here.... just tonight I had probably over 10 something spikes into 10 and the only way to stop them was to keep restarting services like mentioned above.

    I am too also running the latest kernel after the previous ones gave me nothing but trouble as well. /sighs

    I have also tried so many configurations with no luck in tuning ;/
     
  4. tiff2342

    tiff2342 Well-Known Member

    Joined:
    Apr 20, 2012
    Messages:
    140
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    JS. What configurations have you done? We have been through the book trying them and have had issues no idea why the spikes randomly happen and just about crash the server. its almost like need a cron job to restart if high i/o load.
     
  5. NixTree

    NixTree Well-Known Member

    Joined:
    Aug 19, 2010
    Messages:
    386
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Gods Own Country
    cPanel Access Level:
    Root Administrator
    Yes, I also would like to know what all values tweaked. I tried to tweak some pdflush and kswap kernel parametres which didn't help! And interestingly we got cagefs installed on the server automatically. Initially I thought the abnormal behaviuors is due to that, but still the same issues after removing it.. ( and we get kernel panic when the server goes down )

    Igor, please comment your suggestions!

    Thank you.
     
  6. iseletsk

    iseletsk Well-Known Member

    Joined:
    Mar 3, 2010
    Messages:
    163
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Princeton, New Jersey, United States
    There are two known issues with CL6 kernel related to iowait. One is CPU contention issue which was fixed here:
    Beta: new CL6 kernel available
    The problem is that latest CL6 beta kernel is crashing (so far crashed on two customer servers), so it is not recommended for widespread use. I hope to have another kernel by Monday that will fix the crashes.

    The second issue is related to inefficient disk caching page reclamation that can cause overall bad IO performance once there is lack of RAM. This one should be fixed within 2-3 weeks (it is fairly complex problem/lots of work is being done on it).
     
  7. NixTree

    NixTree Well-Known Member

    Joined:
    Aug 19, 2010
    Messages:
    386
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Gods Own Country
    cPanel Access Level:
    Root Administrator
    Thank you Igor; I can confirm that our server is suffering from the second issue!
     
  8. tiff2342

    tiff2342 Well-Known Member

    Joined:
    Apr 20, 2012
    Messages:
    140
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Our's too. We have been messing with trying to clear out cache to ensure we always have RAM free using sysctl and kernel tuning.

    Latest spike:
    One idea is to use drop_caches to keep it cleaned. Maybe a restart service cron job every now and again.

    We tried messing with vm. like:
    vm.dirty_ratio = 20
    vm.dirty_background_ratio = 1

    Considering maybe best move is to run
    sync; echo 3 > /proc/sys/vm/drop_caches
    every so often to keep memory free to avoid the i/o?
     
    #8 tiff2342, Nov 15, 2012
    Last edited: Nov 15, 2012
  9. iseletsk

    iseletsk Well-Known Member

    Joined:
    Mar 3, 2010
    Messages:
    163
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Princeton, New Jersey, United States
    drop cache will raise IO even more. Hopefully the issue will be resolved in 2 weeks, now that we know what is the real cause.
     
  10. tiff2342

    tiff2342 Well-Known Member

    Joined:
    Apr 20, 2012
    Messages:
    140
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    What is the real cause?

    --

    Would compiling Apache to work as worker/event be best? Prefork is causing too many apache process to spawn taking out RAM faster.


    Current settings:
     
    #10 tiff2342, Nov 15, 2012
    Last edited: Nov 15, 2012
  11. NixTree

    NixTree Well-Known Member

    Joined:
    Aug 19, 2010
    Messages:
    386
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Gods Own Country
    cPanel Access Level:
    Root Administrator
    Yes, do not drop cache frequently..its a bad idea and can raise IO! What I have done is to restart the high memory usage services every n hours based on how horribly it comsume memory and how a restart will affect the clients! I don't think recompiling apache will help as the real issue is out of Apache, but with the memory mamanement of kernel!
     
    #11 NixTree, Nov 16, 2012
    Last edited: Nov 16, 2012
  12. tiff2342

    tiff2342 Well-Known Member

    Joined:
    Apr 20, 2012
    Messages:
    140
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    thinking of going to BETTERLINUX now. This problem is getting old fast.

    this stuff help any?
    I think we have ran like 10+ load average stable just about all last night and this morning. it's crazy.

    surely there is a way to free linux ram?


    how bad this issue is now

    This this would help?

     
    #12 tiff2342, Nov 16, 2012
    Last edited: Nov 16, 2012
  13. tiff2342

    tiff2342 Well-Known Member

    Joined:
    Apr 20, 2012
    Messages:
    140
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    For all those having issues try the following. did some research and this seemed to have helped:

    remember to type sysctl -p after editing values in /etc/sysctl.conf
     
    #13 tiff2342, Nov 29, 2012
    Last edited: Nov 29, 2012
  14. ANKUR KUMAR

    ANKUR KUMAR Active Member

    Joined:
    Oct 28, 2012
    Messages:
    26
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    India
    cPanel Access Level:
    Root Administrator
    After Cloud Linux Support Installed Cloud Linux Kernel on my server, i am facing the same issue .

    Though Cloud Linux is working but Server Wen slow and Seems High I/O wait . My data Center says , there isnt any trouble with hardware .

    The load of the server is too much so that i can take backup of accounts and move to separate server as we are facing downtime since last 4 days .

    The My SQl data base is not at all woking after this , and crashed each time the load is high .

    An urgent Help is required .

    tiff2342 , i can not opt for solution you have as i do not wish to opt untill cloud linux representative come and let me know if this is the solution .

    Please cooperate ...:(:(
     
  15. NixTree

    NixTree Well-Known Member

    Joined:
    Aug 19, 2010
    Messages:
    386
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Gods Own Country
    cPanel Access Level:
    Root Administrator
    you can try the solution suggested by tiff2342; I have did almost the same ( + other parametres ). what all we can do is to keep as much of the RAM free till the issue is fixed.
     
  16. iseletsk

    iseletsk Well-Known Member

    Joined:
    Mar 3, 2010
    Messages:
    163
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Princeton, New Jersey, United States
  17. iseletsk

    iseletsk Well-Known Member

    Joined:
    Mar 3, 2010
    Messages:
    163
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Princeton, New Jersey, United States
  18. manuki

    manuki Member

    Joined:
    Dec 18, 2007
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
  19. iseletsk

    iseletsk Well-Known Member

    Joined:
    Mar 3, 2010
    Messages:
    163
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Princeton, New Jersey, United States
    The IO Wait issue is fixed & working very well.
    The load averages reporting still needs some work -- but so far -- it is just a reporting. The performance of the servers with new kernel improved considerably.
     
  20. Sannin

    Sannin Active Member

    Joined:
    May 19, 2011
    Messages:
    34
    Likes Received:
    0
    Trophy Points:
    6
    Hello!

    The new kernel indeed fixed the IO issue but with a minor setback. Sysctl is broken. This kernel does not respect my vm.swapiness value and there are time i have 1Gb swap file with 10 Gbs free ram.

    When do you plan to fix this issue?
     
Loading...
Similar Threads - High IOwait white
  1. 4u123
    Replies:
    1
    Views:
    1,994

Share This Page