The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Unused memory and high swap / io wait

Discussion in 'Workarounds and Optimization' started by LeeCardiff, Apr 2, 2012.

  1. LeeCardiff

    LeeCardiff Registered

    Joined:
    Apr 2, 2012
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Hiya :)

    A few weeks ago my VPS crashed after running fine for over a year. Since the reboot i have had some serious speed issues as detailed below.

    memory_usage.jpg

    Before the crash commit memory was reasonable and swap usage was very low. Cache memory looks like it was working correctly and all the sites were very fast. You can see in "Week 11" that the "unsued memory" is now at ~250MB and swap usage has gone through the roof.

    Most of the swap seems to be used by mysql, and sites that use the database mostly Zen Cart's and Joomla install really slow down after a few days.

    CPU cycles are not the issue, its the iowait due to mysql using the swap file. Any advice on how to:
    1. keey mysql out of the swap ?
    2. Get the cache working again ?

    I stress that i made no configuration changes until week 12 :(

    Thanks, Lee.
     
  2. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,478
    Likes Received:
    203
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    You might look here for clues:
    WHM > Server Status > Daily Process Log

    At top of that page you can click the dates and go back thru these logs a bit. If you can locate the account spiking resources (they'd be shown highlighted in red, IIRC) you can take a closer look at the account to be sure all scripts on it, are up to date and secure. My guess is, something is not up to date on at least one account and is being hit by bad traffic looking for a way in. Or worse.
     
  3. LeeCardiff

    LeeCardiff Registered

    Joined:
    Apr 2, 2012
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    The load / visitors to the server is steady and not increased by a large amount over the time period. LFD & CSF are running on the server and all the logs are fine. All the traffic is legitimate to 1 or 2 main sites that heavily use sql and php (checked for security with a local backup of my original files).

    The issue i have is the fact that mysql is constantly swapping to disk (up to 250M swap now) while there is free ram, the performance drop off is extreme. At this point i am seriously looking at an OS rebuild, and restoring the cpanel backups or even changing hosts for less downtime.

    I think it will be something really simple but for once Google has let me down :)
     
  4. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,478
    Likes Received:
    203
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Not sure an OS reload is warranted here to be honest. Let's back up a sec:

    You don't mention the changes you made here. Or was this worded incorrectly?
     
  5. LeeCardiff

    LeeCardiff Registered

    Joined:
    Apr 2, 2012
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Ahh in week 12 (after all the swapping started) i changed "swappiness = 60" to "swappiness = 1" and altered my.cnf to the following.

    <code>
    [mysqld]
    skip-innodb
    set-variable = max_connections=30
    key_buffer_size = 128M
    myisam_sort_buffer_size = 64M
    join_buffer_size = 256k
    read_buffer_size = 256K
    sort_buffer_size = 256K
    tmp_table_size = 64M
    max_heap_table_size = 64M
    table_cache = 1024
    thread_cache_size = 16
    query_cache_limit = 1M
    query_cache_size = 32M
    query_cache_type = 1
    log-slow-queries
    safe-show-database
    </code>

    key_buffer_size = 128M was way to high and has now been set to 32M.

    top output = (mysqld was restarted an hour ago)

    Tasks: 109 total, 2 running, 106 sleeping, 0 stopped, 1 zombie
    Cpu(s): 2.4%us, 1.0%sy, 0.0%ni, 94.3%id, 2.3%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 524472k total, 264360k used, 260112k free, 10340k buffers
    Swap: 1048568k total, 56200k used, 992368k free, 48036k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    28027 bradford 16 0 0 0 0 Z 6.0 0.0 0:00.18 php <defunct>
    23828 mysql 15 0 117m 47m 3668 S 0.3 9.3 0:08.06 mysqld
    26596 nobody 15 0 13292 5116 2400 S 0.3 1.0 0:00.06 httpd

    free -m =
    total used free shared buffers cached
    Mem: 512 263 248 0 8 40
    -/+ buffers/cache: 213 298
    Swap: 1023 54 969
     
  6. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,478
    Likes Received:
    203
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Someone else may have a more in depth comment for you on your settings, I don't.

    I'm a firm believer in looking at whats right in front of me first, if my server is running just fine and then all of a sudden starts acting up like this, swapping, then something has of course changed. If not something you did wrong, then its on an account.
    It's either 1, the other or both. Still, those main sites should be looked at closer for something out of date. And old plugin, a chat script, etc.

    If I understand you correctly here, you looked back at that daily process log to Week 12 and found no accounts shown to you in red?
     
  7. LeeCardiff

    LeeCardiff Registered

    Joined:
    Apr 2, 2012
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Nope, its just mysql swapping to disk while i have ~250M ram listed as free. You can see in the first pic the nice yellow part that should be purple / cache. Process log is shown below.
    processes.jpg
    Disk access has also increased on the main partition due to the cache being a tenth the size it was before the crash.
     
    #7 LeeCardiff, Apr 2, 2012
    Last edited: Apr 2, 2012
Loading...

Share This Page