Unused memory and high swap / io wait

[LeeCardiff]

Hiya

A few weeks ago my VPS crashed after running fine for over a year. Since the reboot i have had some serious speed issues as detailed below.



Before the crash commit memory was reasonable and swap usage was very low. Cache memory looks like it was working correctly and all the sites were very fast. You can see in "Week 11" that the "unsued memory" is now at ~250MB and swap usage has gone through the roof.

Most of the swap seems to be used by mysql, and sites that use the database mostly Zen Cart's and Joomla install really slow down after a few days.

CPU cycles are not the issue, its the iowait due to mysql using the swap file. Any advice on how to:
1. keey mysql out of the swap ?
2. Get the cache working again ?

I stress that i made no configuration changes until week 12

Thanks, Lee.

 

[Infopro]

You might look here for clues:
WHM > Server Status > Daily Process Log

At top of that page you can click the dates and go back thru these logs a bit. If you can locate the account spiking resources (they'd be shown highlighted in red, IIRC) you can take a closer look at the account to be sure all scripts on it, are up to date and secure. My guess is, something is not up to date on at least one account and is being hit by bad traffic looking for a way in. Or worse.

 

[LeeCardiff]

The load / visitors to the server is steady and not increased by a large amount over the time period. LFD & CSF are running on the server and all the logs are fine. All the traffic is legitimate to 1 or 2 main sites that heavily use sql and php (checked for security with a local backup of my original files).

The issue i have is the fact that mysql is constantly swapping to disk (up to 250M swap now) while there is free ram, the performance drop off is extreme. At this point i am seriously looking at an OS rebuild, and restoring the cpanel backups or even changing hosts for less downtime.

I think it will be something really simple but for once Google has let me down

 

[Infopro]

Not sure an OS reload is warranted here to be honest. Let's back up a sec:

I stress that i made no configuration changes until week 12

You don't mention the changes you made here. Or was this worded incorrectly?

 

[LeeCardiff]

Ahh in week 12 (after all the swapping started) i changed "swappiness = 60" to "swappiness = 1" and altered my.cnf to the following.

<code>
[mysqld]
skip-innodb
set-variable = max_connections=30
key_buffer_size = 128M
myisam_sort_buffer_size = 64M
join_buffer_size = 256k
read_buffer_size = 256K
sort_buffer_size = 256K
tmp_table_size = 64M
max_heap_table_size = 64M
table_cache = 1024
thread_cache_size = 16
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1
log-slow-queries
safe-show-database
</code>

key_buffer_size = 128M was way to high and has now been set to 32M.

top output = (mysqld was restarted an hour ago)

Tasks: 109 total, 2 running, 106 sleeping, 0 stopped, 1 zombie
Cpu(s): 2.4%us, 1.0%sy, 0.0%ni, 94.3%id, 2.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 524472k total, 264360k used, 260112k free, 10340k buffers
Swap: 1048568k total, 56200k used, 992368k free, 48036k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28027 bradford 16 0 0 0 0 Z 6.0 0.0 0:00.18 php <defunct>
23828 mysql 15 0 117m 47m 3668 S 0.3 9.3 0:08.06 mysqld
26596 nobody 15 0 13292 5116 2400 S 0.3 1.0 0:00.06 httpd

free -m =
total used free shared buffers cached
Mem: 512 263 248 0 8 40
-/+ buffers/cache: 213 298
Swap: 1023 54 969

 

[Infopro]

Someone else may have a more in depth comment for you on your settings, I don't.

I'm a firm believer in looking at whats right in front of me first, if my server is running just fine and then all of a sudden starts acting up like this, swapping, then something has of course changed. If not something you did wrong, then its on an account.

All the traffic is legitimate to 1 or 2 main sites that heavily use sql and php (checked for security with a local backup of my original files).

It's either 1, the other or both. Still, those main sites should be looked at closer for something out of date. And old plugin, a chat script, etc.

The load / visitors to the server is steady and not increased by a large amount over the time period.

If I understand you correctly here, you looked back at that daily process log to Week 12 and found no accounts shown to you in red?

 

[LeeCardiff]

Nope, its just mysql swapping to disk while i have ~250M ram listed as free. You can see in the first pic the nice yellow part that should be purple / cache. Process log is shown below.

Disk access has also increased on the main partition due to the cache being a tenth the size it was before the crash.