Running out of swap space

[BigLebowski]

hi there

I have a busy Centos machine running Cpanel with 3GB RAM and 2GB swap.

Twice a day the swap space gradually fills up. When it maxes, kswapd goes mad, the load shoots up and the box dies.

How can I:

a) identify the processes using up all the swap space and
b) kill them off to free more space

Ta
Dude

[thewebhosting]

1. You can see the processes running on your machine using top -c command.
2. You can use kill -9 process_id to kill particular process.

[BigLebowski]

Thanks, the problem is that there's nothing immediately obvious to kill when the swap space is all maxed.

I think I will use in WHM:

Server Status -> CPU/Memory/MySQL Usage

This has already identified one culprit that I've had trouble with before.

Thanks
Dude

[stdout]

Dear "The Dude",
1. Great Movie.
2. Use top -c; when in top, hold shift and hit M.
This will sort your top processes by Memory.

3. Check your *raw dcpumon logs under /var/log/dcpumon/.
The "dcpumonview" will just calculate average figures, instead view the raw logs.

ie. Do a "ls -lh on /var/log/dcpumon/toplog.*" and determine which file is the largest.
The larger files mean more processes (obviously).
Use this command to grep for any processes not running by system users.

This should help determine whether it is a user or system user consuming all your memory.

Example:

cat /var/log/dcpumon/toplog.1228137601  | egrep -v -E 'root|mailnull|postgres|dbus|mysql|tomcat|named|courier|mailman|xfs'

[stdout]

A piece of FYI: The "/usr/local/cpanel/bin/dcpumonview" tool is equivalent to the WHM's "CPU/Memory/mySQL Usage" tool. These are not real-time. Instead, try the command(s) I gave you for more accurate investigating..

I think I will use in WHM:

Server Status -> CPU/Memory/MySQL Usage

[BigLebowski]

excellent, thanks, very good info

Dude

[BigLebowski]

Something peculiar is happening.

The swapspace was at 600MB again just now. I shut down MySQL, httpd, exim, courier-imap etc. Down to about 90 processes but swap space still at 55MB. I can't figure why? Why doesn't it clear completely?

The load average is now very low and nothing is obviously consuming vast amounts of memory from top (below). However something is still gobbling it up!

top - 16:07:56 up 3:56, 1 user, load average: 0.63, 2.22, 2.68
Tasks: 97 total, 1 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7% us, 0.0% sy, 0.0% ni, 98.3% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 3114948k total, 1420388k used, 1694560k free, 99672k buffers
Swap: 2048248k total, 55732k used, 1992516k free, 1162960k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3473 named 18 0 85184 16m 1928 S 0.0 0.6 3:02.77 /usr/sbin/named -u named
14624 root 34 19 14276 5864 1876 S 0.0 0.2 0:33.13 cpanellogd - sleeping for logs
14568 root 16 0 9000 5404 1744 S 0.0 0.2 0:57.89 tailwatchd
14811 root 15 0 16672 5216 1036 S 0.0 0.2 0:04.95 cpsrvd - waiting for connections
14686 mailman 16 0 12684 4900 2260 S 0.0 0.2 0:51.18 /usr/local/bin/python2.

Dude

[BigLebowski]

I am posting a follow up for forum interest.

I located a few "sticky" httpd processes lurking and using up around 8 to 11% memory. I kept finding the same two users. I moved them to another server running SuPHP. Within a few hours one of them was maxing out one of the CPUs with 10 perl cgi scripts. It was very easy to see who it was with SuPHP enabled and clobber them.

Dude