Server crash with mem full, please help

I'm experiencing daily server crash.
I've noticed that Swap keep increasing during the day till the full crash.

Usual memory load during the day is 75%, 50-60 processes.
When it hangs is not possible to connect in SSH or http.

SWAP is never released. Increase only. (is this normal ?)

Running Fedora 2.4.26
WHM 9.9.9 cPanel 9.9.9-S15
Fedora i686 - WHM X v3.1.0

This is are sample top headers during a server hang:

Code:

 21:51:52  up 20:38,  1 user,  load average: 5.17, 32.45, 61.36
218 processes: 214 sleeping, 2 running, 1 zombie, 1 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total   90.4%    0.0%    9.5%   0.0%     0.0%    0.0%    0.0%
Mem:   188188k av,  185408k used,    2780k free,       0k shrd,    2536k buff
        36116k active,             129192k inactive
Swap:  262136k av,  262000k used,     136k free                   10504k cached

Code:

 22:16:04  up 21:02,  1 user,  load average: 0.97, 2.25, 14.68
206 processes: 202 sleeping, 1 running, 1 zombie, 2 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    1.1%    0.0%   34.6%   0.0%     0.0%    0.0%   64.2%
Mem:   188188k av,  186036k used,    2152k free,       0k shrd,    1188k buff
        12668k active,             154512k inactive
Swap:  262136k av,  262136k used,       0k free                    6436k cached

while this is a sample of normal operation load:

Code:

 00:44:07  up 34 min,  1 user,  load average: 2.37, 1.53, 1.13
68 processes: 66 sleeping, 1 running, 1 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.4%    0.0%    3.3%   0.0%     0.0%    0.0%   96.1%
Mem:   188188k av,  137444k used,   50744k free,       0k shrd,   26760k buff
        59800k active,              66560k inactive
Swap:  262136k av,       0k used,  262136k free                   59660k cached

Any Idea?

Simone

 

Yes, I know upgrading ram may solve, or maybe not.
During all the day my typical top is like this:

Code:

 08:37:35  up  8:28,  1 user,  load average: 0.42, 0.55, 0.45
57 processes: 56 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.9%   0.0%     0.0%    0.0%   99.0%
Mem:   188188k av,  162344k used,   25844k free,       0k shrd,   37744k buff
        65888k active,              84952k inactive
Swap:  262136k av,    1604k used,  260532k free                   72444k cached

the +200 processes are a consequence, not the cause.

I'm currently montiroring with mrtg my swap and mem, and I know swap is usually below the 8% or ram usage, except when everything crash.

 

What is not cleat to me is why allocated swap does not decrease when physical ram is available.

Swap keep increasing costantly and is never released till aprox 15% of ram, then everything hang.

That is really something I cannot understand (I'm not a *nix expert)

By the way after another crash debug session, I'm suspecting apache 1.3.3 could be the cause of swap increase due to some spidering activities.

 

I've added more ram but problem still persist.

Now Swap do not fill all space available but it increase till some service fail.
Sometimes Mysql sometimes Apache.

Don't know if is the main cause, but yesterday while running top on my server happened again, and I've been able to see that someone was using an offline browser to download all my site, and there were tens of cgi loaded consuming CPU.

How could I resolve this too ? Maybe APF could be of help ?