memory not being used

[chilihost]

I have a server that has 1GB RAM, when I do top, it shows this:
Mem: 1034428k total, 206940k used, 827488k free, 4780k buffers
Swap: 2096376k total, 116440k used, 1979936k free, 26036k cached

This is pretty consistent, checking it over several days now. It looke like the memory is not being used but the swapfile is!!! One of the top processes is consistently kswapd0

I have already removed my Rlimits in httpd.conf:
#-
#Rlimit added by apachelimits.pl
#-
#RLimitMEM 207532714
#RLimitCPU 240
##

What can it be that is limiting memory usage? Is there anywhere else I should be checking & reconfiguring?

 

[nickp666]

what kernel are you running?

 

[chilihost]

2.6.9-34.0.2.ELsmp with mysql 4.0.27 and php 4.4.2. The server is running a few heavy sql databases, so let me include the my.cnf file:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_connections=500
interactive_timeout=100
wait_timeout=100
connect_timeout=10
thread_cache_size=128
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=1024
record_buffer=1M
sort_buffer_size=2M
read_buffer_size=2M
max_connect_errors=10
# Try number of CPU's*2 for thread_concurrency
thread_concurrency=4
myisam_sort_buffer_size=64M
log-bin
server-id=1

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash
#safe-updates

[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[mysqlhotcopy]
interactive-timeout

 

[Spiral]

I have a server that has 1GB RAM, when I do top, it shows this:
Mem: 1034428k total, 206940k used, 827488k free, 4780k buffers
Swap: 2096376k total, 116440k used, 1979936k free, 26036k cached

That's perfectly normal for Linux!

And your stats above are looking very good ...

I definitely would not be complaining!



Regarding your swap file, when Linux needs to use more memory than physically
available, it will use up your available memory and then crate a swap buffer and use
that for temporary memory storage.

When the process requiring the excess memory ends, Linux will deallocate the
physical memory and make it available again. However the swap file will remain
and normally not be deleted but can be reused if your computer suddenly finds
it is again needing more memory than is physically available.

Your computer is actually using your memory correctly. You just are not seeing
it because it is making the memory available again after the memory intensive
processes are finished running.

Incidentally, your backup and statistics updates are probably what is causing your
swap to be used incidentally and is probably happening at late night times when
those cron processes run. After they are done, your physical memory is falling
back to normal usage levels again like it is supposed to do!

 

[chilihost]

The problem is the load average is thru the roof, constantly going into double digits:

top - 15:28:05 up 3 days, 2 min, 1 user, load average: 42.78, 26.51, 15.13
Tasks: 358 total, 1 running, 349 sleeping, 0 stopped, 8 zombie
Cpu(s): 3.9% us, 2.9% sy, 0.3% ni, 62.5% id, 30.4% wa, 0.1% hi, 0.0% si
Mem: 1034428k total, 402164k used, 632264k free, 3288k buffers
Swap: 2096376k total, 318688k used, 1777688k free, 21308k cached

I know what is using up processing power, its some traffic trade scripts that run every minute via cron. But even when they are running (it takes about 5 seconds for them to run), memory is still mostly unused.

 

[nickp666]

its the kernel itself thats caining the resources, I have a similar setup and recently went up to the current stable centos kernel version, my load has trippled, the best suggestion (according to another thread here) is to update to a newer kernel

 

[jester.ro]

actually that doesn't look too good

cat you do a

cat /proc/sys/vm/swappiness

and tell us the value? Default should be 60. If it's 60, you could try to decrease it (to say 30), and see how it goes.

Also, have these problems appear after the upgrade to 0.2 kernel or it was the same before?

The load average is so high becose of the swapping, processes are io/waiting for others to finish.

If you have 2 hard drives in that machine, i'd suggest moving the swap to the other drive. Might help.

 

[chilihost]

nickp666, unfortunately there is no newer kernel to upgrade, 2.6.9-34.0.2.ELsmp is the latest one. I was running 2.6.9-34.ELsmp and tried to fix it with the kernel upgrade but that did not help. I can swap it back to that one or 2.6.9-22.ELsmp but I am not keen on going that far back.

jester.ro, I have already set /proc/sys/vm/swappiness and /proc/sys/vm/vfs_cache_pressure to 0 as per this thread but this did not help

I will try moving the swap to the second disk, that should speed things up a bit

 

[nickp666]

you will have to do a manual compile or download rpms of newer kernels as you are correct, the version you have is the latest in the CentOS repositories

 

[gahelm]

I can vouch for this as well. I upgraded to 2.6.9_34.0.2 and it nearly killed my server. Would only use about a quarter of the ram and was swapping like crazy. I manually downloaded 2.6.9_39.0.2 and installed it via the rpm command and it fixed my problems completely. Now my server is utilizing 800 to 900 megs of the 1gig of installed ram and only 144k of swap. All sites are much faster, and 0% IOWait.

 

[chilihost]

I have downloaded and installed kernel 2.6.9-39.0.2.ELsmp from rpm directly from http://dev.centos.org/centos/4/testing/i386/RPMS/ and it rebooted without a problem (always a good first indication)

now wish me luck, I will update the results after its been running for a while.

 

[chilihost]

Things are looking AMAZING so far with this new kernel, memory is being used properly and we haven't even touched the swap file yet, almost 1.5 hours after the reboot.

top - 12:25:51 up 1:24, 1 user, load average: 0.07, 0.09, 0.06
Tasks: 240 total, 1 running, 227 sleeping, 0 stopped, 12 zombie
Cpu(s): 3.5% us, 2.8% sy, 0.0% ni, 91.7% id, 2.0% wa, 0.0% hi, 0.0% si
Mem: 1034416k total, 546960k used, 487456k free, 47504k buffers
Swap: 2096376k total, 0k used, 2096376k free, 206516k cached

a huge thank you for all your help.

 

[gahelm]

I was pretty sure you'd be happy. So far mine has been running great with only 144k of swap used. 3 days and counting....

 

[chilihost]

prolly should open a new topic for this one....but is there a guide to kernel numbering somewhere?

 

[mahinder]

i think you have problem with your kernel. I had similar problem with one my new server and my DC re-installed kernel and problem now seems fixed. I think this problem with with new CENTOS kernel.

well, it if gets resolved, let us all know how. i told you mine

 

[chilihost]

mahinder, i DID get this fixed, please re-read the above responses : )

 

[Echelon17]

As I already stated weeks ago when I created the CentOS Kernel thread, this is a known problem. The Kernel does NOT handle memory well and only the recent 'beta' Kernels have this issue resolved.

If you are using CentOS on a server that was freshly built in the last 3-4 months, you MUST upgrade or you WILL experience this terrible performance, no ifs and buts, it WILL happen.

Upgrade immediately, it takes you no more than a couple of minutes.

 

[Webjunkie]

Had the same problem with a CentOS 4.3 box. Downloaded 2.6.9-39.0.2.ELsmp, did rpm -Uvh, reboot, and the problem is solved. Pretty easy to fix.