Memory Soars to 90% after 4am each day

mydomain

Well-Known Member
Aug 10, 2003
138
0
166
RHEL Memory Soars to 90% after 4am each day

Hello,

We are running 2 cPanel servers with RHEL kernel 2.4.21-4.0.1.ELsmp (#1 SMP Thu Oct 23 01:27:36 EDT 2003)

Box is a P4 2.4GHz with 1GB DDR Ram and 2x80GB HDD.

We upgraded from RH9 to RHEL all ok. For 2 days we had Load Spikes on the server constantly with Apache utilising way too high CPU usage for extended periods which increased the loads to 10-40 regularly. We fought this with upgrades to Release, Current, Edge, Release then Edge (fun) and also installed mod_security just in case security was an issue, along with modifications to httpd.conf and my.cnf to optimise the server.

Load values have been good now for the last 20 hours however the only thing we are left with now is that every day at 4am (UK/server time) it seems that the SWAP memory becomes fully utlised, Memory usage goes up to 80-90% and i have no idea why this is happening - I am assuming that at that time normally the backups run or the Cron Jobs run daily....logwatch also. The memory usage just stays at 90% and the swap memory remains fully utilised - Load values are good however.

Please can someone help me out with some advice on setermining the cause of the memory usage jump - never had this when we were on RH9 for this server previously.

TIA
Mike
 
Last edited:

fizz

Well-Known Member
Jan 25, 2002
202
0
316
when this happens, run top to see whats consuming all memory. Sounds like updatedb might be running.
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,466
31
473
Go on, have a guess
Mike,

While your running top, as fizz suggests, check yor iowait percentage. If it is continuously high, then you may be suffering from a known bug with RHE3 kernels.

If the iowait suggests this, then you could try the following as recommended in other forums after being prised out or RH Support:

SSH in as root and invoke:

echo 100 > /proc/sys/vm/inactive_clean_percent
echo 2 10 20 > /proc/sys/vm/pagecache

This can be safely done on a running box and there is no need to reboot. If the problem was related to this issue, you should see things improving.

You should also put the above 2 lines at the bottom of your /etc/rc.d/rc.local to re-invoke them on a reboot.
 

mydomain

Well-Known Member
Aug 10, 2003
138
0
166
ok guys thanks for the input , will do the check from top later if possible -its not a single process as PRM should kill it and email a warning if that was the case? From recollection the IOWAIT was always quite reasonable - its currently as we speak 0%, CPU is 92% idle.

the load is currently a lot better than before I added mod_security to apache and tweaking my.cnf, the memory usage is now however slowly increasing over time along with SWAP usage...

I killed all mysqld processes, it went down to 38MB swap used, then restarted and it slowly worked its way up again - maybe its the my.cnf??
 
Last edited:

mydomain

Well-Known Member
Aug 10, 2003
138
0
166
Hello,

Well swap usage this morning is only 50MB, overall memory usage is 40%, this is on a box with over 200 users, load is now stable, spiked this morning when running jobs at 4am but only up to a load value of 4.8 over back down to normal at 5am and average is 0.37 which looks good - havent changed anything since last posted so it looks like everything has calmed down a little -IoWait was good at 0 - 0.7 %

Hope it stays like this now as its been a lot of work to get RHEL and cPanel running stable compared to how it was with RH9 before we upgraded.

Thanks for the help and ideas.
 

mydomain

Well-Known Member
Aug 10, 2003
138
0
166
Thanks Guys - any chance of a hand dealing with the memory leak though !!! :D
 

stftk14

Well-Known Member
Jul 10, 2003
52
0
156
just a thought, but why on earth are you running an SMP kernel when your machine only has one processor?

Try getting the normal kernel.
 

mydomain

Well-Known Member
Aug 10, 2003
138
0
166
Top doesnt show much, ps aux shows that 76% of the memory usage is from mySQL as I suspected, there are 19 entries of 4.1% each for /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=m...
 

perlchild

Well-Known Member
Sep 1, 2002
279
0
166
Re: RHEL Memory Soars to 90% after 4am each day

Originally posted by mydomain
Hello,

We are running 2 cPanel servers with RHEL kernel 2.4.21-4.0.1.ELsmp (#1 SMP Thu Oct 23 01:27:36 EDT 2003)

Box is a P4 2.4GHz with 1GB DDR Ram and 2x80GB HDD.

We upgraded from RH9 to RHEL all ok. For 2 days we had Load Spikes on the server constantly with Apache utilising way too high CPU usage for extended periods which increased the loads to 10-40 regularly. We fought this with upgrades to Release, Current, Edge, Release then Edge (fun) and also installed mod_security just in case security was an issue, along with modifications to httpd.conf and my.cnf to optimise the server.

Load values have been good now for the last 20 hours however the only thing we are left with now is that every day at 4am (UK/server time) it seems that the SWAP memory becomes fully utlised, Memory usage goes up to 80-90% and i have no idea why this is happening - I am assuming that at that time normally the backups run or the Cron Jobs run daily....logwatch also. The memory usage just stays at 90% and the swap memory remains fully utilised - Load values are good however.

Please can someone help me out with some advice on setermining the cause of the memory usage jump - never had this when we were on RH9 for this server previously.

TIA
Mike
This thread has a lot of good suggestions, but I'll had my two cents...

The time of your spike, and the symptoms you mention look like you have one, very popular webhost, whose stats get processed around that time, probably at the same time as your backup is running. It's happened on a machine I manage, and it's tricky to fiddle with the schedule just right so the two don't overlap.

On a related note:
Do you use cpbackup, or some other backup solution?
And which statistics are enabled?
 

mydomain

Well-Known Member
Aug 10, 2003
138
0
166
Hi thanks for the feedback.

The load is ok now - it was a problem about 24-48 hours ago as it was spiking all day long. I have disabled analog stats after reading about current problems users on RHEL were having with it. I also enabled mod_security and tweaked the my.cnf file as follows, but the memory usage is at 600MB out of 1GB RAM in the machine which seems too high, and the swap is 124MB used - not normal is it - or should I just stick in another 512MB of RAM in there?

[mysqld]
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_user_connections=45
max_connections=500
interactive_timeout=100
wait_timeout=100
connect_timeout=10
thread_cache_size=128
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=1024
record_buffer=1M
sort_buffer_size=1024K
read_buffer_size=1024K
max_connect_errors=10
read_rnd_buffer_size=1024K
# Don't change thread conc. below unless you have 2 cpu's
thread_concurrency=2
myisam_sort_buffer_size=32M

[safe_mysqld]
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M

[mysqlhotcopy]
interactive-timeout

It does appear that mySQL seems to be taking up most of the memory here?

cpbackup is used and runs weekly.
 
Last edited:

mydomain

Well-Known Member
Aug 10, 2003
138
0
166
Originally posted by stftk14
just a thought, but why on earth are you running an SMP kernel when your machine only has one processor?

Try getting the normal kernel.
Yeah thats a good point, when I spoke to EV1 about it after they put the image in the reply was:

"smp kernel is used if the mainboard is capable of hyperthreading. It makes no difference to performance so it is used for the 2.4 P4 also"

I could edit /etc/lilo.conf and boot up 2.4.21-4.0.1.EL - do you think it will make a difference to the memory issue or mYSQL using all the memory? Will give it a go.
 
Last edited:

Bluexnet2

Active Member
Dec 4, 2003
28
0
151
Hi all,

I'm, having the same problem in one server,

I'm running RH9 Kernel Version 2.4.20-31.9. in
a super celeron 2.4 with 512mb in ram:

18354 root 39 19 304 264 136 R N 99.8 0.0 295:50 0 updatedb

the process still running and keep the load average as 1.0 minimum, then after a day the load gets to 2.0, then 3.0 etc. and cant kill it and after 4-5 days a reboot is needed.

My memory shows negative numbers:

Physical Memory -3562157121657% 17179869184.37 GB -18014398509361000.00 KB 493.86 MB

do you think that if i change the kernel version this problem can be fixed?

Thank You
Carlos