The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Memory Soars to 90% after 4am each day

Discussion in 'General Discussion' started by mydomain, Apr 7, 2004.

  1. mydomain

    mydomain Well-Known Member

    Joined:
    Aug 10, 2003
    Messages:
    138
    Likes Received:
    0
    Trophy Points:
    16
    RHEL Memory Soars to 90% after 4am each day

    Hello,

    We are running 2 cPanel servers with RHEL kernel 2.4.21-4.0.1.ELsmp (#1 SMP Thu Oct 23 01:27:36 EDT 2003)

    Box is a P4 2.4GHz with 1GB DDR Ram and 2x80GB HDD.

    We upgraded from RH9 to RHEL all ok. For 2 days we had Load Spikes on the server constantly with Apache utilising way too high CPU usage for extended periods which increased the loads to 10-40 regularly. We fought this with upgrades to Release, Current, Edge, Release then Edge (fun) and also installed mod_security just in case security was an issue, along with modifications to httpd.conf and my.cnf to optimise the server.

    Load values have been good now for the last 20 hours however the only thing we are left with now is that every day at 4am (UK/server time) it seems that the SWAP memory becomes fully utlised, Memory usage goes up to 80-90% and i have no idea why this is happening - I am assuming that at that time normally the backups run or the Cron Jobs run daily....logwatch also. The memory usage just stays at 90% and the swap memory remains fully utilised - Load values are good however.

    Please can someone help me out with some advice on setermining the cause of the memory usage jump - never had this when we were on RH9 for this server previously.

    TIA
    Mike
     
    #1 mydomain, Apr 7, 2004
    Last edited: Apr 7, 2004
  2. fizz

    fizz Well-Known Member

    Joined:
    Jan 25, 2002
    Messages:
    202
    Likes Received:
    0
    Trophy Points:
    16
    when this happens, run top to see whats consuming all memory. Sounds like updatedb might be running.
     
  3. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Mike,

    While your running top, as fizz suggests, check yor iowait percentage. If it is continuously high, then you may be suffering from a known bug with RHE3 kernels.

    If the iowait suggests this, then you could try the following as recommended in other forums after being prised out or RH Support:

    SSH in as root and invoke:

    echo 100 > /proc/sys/vm/inactive_clean_percent
    echo 2 10 20 > /proc/sys/vm/pagecache

    This can be safely done on a running box and there is no need to reboot. If the problem was related to this issue, you should see things improving.

    You should also put the above 2 lines at the bottom of your /etc/rc.d/rc.local to re-invoke them on a reboot.
     
  4. mydomain

    mydomain Well-Known Member

    Joined:
    Aug 10, 2003
    Messages:
    138
    Likes Received:
    0
    Trophy Points:
    16
    ok guys thanks for the input , will do the check from top later if possible -its not a single process as PRM should kill it and email a warning if that was the case? From recollection the IOWAIT was always quite reasonable - its currently as we speak 0%, CPU is 92% idle.

    the load is currently a lot better than before I added mod_security to apache and tweaking my.cnf, the memory usage is now however slowly increasing over time along with SWAP usage...

    I killed all mysqld processes, it went down to 38MB swap used, then restarted and it slowly worked its way up again - maybe its the my.cnf??
     
    #4 mydomain, Apr 7, 2004
    Last edited: Apr 7, 2004
  5. mydomain

    mydomain Well-Known Member

    Joined:
    Aug 10, 2003
    Messages:
    138
    Likes Received:
    0
    Trophy Points:
    16
    Hello,

    Well swap usage this morning is only 50MB, overall memory usage is 40%, this is on a box with over 200 users, load is now stable, spiked this morning when running jobs at 4am but only up to a load value of 4.8 over back down to normal at 5am and average is 0.37 which looks good - havent changed anything since last posted so it looks like everything has calmed down a little -IoWait was good at 0 - 0.7 %

    Hope it stays like this now as its been a lot of work to get RHEL and cPanel running stable compared to how it was with RH9 before we upgraded.

    Thanks for the help and ideas.
     
  6. mydomain

    mydomain Well-Known Member

    Joined:
    Aug 10, 2003
    Messages:
    138
    Likes Received:
    0
    Trophy Points:
    16
  7. fizz

    fizz Well-Known Member

    Joined:
    Jan 25, 2002
    Messages:
    202
    Likes Received:
    0
    Trophy Points:
    16
    How did you set MRTG up like that? :)
    please tell heh
     
  8. richy81

    richy81 Registered

    Joined:
    Jan 29, 2004
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
  9. mydomain

    mydomain Well-Known Member

    Joined:
    Aug 10, 2003
    Messages:
    138
    Likes Received:
    0
    Trophy Points:
    16
    Thanks Guys - any chance of a hand dealing with the memory leak though !!! :D
     
  10. dave9000

    dave9000 Well-Known Member

    Joined:
    Apr 7, 2003
    Messages:
    891
    Likes Received:
    1
    Trophy Points:
    16
    Location:
    arkansas
    cPanel Access Level:
    Root Administrator
    guess it would help if i payed closer attention
     
    #10 dave9000, Apr 8, 2004
    Last edited: Apr 8, 2004
  11. stftk14

    stftk14 Well-Known Member

    Joined:
    Jul 10, 2003
    Messages:
    52
    Likes Received:
    0
    Trophy Points:
    6
    just a thought, but why on earth are you running an SMP kernel when your machine only has one processor?

    Try getting the normal kernel.
     
  12. mydomain

    mydomain Well-Known Member

    Joined:
    Aug 10, 2003
    Messages:
    138
    Likes Received:
    0
    Trophy Points:
    16
    Top doesnt show much, ps aux shows that 76% of the memory usage is from mySQL as I suspected, there are 19 entries of 4.1% each for /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=m...
     
  13. perlchild

    perlchild Well-Known Member

    Joined:
    Sep 1, 2002
    Messages:
    279
    Likes Received:
    0
    Trophy Points:
    16
    Re: RHEL Memory Soars to 90% after 4am each day

    This thread has a lot of good suggestions, but I'll had my two cents...

    The time of your spike, and the symptoms you mention look like you have one, very popular webhost, whose stats get processed around that time, probably at the same time as your backup is running. It's happened on a machine I manage, and it's tricky to fiddle with the schedule just right so the two don't overlap.

    On a related note:
    Do you use cpbackup, or some other backup solution?
    And which statistics are enabled?
     
  14. mydomain

    mydomain Well-Known Member

    Joined:
    Aug 10, 2003
    Messages:
    138
    Likes Received:
    0
    Trophy Points:
    16
    Hi thanks for the feedback.

    The load is ok now - it was a problem about 24-48 hours ago as it was spiking all day long. I have disabled analog stats after reading about current problems users on RHEL were having with it. I also enabled mod_security and tweaked the my.cnf file as follows, but the memory usage is at 600MB out of 1GB RAM in the machine which seems too high, and the swap is 124MB used - not normal is it - or should I just stick in another 512MB of RAM in there?

    [mysqld]
    skip-locking
    skip-innodb
    query_cache_limit=1M
    query_cache_size=32M
    query_cache_type=1
    max_user_connections=45
    max_connections=500
    interactive_timeout=100
    wait_timeout=100
    connect_timeout=10
    thread_cache_size=128
    key_buffer=16M
    join_buffer=1M
    max_allowed_packet=16M
    table_cache=1024
    record_buffer=1M
    sort_buffer_size=1024K
    read_buffer_size=1024K
    max_connect_errors=10
    read_rnd_buffer_size=1024K
    # Don't change thread conc. below unless you have 2 cpu's
    thread_concurrency=2
    myisam_sort_buffer_size=32M

    [safe_mysqld]
    open_files_limit=8192

    [mysqldump]
    quick
    max_allowed_packet=16M

    [mysql]
    no-auto-rehash

    [isamchk]
    key_buffer=32M
    sort_buffer=32M
    read_buffer=16M
    write_buffer=16M

    [myisamchk]
    key_buffer=32M
    sort_buffer=32M
    read_buffer=16M
    write_buffer=16M

    [mysqlhotcopy]
    interactive-timeout

    It does appear that mySQL seems to be taking up most of the memory here?

    cpbackup is used and runs weekly.
     
    #14 mydomain, Apr 8, 2004
    Last edited: Apr 8, 2004
  15. mydomain

    mydomain Well-Known Member

    Joined:
    Aug 10, 2003
    Messages:
    138
    Likes Received:
    0
    Trophy Points:
    16
    Yeah thats a good point, when I spoke to EV1 about it after they put the image in the reply was:

    "smp kernel is used if the mainboard is capable of hyperthreading. It makes no difference to performance so it is used for the 2.4 P4 also"

    I could edit /etc/lilo.conf and boot up 2.4.21-4.0.1.EL - do you think it will make a difference to the memory issue or mYSQL using all the memory? Will give it a go.
     
    #15 mydomain, Apr 8, 2004
    Last edited: Apr 8, 2004
  16. Bluexnet2

    Bluexnet2 Active Member

    Joined:
    Dec 4, 2003
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    Hi all,

    I'm, having the same problem in one server,

    I'm running RH9 Kernel Version 2.4.20-31.9. in
    a super celeron 2.4 with 512mb in ram:

    18354 root 39 19 304 264 136 R N 99.8 0.0 295:50 0 updatedb

    the process still running and keep the load average as 1.0 minimum, then after a day the load gets to 2.0, then 3.0 etc. and cant kill it and after 4-5 days a reboot is needed.

    My memory shows negative numbers:

    Physical Memory -3562157121657% 17179869184.37 GB -18014398509361000.00 KB 493.86 MB

    do you think that if i change the kernel version this problem can be fixed?

    Thank You
    Carlos
     
Loading...

Share This Page