The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

top free mem swap... and their relationships

Discussion in 'Workarounds and Optimization' started by SupermanInNY, Dec 21, 2004.

  1. SupermanInNY

    SupermanInNY Well-Known Member

    Joined:
    Jul 19, 2003
    Messages:
    255
    Likes Received:
    0
    Trophy Points:
    16
    I need a little explanation on the values that show up in top or in free.
    Everyone keeps saying you can never have enough memory.
    It would be helpful to support this and explain where am I 'loosing' by NOT
    adding more RAM.

    In the past couple of days, I've experienced some high CPU values running up to 18 at some times.
    Normally,. the values are running at the 0.x and sometime inch up to 1.x
    So 18 is obviously is not the norm.

    What could affect the high not normal loads? What can explain this sudden jump in cpu load?

    Well, I have several websites that have forums in them that are very active. Those forums are using no other than MySQL databases, and I imagine they have few unoptimized queries that are keeping the server busy.

    On my server, I have P4 3.06Ghz (not a Xeon) with 1GB of RAM and RAID-5 SATA HD with lots of HD space available. I even have 256MB of RAM on the RAID card itself.

    To make things run even faster,. we have the Zend Performance Suite installed (just the caching portion of the suite - highly recommended if you can afford it).

    But, with all that, we are still running with 1GB of RAM.
    I have about 130 websites/addons domains on the server and the MySQL database is always running busy.

    The question that I'm asking is,. I have the option to add MORE RAM.
    Should I?
    The problem so far is that I don't know what value should I look at as the indicator that tells me: "hey... can't you see you are running low on RAM and we are starting to SWAP"?
    So,.. if someone can explain me what exactly is the meaning and the relationships of the values available in top, such as: used, free, shrd, active, inactive, cached and how they are
    relating to each other.. it would be very helpful.. and I'm sure not just to me.
    I have one more question to this, It is obvious that using the SWAP is not a favorable practice if I have enough RAM to use, and that is why I'm very confused when I see the values in the SWAP that says.. both used, free, avg and cached.

    If I'll add another 1GB of RAM, what should I see as far as the value changes in the top?

    I've pasted here my top and free command values


    top

    03:37:42 up 129 days, 19:46, 1 user, load average: 1.93, 2.33, 1.93
    123 processes: 119 sleeping, 3 running, 0 zombie, 1 stopped
    CPU0 states: 43.0% user 8.0% system 43.0% nice 0.0% iowait 48.0% idle
    CPU1 states: 45.0% user 8.0% system 45.0% nice 0.0% iowait 46.0% idle
    Mem: 1031880k av, 1018664k used, 13216k free, 0k shrd, 17136k buff
    91048k active, 866788k inactive
    Swap: 2097144k av, 363080k used, 1734064k free 645260k cached



    free
    total used free shared buffers cached
    Mem: 1031880 1018892 12988 0 14568 644448
    -/+ buffers/cache: 359876 672004
    Swap: 2097144 363080 1734064



    Thanks,

    -Alon.
     
    #1 SupermanInNY, Dec 21, 2004
    Last edited: Dec 21, 2004
  2. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Let's take this output:
    Code:
    Mem: 1031880k av, 1018664k used, 13216k free, 0k shrd, 17136k buff
    91048k active, 866788k inactive
    Swap: 2097144k av, 363080k used, 1734064k free 645260k cached
    So, you have 1GB of physical RAM in the server (the Mem: total), your server is using most of that (av), and there's 13216KB free. Sounds bad. Well, no, because Linux will use any spare physical RAM to cache frequently used pages in memory, hence the 645260k cache, which is more than 50% of your available memory.

    However, your swap file (which is 2GB in size), which is basically a file on a partition called swap, is being used to the tune of 363080k. This means that the OS has swapped out some pages from memory to the disk because at some point it had insufficient free+cache to cope with demand for pages in memory. It hasn't seen fit to swap them back in, because none of your processes have asked for those pages. Page swappng to disk is bad because it takes up disk I/O which is (relatively) slow and also consumes CPU cycles to perform.

    On the face of it, the server appears to be relatively OK, but the high swap file usage is something to watch. What you really need is some MRTG graph data so that you can easily see how the cache and swap usage changes over the day. If the swap file usage remains high and the cache size fluctuates and becomes very low at times, then you might benefit from more RAM on the server.

    If you were to install more RAM, then you should see your cache size go up, your swap usage drop to something very small, and your load average to improve, though between 1 and 2 for a 2 CPU server is not necessarily that bad, though obviously reasonably loaded.

    That's m take on it ;)
     
  3. SupermanInNY

    SupermanInNY Well-Known Member

    Joined:
    Jul 19, 2003
    Messages:
    255
    Likes Received:
    0
    Trophy Points:
    16
    Hi chirpy,

    Ok.. I've studied your input and could we go over few points again?
    First,.. when we set the server up, we decided not to use a /swap partition on its own.
    I have a RAID-5 for my /home and we decided to just stick it on it if I remember correctly.
    At the time it seem like a working solution as we had zero load (well we had this zero load for almost a year, so apparently this was indeed a working solution till now).

    Second, looking at this quote:
    "your swap file (which is 2GB in size), is being used to the tune of 363080k....It hasn't seen fit to swap them back in,..... Page swappng to disk is bad because it takes up disk I/O which is (relatively) slow and also consumes CPU cycles to perform."

    Does the 363080k matter? meaning,.. do they need to be flushed every so often?
    Do we care for them or not really as they just take HD space (that is my least of worries as I have over capacity in my RAID-5)?
    If there is a way to flush the 363080k and bring it to zero,. I'll be able to track the swapping better and see if there is any swapping done on regular basis to begin with.
    If I just look at the value of 363080k, that by itself means that I was 'short' of 365MB or physical RAM for a better performance load. Is that correct? If so,. then this should definatly
    be of added value that is realized immediatly.
    That is of course true if the swap is indeed running at the 363MB on daily avg usage. If this has been a single instance in some old time processing,.. I wouldn't need to care for this except if I want to cover seriously extreme cases.
    Am I correct with my assumptions here?
    One more question about the SWAP thingy: If the server is using Swap to the HD, does it overwrite previous data or does it add it to the swap file and only after I 'max' the swap file definition, does it start erasing from it?
    I know you referenced it previously in regards to "It hasn't seen fit to swap them back in, because none of your processes have asked for those pages. "
    So with regards to that quote, if another swap action is taking place, does it add to the 363080k and now it will become (assuming 40MB of added swap) 400000k? or will it
    utilize the 363080k area first, and then do new writes?

    AH.. Magic:
    "On the face of it, the server appears to be relatively OK, but the high swap file usage is something to watch. What you really need is some MRTG graph data so that you can easily see how the cache and swap usage changes over the day. If the swap file usage remains high and the cache size fluctuates and becomes very low at times, then you might benefit from more RAM on the server."

    So,. to better understand the actual utilization of the server's memory, cache and swap, I need teh MRTG graph (a new tool for me to learn and will probably bring more questions, but with better understanding).

    So,.. sorry for the shift in the this thread,.. but when I type:
    mrtg
    I get the following:

    mrtg
    Usage: mrtg <config-file>

    mrtg-2.9.17 is the Multi Router Traffic Grapher.

    If you want to know more about this tool, you might want
    to read the docs. They came together with mrtg!

    Home: http://people.ee.ethz.ch/~oetiker/webtools/mrtg/

    So how do I configure it to show? Do we have it already running and available to be seen via WHM? This sounds like 'the thing' to have.

    Just for comperisson sake.. here is my current top:

    13:08:41 up 130 days, 5:16, 1 user, load average: 0.22, 0.30, 0.28
    201 processes: 200 sleeping, 1 running, 0 zombie, 0 stopped
    CPU0 states: 5.0% user 0.1% system 0.0% nice 0.0% iowait 93.0% idle
    CPU1 states: 0.0% user 7.0% system 0.0% nice 0.0% iowait 92.0% idle
    Mem: 1031880k av, 1018388k used, 13492k free, 0k shrd, 54672k buff
    545872k active, 410284k inactive
    Swap: 2097144k av, 306376k used, 1790768k free 567636k cached



    13:34:06 up 130 days, 5:42, 1 user, load average: 0.18, 0.33, 0.26
    185 processes: 184 sleeping, 1 running, 0 zombie, 0 stopped
    CPU0 states: 5.0% user 9.0% system 0.0% nice 0.0% iowait 85.0% idle
    CPU1 states: 5.0% user 0.1% system 0.0% nice 0.0% iowait 93.0% idle
    Mem: 1031880k av, 1012500k used, 19380k free, 0k shrd, 55468k buff
    550152k active, 400372k inactive
    Swap: 2097144k av, 307804k used, 1789340k free 522212k cached


    16:25:42 up 130 days, 8:33, 1 user, load average: 0.40, 0.45, 0.54
    356 processes: 352 sleeping, 4 running, 0 zombie, 0 stopped
    CPU0 states: 6.1% user 3.2% system 0.4% nice 0.0% iowait 89.2% idle
    CPU1 states: 4.2% user 3.0% system 0.1% nice 0.0% iowait 92.1% idle
    Mem: 1031880k av, 1004396k used, 27484k free, 0k shrd, 48828k buff
    412544k active, 521220k inactive
    Swap: 2097144k av, 389552k used, 1707592k free 361428k cached



    Thanks,

    -Alon.
     
    #3 SupermanInNY, Dec 22, 2004
    Last edited: Dec 22, 2004
  4. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    I would imagine that having the swapfile on the same partition as the system would likely only introduce a small degradation in performance and nothing significant (could be wrong)

    The 363080k doesn't matter. It only really matters if you've run out of RAM and start using the swap file continuously (memory thrashing). The pages in the swapfile will be released either when the process that created them asks for access to them, or the process dies. Otherwise, they will stay in there and any further paging to the swapfile will be cumulative.

    The reason that you have pages swapped out to the swap file, is that at some point, a process asked for some memory and there was insufficient available in physical RAM, so the OS paged some old pages (that hadn't been accessed for a long time) out to the swap file and gave the physical RAM over to the process that requested it.

    As you say, it may be nothing to worry about. This could typically happen when upcp runs overnight and the backups kick in.

    Although you have MRTG installed, it isn't configured. You need to do a search for some predefined graph configurations to use.
     
  5. SupermanInNY

    SupermanInNY Well-Known Member

    Joined:
    Jul 19, 2003
    Messages:
    255
    Likes Received:
    0
    Trophy Points:
    16
    Well.. it seems like the 363080k has grown to 575764K, so aparently it does grow in size and obviously that means there is Usage of the Swap. And having that process (of using the swap) is something that is not very desirable, I'm forking the $$ for additional 2GB (1GB+1GB) of Kingston memory (If I'm lucky,. I'll have 3GB on the server,. but if for whatever reason the server doesn't like the mix of memory vendors (the other two banks have 2x512 of DDR 512 400MHz Elixir ), then I'll just have 2GB of the new Kingston.


    23:11:29 up 130 days, 15:18, 1 user, load average: 0.62, 1.07, 1.94
    321 processes: 318 sleeping, 2 running, 1 zombie, 0 stopped
    CPU0 states: 8.0% user 10.0% system 0.0% nice 0.0% iowait 80.0% idle
    CPU1 states: 9.0% user 4.0% system 0.0% nice 0.0% iowait 85.1% idle
    Mem: 1031880k av, 997468k used, 34412k free, 0k shrd, 28196k buff
    605036k active, 329684k inactive
    Swap: 2097144k av, 575764k used, 1521380k free 312020k cached


    The one thing that strikes me as an odd thing is that there are so many good utilities and configuration worth of settings that 'everone' should have.
    Why is it that there are no predefined config files that we can use?

    There is a finite set of available configurations of servers. While there are obviously various options to run and set them, I don't think it is too much to 'ask' to have an aggregation of:

    "If you are running a single P4 with 1GB use this set of configs"
    "If you are running a single P4 with 2GB use this set of configs"
    "If you are running a single P4 with 3+GB use this set of configs"

    "If you are running a single Xeon with 1GB use this set of configs"
    "If you are running a single Xeon with 2GB use this set of configs"
    "If you are running a single Xeon with 3+GB use this set of configs"

    "If you are running a dual Xeon with 1GB use this set of configs"
    "If you are running a dual Xeon with 2GB use this set of configs"
    "If you are running a dual Xeon with 3+GB use this set of configs"

    my.cnf
    httpd.conf (the server configuration optimized for this hardware config)
    MRTG (as you suggested)
    iftop (setup and running)

    All that is needed to use for initial install, is only 9 configuarions or add the 1.5GB as
    another set of optional settings.
    That's it. With that,.. about 80% of the system initial configuration is already optiimized
    for a busy load. How many procesess in the Apache.. the open tables in MySQL etc.

    I've read many "suggested things to do" to optimize one piece or another, but there are missing pieces about the specifics of configurations especially when using cPanel.
    There are some defaults that can be suggested to be used off the top and you shouldn't
    need to 'shop around' for scientific papers trying to figure if you need to have
    #
    MinSpareServers 5
    MaxSpareServers 10
    StartServers 140
    MaxClients 300

    Or some other value. As mentioned, we already know what are the possible settings for the physical hardware,.. and yes.. every year and a half they may be edited or more likely add new pre-defined hardware configs and have them optimized.

    BUT,. I would advise to add one more step to such "pre-defined" configuration files.
    Knowing that if a value of memory attribute like cached or another exceed some value in relation to the physical hardware profile,. you can already suggest the users that if they monitor the server, and specifically point them to look for the specific attributes of the utility they should be using. If they go through a routine check of 10 key elements to monitor a server and they see a 'glitch' of running out of resources, they will have the insight to upgrade the particular element that is showing the High Load values. And.. as mentioned
    it is very important to identify what is a "normal" and what is "start monitoring this" and of course "You better add more stuff here now" values.

    Just my $0.02 rant.

    Thanks for the explanation about the SWAP, and I'll try to get the config thing running with MRTG.

    -Alon
     
  6. SupermanInNY

    SupermanInNY Well-Known Member

    Joined:
    Jul 19, 2003
    Messages:
    255
    Likes Received:
    0
    Trophy Points:
    16
    Here are two utilities that are worth looking at:


    iftop

    sar

    iftop - get a nice overview of your current immediate traffic that is being used
    by your network adapters

    sar - get statistics of memory and swap memory usage.

    -Alon.
     
Loading...

Share This Page