The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server crash with mem full, please help

Discussion in 'General Discussion' started by simone70, Jan 19, 2005.

  1. simone70

    simone70 Member

    Joined:
    Nov 3, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    I'm experiencing daily server crash.
    I've noticed that Swap keep increasing during the day till the full crash.

    Usual memory load during the day is 75%, 50-60 processes.
    When it hangs is not possible to connect in SSH or http.

    SWAP is never released. Increase only. (is this normal ?)

    Running Fedora 2.4.26
    WHM 9.9.9 cPanel 9.9.9-S15
    Fedora i686 - WHM X v3.1.0

    This is are sample top headers during a server hang:

    Code:
     21:51:52  up 20:38,  1 user,  load average: 5.17, 32.45, 61.36
    218 processes: 214 sleeping, 2 running, 1 zombie, 1 stopped
    CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
               total   90.4%    0.0%    9.5%   0.0%     0.0%    0.0%    0.0%
    Mem:   188188k av,  185408k used,    2780k free,       0k shrd,    2536k buff
            36116k active,             129192k inactive
    Swap:  262136k av,  262000k used,     136k free                   10504k cached
    Code:
     22:16:04  up 21:02,  1 user,  load average: 0.97, 2.25, 14.68
    206 processes: 202 sleeping, 1 running, 1 zombie, 2 stopped
    CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
               total    1.1%    0.0%   34.6%   0.0%     0.0%    0.0%   64.2%
    Mem:   188188k av,  186036k used,    2152k free,       0k shrd,    1188k buff
            12668k active,             154512k inactive
    Swap:  262136k av,  262136k used,       0k free                    6436k cached
    while this is a sample of normal operation load:

    Code:
     00:44:07  up 34 min,  1 user,  load average: 2.37, 1.53, 1.13
    68 processes: 66 sleeping, 1 running, 1 zombie, 0 stopped
    CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
               total    0.4%    0.0%    3.3%   0.0%     0.0%    0.0%   96.1%
    Mem:   188188k av,  137444k used,   50744k free,       0k shrd,   26760k buff
            59800k active,              66560k inactive
    Swap:  262136k av,       0k used,  262136k free                   59660k cached

    Any Idea? :confused:

    Simone
     
  2. Donna

    Donna BANNED

    Joined:
    Feb 15, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Toronto, Canada
    Looks to me like whatever you're doing with the server is just using a lot of ram. Easy solution is to have your DC throw another gig of ram in there.
     
  3. jdonoso

    jdonoso Well-Known Member

    Joined:
    Nov 15, 2004
    Messages:
    61
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    The Third Rock from The Sun!
    Erm.. what kind of answer is that??
     
  4. Donna

    Donna BANNED

    Joined:
    Feb 15, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Toronto, Canada
    It's the obvious answer based on what info posted. It shows over 200 processes, most of the ram in constant use and he indicated the swap file is being used.

    Point blank he's asking more from the hardware than it can do so it's either get more ram (or a new server with more ram) or cut the load on the existing server by dropping some busy clients/sites or eliminating some of the things they are running.
     
  5. jdonoso

    jdonoso Well-Known Member

    Joined:
    Nov 15, 2004
    Messages:
    61
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    The Third Rock from The Sun!
    Well, for the things he says is having running, and the amount of processes, have you considered perhaps other problems? a worm?, kernel issue? apache missconfiguration? There is an option to limit the amount of memory apache can consume, as possible partial solution. By just telling him "upgrade your memory", you don't solve the problem, only the symptom.

    Regards,
     
  6. simone70

    simone70 Member

    Joined:
    Nov 3, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    Yes, I know upgrading ram may solve, or maybe not.
    During all the day my typical top is like this:

    Code:
     08:37:35  up  8:28,  1 user,  load average: 0.42, 0.55, 0.45
    57 processes: 56 sleeping, 1 running, 0 zombie, 0 stopped
    CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
               total    0.0%    0.0%    0.9%   0.0%     0.0%    0.0%   99.0%
    Mem:   188188k av,  162344k used,   25844k free,       0k shrd,   37744k buff
            65888k active,              84952k inactive
    Swap:  262136k av,    1604k used,  260532k free                   72444k cached
    
    the +200 processes are a consequence, not the cause.

    I'm currently montiroring with mrtg my swap and mem, and I know swap is usually below the 8% or ram usage, except when everything crash.
     
  7. dave9000

    dave9000 Well-Known Member

    Joined:
    Apr 7, 2003
    Messages:
    891
    Likes Received:
    1
    Trophy Points:
    16
    Location:
    arkansas
    cPanel Access Level:
    Root Administrator
    good start would be upgrade to the 2.6 kernel, it seems to manage memory a lot better than the 2.4

    This fixed a box we had with similar issues
     
  8. Donna

    Donna BANNED

    Joined:
    Feb 15, 2003
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Toronto, Canada
    Well if the high process count is an abnormal thing you should check top to see what is running that's causing all the usage.

    Another good resource that most sysadmins tend to ignore is cpanel's CPU/mySQL usage feature in WHM. It doesn't update often but if a specific process or two is using all that ram etc up then it should come up there at or near the top, highlighted in red.

    If we had more info we could better determine what the cause is. Could be anything from a popular download file to something someone has slipped on your server.
     
  9. brentp

    brentp Well-Known Member

    Joined:
    Mar 11, 2004
    Messages:
    324
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Ayr, North Queensland, Australia
    yes, i would also be reccomending you to upgrade to the 2.6 kernel as it can help with memory usage.

    Btw: add elevator=deadline to the grub block for that kernel, mine looks like:

    title CentOS (2.6.10-1.9_FC2)
    root (hd0,0)
    kernel /vmlinuz-2.6.10-1.9_FC2 ro root=/dev/hda3 elevator=deadline

    and its working fine.

    Regards,
    Brent
     
  10. simone70

    simone70 Member

    Joined:
    Nov 3, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    What is not cleat to me is why allocated swap does not decrease when physical ram is available. :confused:

    Swap keep increasing costantly and is never released till aprox 15% of ram, then everything hang.

    That is really something I cannot understand (I'm not a *nix expert)

    By the way after another crash debug session, I'm suspecting apache 1.3.3 could be the cause of swap increase due to some spidering activities.
     
  11. simone70

    simone70 Member

    Joined:
    Nov 3, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    I've added more ram but problem still persist.

    Now Swap do not fill all space available but it increase till some service fail.
    Sometimes Mysql sometimes Apache.

    Don't know if is the main cause, but yesterday while running top on my server happened again, and I've been able to see that someone was using an offline browser to download all my site, and there were tens of cgi loaded consuming CPU.

    How could I resolve this too ? Maybe APF could be of help ?
     
Loading...

Share This Page