The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Troubleshooting memory & swap usage

Discussion in 'General Discussion' started by headout, Apr 23, 2007.

  1. headout

    headout Well-Known Member

    Joined:
    Aug 20, 2003
    Messages:
    78
    Likes Received:
    0
    Trophy Points:
    6
    Hi all,

    we are having issues with high load, high memory and swap usage.
    Code:
    Apr 23 14:59:22 s01 /kernel: pid 87516 (httpd), uid 65534, was killed: out of swap space
    Apr 23 15:01:46 s01 /kernel: swap_pager_getswapspace: failed
    Apr 23 15:02:00 s01 last message repeated 287 times
    Apr 23 15:02:00 s01 /kernel: pid 86953 (httpd), uid 65534, was killed: out of swap space
    Apr 23 15:06:58 s01 /kernel: swap_pager_getswapspace: failed
    Apr 23 15:07:04 s01 last message repeated 465 times
    Apr 23 15:07:05 s01 /kernel: pid 53556 (httpd), uid 65534, was killed: out of swap space
    This is happinging for acouple of days right now. Restarting httpd solves it for just a few hours.
    Top:
    Code:
    last pid:  2588;  load averages:  2.16,  2.16,  2.35                                                                                up 34+02:06:25  15:31:19
    94 processes:  3 running, 86 sleeping, 4 stopped, 1 zombie
    CPU states: 59.6% user,  0.0% nice, 34.9% system,  0.0% interrupt,  5.5% idle
    Mem: 1171M Active, 172M Inact, 576M Wired, 88M Cache, 199M Buf, 3004K Free
    Swap: 2048M Total, 934M Used, 1114M Free, 45% Inuse, 340K In
    
      PID USERNAME PRI NICE  SIZE    RES STATE    TIME   WCPU    CPU COMMAND
    49195 mysql     43   0   386M 74244K RUN     69:33 19.87% 19.87% mysqld
    89881 nobody     2   0   415M   284M sbwait   1:37 11.04% 11.04% httpd
    89877 nobody     2   0   206M   176M sbwait   0:47 10.89% 10.89% httpd
     2285 nobody     2   0   220M   204M sbwait   0:43 10.40% 10.40% httpd
     2511 nobody    36   0 51512K 35748K RUN      0:07 10.12%  9.91% httpd
     2463 root      33   0  2064K  1084K RUN      0:05  3.24%  3.22% top
    89889 nobody     2   0 32200K 14172K sbwait   0:06  2.14%  1.66% httpd
    90249 nobody     2   0 30860K 10256K sbwait   0:06  1.44%  1.12% httpd
    89909 nobody     2   0   296M 24376K sbwait   1:00  1.15%  0.88% httpd
     1951 nobody     2   0 32564K 15964K sbwait   0:04  0.88%  0.88% httpd
     2380 nobody     2   0 25476K  9748K sbwait   0:01  0.73%  0.73% httpd
     2314 root       2   0 65612K 41896K poll     0:02  0.81%  0.68% perl
    89910 nobody     2   0 31116K  9304K sbwait   0:07  0.59%  0.49% httpd
    89884 nobody     2   0 29228K 13428K sbwait   0:05  0.44%  0.44% httpd
    89874 nobody     2   0   377M 22932K sbwait   1:19  0.34%  0.34% httpd
     2482 nobody     2   0 26528K 10848K sbwait   0:01  0.30%  0.29% httpd
    89875 nobody     2   0 30808K 10332K sbwait   0:05  0.24%  0.24% httpd
     2584 nobody     2   0 25668K  9824K sbwait   0:00  2.56%  0.24% httpd
    89886 nobody     2   0 32060K  8712K sbwait   0:07  0.15%  0.15% httpd
     2457 nobody     2   0 22664K  6852K sbwait   0:00  0.10%  0.10% httpd
     1953 nobody     2   0   338M   323M sbwait   1:05  0.05%  0.05% httpd
      484 root       2   0  3864K   944K select  84:17  0.00%  0.00% <snmpd>
    93868 bind       2   0 10448K  3240K select   5:52  0.00%  0.00% named
      124 root       2   0   984K   284K select   4:52  0.00%  0.00% syslogd
      140 root       2   0  3008K     0K select   4:31  0.00%  0.00% <sshd>
      549 root       2   0  2392K     0K accept   0:47  0.00%  0.00% <pure-authd>
     4240 mailman    2   0  8756K  1084K poll     0:28  0.00%  0.00% python2.4
     4244 mailman    2   0 10120K  1060K poll     0:26  0.00%  0.00% python2.4
      138 root      10   0  1024K   252K nanslp   0:26  0.00%  0.00% cron
     4245 mailman    2   0 10016K  1020K poll     0:24  0.00%  0.00% python2.4
     4242 mailman    2   0 10136K  1032K poll     0:24  0.00%  0.00% python2.4
     4241 mailman    2   0  8720K  1000K poll     0:23  0.00%  0.00% python2.4
     4239 mailman    2   0  8820K   976K poll     0:23  0.00%  0.00% python2.4
     4243 mailman    2   0  8112K   948K poll     0:23  0.00%  0.00% python2.4
    24730 root       2   0 63272K  7188K poll     0:14  0.00%  0.00% perl
     3357 mailnull   2   0  5712K   756K poll     0:13  0.00%  0.00% exim-4.64-0
      547 root       2   0  2632K     0K select   0:12  0.00%  0.00% <pure-ftpd>
    89883 nobody     2   0 29264K  9040K sbwait   0:07  0.00%  0.00% httpd
    47997 headout    2   0  5708K     0K select   0:07  0.00%  0.00% <sshd>
    89879 nobody     2   0 31348K  8676K sbwait   0:07  0.00%  0.00% httpd
    89888 nobody     2   0 32056K  9028K sbwait   0:07  0.00%  0.00% httpd
    89887 nobody     2   0 30544K  3344K select   0:05  0.00%  0.00% httpd
    89878 nobody     2   0 27684K  8632K sbwait   0:05  0.00%  0.00% httpd
    89907 nobody     2   0 30640K  3128K sbwait   0:05  0.00%  0.00% httpd
      136 root       2   0  1096K     0K select   0:04  0.00%  0.00% <inetd>
    64681 headout    2   0  5712K   404K select   0:04  0.00%  0.00% sshd
    25032 root       2   0 15276K     0K select   0:02  0.00%  0.00% <cpsrvd-ssl>
    26980 root       2  19 10856K     0K poll     0:01  0.00%  0.00% <perl5.8.7>
    90106 root       2   0  9812K  3016K select   0:01  0.00%  0.00% cppop
    89828 root       2   0 22540K  6112K select   0:01  0.00%  0.00% httpd
    90424 root       2   0 65028K     0K poll     0:01  0.00%  0.00% <perl>
     4246 mailman    2   0  8096K     0K poll     0:01  0.00%  0.00% <python2.4>
    
    We recently restarted httpd, so swap usage is just :rolleyes: 45%.

    Info about our system:
    Code:
    Server Version: Apache/1.3.37 (Unix) mod_gzip/1.3.26.1a mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7d
    PHP 4.4.4 (cgi) (built: Oct 11 2006 00:09:37)
    Copyright (c) 1997-2006 The PHP Group
    Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
        with Zend Extension Manager v1.0.8, Copyright (c) 2003-2005, by Zend Technologies
        with Zend Optimizer v2.5.10, Copyright (c) 1998-2005, by Zend Technologies
    Mysql 4.1.22
    FreeBSD 4.9-RELEASE
    Processor #1 Name: Intel(R) Pentium(R) 4 CPU 2.80GHz
    real memory  = 2147418112 (2097088K bytes)
    ad0: 78167MB  [158816/16/63] at ata0-master UDMA100
    ad1: 78167MB  [158816/16/63] at ata0-slave UDMA100
    
    Can anyone give us a advice where to look (for)?
     
  2. nxds

    nxds Well-Known Member

    Joined:
    Jan 6, 2006
    Messages:
    53
    Likes Received:
    0
    Trophy Points:
    6
    Limit Apache memory usage with "Modify Apache Memory Usage" from WHM Security menu,
     
  3. headout

    headout Well-Known Member

    Joined:
    Aug 20, 2003
    Messages:
    78
    Likes Received:
    0
    Trophy Points:
    6
    We did that, and it didn't make any difference actually. I removed the lines from httpd.conf, but i will try again.
     
  4. headout

    headout Well-Known Member

    Joined:
    Aug 20, 2003
    Messages:
    78
    Likes Received:
    0
    Trophy Points:
    6
    No difference at all:
    When we do a "lsof -p PID" on topconsuming memory processes, we just see the domlogs. Why do they keep on rotating the whole day? None of the domlogs are above the 2 GB limitsize.
     
  5. jayh38

    jayh38 Well-Known Member

    Joined:
    Mar 3, 2006
    Messages:
    1,215
    Likes Received:
    0
    Trophy Points:
    36
    You seem to have a fair amount of activity with mysql being a significant usage by clients. Are you hosting forums? Also what are the box specs, single core?

    Anyway, for your current process usage, 1 gig ram will not cut it. Especially since your swap is building to 2 gig over the available ram. I would suggest upgrading to 2 gig ram, 3 or 4 better yet.

    Also, be rid of mailman service. If your box is small, one user will really suck you dry of resources in a hurry.

    As far as your domlogs, yes, they may be delaying on your system due to loading and much of the loading of course is coming from your swap file which of course is due to unavailable ram.

    So overall, consider more ram. Perhaps you will not need as much by disabling services you do not really need. Also, offer only one stats suite, such as awstats and disable the others if you have no need.
     
  6. headout

    headout Well-Known Member

    Joined:
    Aug 20, 2003
    Messages:
    78
    Likes Received:
    0
    Trophy Points:
    6
    We host a reasonable amount of forums on this box, for now three years. The issue came up since a few days actually.
    Specs: P4 2.8 Ghz and 2 GB Ram.
    The server has already 2 GB ram, so maybe we upgrade it to 4.
    One customer uses it, since 2 years (maybe longer), so it seems this can't create a problem at this moment. But i will keep your advice in my memory.
    Ok, thanks.

    About MySQL and the usage by clients: how to track the customer which uses an unreasonable amount of resources? With the processlist there's nothing particulary to see.

    MyTop:
    Code:
    MySQL on localhost (4.1.22)                                                                                                          up 1+04:52:12 [13:32:56]
     Queries: 278.9M  qps: 2814 Slow:   872.0         Se/In/Up/De(%):    100/00/00/00
                 qps now:   30 Slow qps: 0.0  Threads:    2 (   1/  91) 65/03/16/04
     Cache Hits: 277.7M Hits/s: 2801.8 Hits now:  12.1  Ratio: 99.8% Ratio now: 61.0%
     Key Efficiency: 99.5%  Bps in/out: 28.9k/230.5   Now in/out:  4.7k/45.6k
    
          Id      User         Host/IP         DB      Time    Cmd Query or State
          --      ----         -------         --      ----    --- ----------
      113642      root       localhost   gsf_gsf2         0  Query show full processlist
      113628 colorman_       localhost colorman_p        32  Sleep
    
    
     
    #6 headout, Apr 24, 2007
    Last edited: Apr 24, 2007
  7. rachweb

    rachweb Well-Known Member

    Joined:
    Jun 26, 2004
    Messages:
    268
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    amsterdam
  8. niccell

    niccell Well-Known Member

    Joined:
    Aug 10, 2005
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    6
    Track memory/cpu Usage

    Hello!

    I'm certainly no expert but I've run across this a time or twelve.. :(

    If you install mytop it will show you the mysql processes. But I don't think mysql processes are the issue, it's the web page(s) that are driving them. some of those pages look like they are processing for an awful long time, and probably driving the mysql database into a fit.

    if you log into root and go to /proc/processname ls -la

    for some of the big resource httpd threads, it should show you which user is causing you grief. To be certain it is them, suspend them, restart httpd and see if the resources stay within normal parameters. Sometimes it is more than one account.
     
  9. headout

    headout Well-Known Member

    Joined:
    Aug 20, 2003
    Messages:
    78
    Likes Received:
    0
    Trophy Points:
    6
    We fixed it. We saw in "WHM ---> CPU/Memory/MySQL Usage" an account which was consuming huge resources. We suspended this account, about 12 hours ago from now. That fixed it for now.

    Thanks to all for the suggestions made!
     
Loading...

Share This Page