The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

I can't solve this - Apache eating 170mb per process

Discussion in 'Workarounds and Optimization' started by darthvader, Sep 20, 2011.

  1. darthvader

    darthvader Registered

    Joined:
    Sep 20, 2011
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Hi, I have read a LOT about apache optimization to solve this issue, but nothing worked :S

    on my server every httpd process is consuming like 150-180mb per process, with 'normal traffic' the server load average is like 1 - 1.2, I have just one website (Drupal based) with about 800K pageviews month, but when the traffic increase the load goes over 40, 50 etc and so the server goes down.

    Could somebody please help me ? :confused:
    Code:
    Intel Dual Xeon E5645 Hexa Core @ 2.40GHz
    4GB Total RAM
    Apache/2.2.19 -[B]prefork[/B]- (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4
    PHP/5.2.17 [B]-dso-[/B]
    500GB HD RAID 1
    APC Installed
    Drupal with Boost Module installed, INNODB tables
    

    Top with normal traffic

    Code:
    top - 16:27:19 up 1 day, 3 min,  1 user,  load average: 0.90, 0.95, 0.98
    Tasks: 543 total,   1 running, 541 sleeping,   0 stopped,   1 zombie
    Cpu(s):  3.8%us,  0.6%sy,  0.0%ni, 95.2%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:   4034276k total,  3609412k used,   424864k free,   217392k buffers
    Swap:  6088624k total,   297124k used,  5791500k free,  1194644k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                       
     9044 mysql     15   0 1352m 872m 4440 S 31.9 22.1 172:58.66 mysqld                                        
    25220 nobody    15   0  371m 169m  26m S  0.0  4.3   0:03.50 httpd                                         
    25353 nobody    15   0  369m 169m  25m S  0.0  4.3   0:04.30 httpd                                         
    25378 nobody    15   0  369m 169m  25m S  7.0  4.3   0:02.37 httpd                                         
    25580 nobody    15   0  371m 169m  26m S  0.0  4.3   0:02.83 httpd                                         
    25597 nobody    15   0  367m 165m  26m S  0.0  4.2   0:02.26 httpd                                         
    25575 nobody    16   0  366m 165m  25m S  0.0  4.2   0:02.82 httpd                                         
    25372 nobody    15   0  362m 163m  27m S  0.0  4.2   0:04.95 httpd                                         
    25348 nobody    15   0  365m 163m  26m S  0.0  4.2   0:04.37 httpd                                         
    25354 nobody    15   0  364m 163m  26m S  0.3  4.1   0:03.81 httpd                                         
    25363 nobody    15   0  364m 162m  26m S  0.0  4.1   0:03.13 httpd                                         
    25608 nobody    15   0  363m 162m  25m S  0.0  4.1   0:03.30 httpd                                         
    25113 nobody    15   0  362m 162m  25m S  0.7  4.1   0:05.54 httpd                                         
    25219 nobody    15   0  364m 162m  26m S  0.0  4.1   0:03.51 httpd                                         
    25343 nobody    15   0  364m 162m  26m S  1.0  4.1   0:03.42 httpd                                         
    25232 nobody    15   0  364m 162m  26m S  0.7  4.1   0:03.59 httpd                                         
    25605 nobody    15   0  362m 161m  25m S  0.0  4.1   0:02.25 httpd                                         
    25698 nobody    15   0  362m 161m  25m S  8.3  4.1   0:01.82 httpd                                         
    25695 nobody    15   0  364m 161m  25m S  0.7  4.1   0:00.57 httpd                                         
    25131 nobody    16   0  362m 161m  25m S  8.3  4.1   0:05.60 httpd                                         
    25796 nobody    16   0  362m 161m  24m S  9.0  4.1   0:00.52 httpd                                         
    25216 nobody    15   0  361m 161m  26m S  3.7  4.1   0:05.07 httpd                                         
    25559 nobody    15   0  361m 161m  25m S  0.3  4.1   0:03.28 httpd                                         
    25609 nobody    15   0  361m 161m  25m S  0.0  4.1   0:01.70 httpd
    Top with server load over 200
    Code:
    top - 12:27:13 up 5 days,  9:04,  1 user,  load average: 219.36, 189.93, 130.56
        Tasks: 750 total,   1 running, 749 sleeping,   0 stopped,   0 zombie
        Cpu(s):  1.3%us,  1.0%sy,  0.1%ni, 49.7%id, 47.8%wa,  0.0%hi,  0.1%si,  0.0%st
        Mem:   4034276k total,  4014052k used,    20224k free,    13404k buffers
        Swap:  6088624k total,  3036872k used,  3051752k free,    71272k cached
        
          PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                       
        15653 mysql     15   0 1444m 154m 3384 S  0.0  3.9 192:42.76 mysqld                                        
        23800 nobody    16   0  422m  96m  29m D  0.3  2.4   0:06.96 httpd                                         
        23527 nobody    16   0  422m  93m  29m D  0.0  2.4   0:05.11 httpd                                         
        23759 nobody    15   0  421m  91m  26m S  0.7  2.3   0:03.97 httpd                                         
        23842 nobody    16   0  422m  91m  26m D  0.3  2.3   0:04.88 httpd                                         
        23819 nobody    16   0  421m  91m  26m D  0.3  2.3   0:07.11 httpd                                         
        23739 nobody    16   0  421m  91m  26m D  0.0  2.3   0:10.27 httpd                                         
        23778 nobody    15   0  421m  91m  26m S  0.0  2.3   0:04.81 httpd                                         
        23790 nobody    15   0  421m  91m  26m S  0.0  2.3   0:03.86 httpd                                         
        23754 nobody    16   0  421m  91m  26m D  0.0  2.3   0:08.19 httpd                                         
        23700 nobody    16   0  421m  90m  26m D  0.3  2.3   0:05.45 httpd                                         
        23843 nobody    16   0  420m  90m  26m S  0.0  2.3   0:06.39 httpd                                         
        23510 nobody    16   0  426m  90m  24m D  0.0  2.3   0:04.98 httpd                                         
        23841 nobody    16   0  416m  89m  29m D  0.0  2.3   0:03.53 httpd                                         
        23836 nobody    15   0  414m  89m  30m S  0.0  2.3   0:05.82 httpd                                         
        23849 nobody    15   0  418m  88m  25m S  0.0  2.3   0:05.78 httpd                                         
        23833 nobody    16   0  429m  88m  24m D  0.0  2.3   0:05.59 httpd                                         
        23832 nobody    16   0  418m  88m  25m S  0.0  2.2   0:09.25 httpd                                         
        23746 nobody    16   0  428m  88m  25m D  0.0  2.2   0:04.13 httpd                                         
        23851 nobody    16   0  428m  88m  24m D  0.0  2.2   0:03.60 httpd                                         
        23816 nobody    15   0  418m  88m  25m S  0.0  2.2   0:07.00 httpd                                         
        23282 nobody    15   0  416m  87m  28m S  0.0  2.2   0:11.29 httpd                                         
        23742 nobody    15   0  416m  86m  26m S  0.0  2.2   0:09.37 httpd                                         
        23837 nobody    16   0  425m  86m  25m D  0.3  2.2   0:05.20 httpd                                         
        23093 nobody    16   0  430m  86m  24m D  0.0  2.2   0:04.19 httpd                                         
        23732 nobody    16   0  421m  86m  24m D  0.0  2.2   0:05.55 httpd                                         
        23772 nobody    15   0  415m  85m  29m S  1.0  2.2   0:14.55 httpd
    my.cnf

    Code:
    [mysqld]
    max_connections = 120
    safe-show-database
    skip-locking
    key_buffer = 148M
    max_allowed_packet = 14M
    table_cache = 640
    sort_buffer_size = 2M
    read_buffer_size = 2M
    read_rnd_buffer_size = 2M
    myisam_sort_buffer_size = 64M
    thread_cache_size = 24
    query_cache_size= 136M
    thread_concurrency = 48
    wait_timeout = 45
    innodb_file_per_table
    innodb_log_file_size = 10485760
    open_files_limit = 8192
    tmp_table_size=296M
    max_heap_table_size=296M
    innodb_buffer_pool_size=596M
    local-infile=0
    log_slow_queries = /var/log/slow.log
    long_query_time = 5
    interactive_timeout = 100
    
    
    [mysqldump]
    quick
    max_allowed_packet = 16M
    
    [mysqld_safe]
    log-error=/var/log/mysqld.log
    
    [isamchk]
    key_buffer = 128M
    sort_buffer_size = 64M
    read_buffer = 2M
    write_buffer = 2M
    
    [myisamchk]
    key_buffer = 128M
    sort_buffer_size = 64M
    read_buffer = 2M
    write_buffer = 2M
    
    httpd.conf
    Code:
    Timeout 30
    TraceEnable On
    ServerSignature Off
    ServerTokens Full
    FileETag All
    StartServers 5
    <IfModule prefork.c>
    MinSpareServers 5
    MaxSpareServers 20
    </IfModule>
    ServerLimit 256
    MaxClients 150
    MaxRequestsPerChild 500
    KeepAlive On
    KeepAliveTimeout 5
    MaxKeepAliveRequests 50
    
    
    PHP.ini http://dl.dropbox.com/u/33784/info.html

    iostat -dx 5
    Code:
    [root@host /home/dev]# vi /etc/my.cnf
    [root@host /home/dev]# vim /etc/httpd/conf/httpd.conf
    [root@host /home/dev]# iostat -dx 5����
    
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               9.22   265.34 12.59 63.69   674.29  2635.49    43.39     5.75   75.42   2.42  18.46
    sda1              0.15     0.00  0.00  0.00     0.30     0.00    67.13     0.00    2.26   1.57   0.00
    sda2              1.42     2.81  0.85  1.66    64.76    35.47    39.95     0.33  132.67   8.04   2.02
    sda3              3.52    13.16  0.90  0.78    35.31   115.34    90.18     0.37  221.54   3.33   0.56
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     2.00     0.00    0.25   0.25   0.00
    sda5              0.18    93.17  2.83 39.28    87.68  1059.64    27.24     1.76   41.73   2.31   9.74
    
    
    I will really appreciate your help, I tried like 20 different configurations for apache and mysql, but nothing worked, Im starting to think is something related to hardware =/

    Thank you
     
    #1 darthvader, Sep 20, 2011
    Last edited: Sep 20, 2011
  2. cPdans

    cPdans Member
    Staff Member

    Joined:
    Sep 6, 2011
    Messages:
    13
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Hello,

    This would not be a hardware related issue. You might try reducing maxrequestsperchild further. In addition, you might reduce keepalive requests or turn off keepalive altogether. MPM worker can also reduce the memory footprint but can have issues. All of these are just suggestions. You should check the site after implementing them to verify that they are working. Beyond this, you should have an apache admin review the configuration and see if they can make more suggestions.

    On a drupal site, this is usually related to modules, "plugins" or addons, whichever you prefer. You can most likely find the cause by disabling all of them and reenabling them one at a time.
     
  3. srpurdy

    srpurdy Well-Known Member

    Joined:
    Jun 1, 2011
    Messages:
    101
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    KeepAlive Off

    Also are you running a heavy amount of mod_security rules? you might want to adjust those if you are. Antimalware from gotroot for example uses a ton of memory on httpd. Although it's useful if you can handle it, but with keepalive on and a typical drupal site probably using a ridiculous amount of http requests can easily cause this. The other option is to optimize the website itself and reduce the http requests, you should do that anyway but that could be a long job. So first thing would be to not use keepalive and see this improves things, then work torward optimizing the website itself.

    Shawn
     
  4. tquang

    tquang Member

    Joined:
    Sep 22, 2011
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Besides optimize for apache, you need take care security (it will decrease resource from dos, illegal access). And as @srpurdy said, you need install mod_security to drop illegal access

    And within mysql, if your server do not serve for outside, you need prevent that (by config mysql or firewall)

    Code:
    Timeout 30
    TraceEnable Off
    ServerSignature Off
    ServerTokens Prod
    FileETag None
    StartServers 5
    <IfModule prefork.c>
    MinSpareServers 2
    #MaxSpareServers 20
    </IfModule>
    ServerLimit 256
    MaxClients 150
    MaxRequestsPerChild 10000
    KeepAlive Off
    KeepAliveTimeout 5
    MaxKeepAliveRequests 50
    And finally, config iptables for limit request from client, drop illegal packet
     
    #4 tquang, Sep 22, 2011
    Last edited: Sep 22, 2011
  5. sOliver

    sOliver Active Member

    Joined:
    Oct 25, 2010
    Messages:
    32
    Likes Received:
    0
    Trophy Points:
    6
    Do you have a caching plugin for Drupal? Especially database caching? If not install it.

    Check /var/log/slow.log for long queries - disable all plugins that are not 100% required. Seriously, disable them, it will help

    Make sure to clean your database from old unused plugins. Sometimes, CMS's like WP and Drupal still have lots of stuff there that needs to be removed.

    Change your timeout to 15 in httpd.conf, but KeepAlive needs to be ON if you have a lot of images

    Check your error_log for any suspicious messages. Is your MaxClients ok?

    Download mysqltuner.pl and tuning-primer.sh and check with both tools to optimize sql

    Install mod_security, CSF firewall. Consider buying Gotroot rules (ASL-lite)

    Make sure you are not getting DDOS'd - check your domlogs and grep the most accessed URL and the IP with most accesses. Go to ip2location.com and possible block the country (CN, IR, VN, TR, IR.... )
     
  6. WebHostDog

    WebHostDog Well-Known Member

    Joined:
    Sep 3, 2006
    Messages:
    144
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Website Owner
    Try to update Apache to version 2.2.21 to see if this will help any. Can install cpnginx for static content too.
     
Loading...

Share This Page