Very high memory usage from Apache

[Sannin]

Hi there!

I have a Xen VPS with 512 guaranteed ram and 512 swap on Centos 5.6 and Cpanel 11.28.87. A few days ago i transferred a Wordpress site on my server. I didn't check how many people tried to open the site, but my server crashed after 2 - 3 mins with 350% CPU usage and disk IO peaking at 12k. I restarted my VPS but after 2 minutes same thing happened. I communicated with the owner of the Wordpress site and told him to log in one by one (that was kinda embarrassing actually ) until i figure out what was the problem. After i communicated the datacenter i understood that my VPS was swapping and eventually dieing after swap space was filled up too.

"Top" showed me that there was 10 apache processes at 29 Mbs each. Apache had the default configuration. I searched the net about how i can reduce the memory usage of the apache processes and optimize my VPS for better performance with php sites like WP and Joomla. First thing i did, was to recompile apache with EasyApache with a minimum modules that i thought they were necessary for Wordpress to work optimally. That didn't seem to work well as i will show you in a while. Second thing i did was to tweak MP prefork settings to ensure that the server would not crash again. The memory pool available for apache is 290 mbs but with 29 mbs each process's size i reduced MaxClients setting to only 9!

Fortunately the site will be down for redesigning for a few days so i have some days to try to correct things hopefully with your help. I will provide some information below, if you need some more info from me please tell me.

Php handler for php 5 is Suphp

httpd -M:

Loaded Modules:
 core_module (static)
 authn_file_module (static)
 authn_default_module (static)
 authz_host_module (static)
 authz_groupfile_module (static)
 authz_user_module (static)
 authz_default_module (static)
 auth_basic_module (static)
 include_module (static)
 filter_module (static)
 deflate_module (static)
 log_config_module (static)
 logio_module (static)
 env_module (static)
 expires_module (static)
 headers_module (static)
 unique_id_module (static)
 setenvif_module (static)
 version_module (static)
 proxy_module (static)
 proxy_connect_module (static)
 proxy_ftp_module (static)
 proxy_http_module (static)
 proxy_scgi_module (static)
 proxy_ajp_module (static)
 proxy_balancer_module (static)
 ssl_module (static)
 mpm_prefork_module (static)
 http_module (static)
 mime_module (static)
 status_module (static)
 autoindex_module (static)
 asis_module (static)
 info_module (static)
 suexec_module (static)
 cgi_module (static)
 negotiation_module (static)
 dir_module (static)
 actions_module (static)
 userdir_module (static)
 alias_module (static)
 rewrite_module (static)
 so_module (static)
 bwlimited_module (shared)
 suphp_module (shared)
 security2_module (shared)
Syntax OK

top:

top - 04:04:09 up 1 day, 47 min,  1 user,  load average: 0.14, 0.15, 0.09
Tasks:  96 total,   1 running,  95 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    502008k total,   396256k used,   105752k free,     5680k buffers
Swap:   524284k total,    77400k used,   446884k free,    91308k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                               
 5300 root      20   0  220m 150m 2956 S  0.0 30.8   2:13.44 httpd                                                                                                                 
21297 nobody    20   0  220m 150m 1840 S  0.0 30.7   0:01.01 httpd                                                                                                                 
19991 nobody    20   0  220m 150m 1840 S  0.0 30.6   0:01.18 httpd                                                                                                                 
20062 nobody    20   0  220m 150m 1856 S  0.0 30.6   0:00.83 httpd                                                                                                                 
19990 nobody    20   0  220m 150m 1840 S  0.0 30.6   0:01.10 httpd                                                                                                                 
22862 nobody    20   0  220m 150m 1840 S  0.0 30.6   0:00.50 httpd                                                                                                                 
21310 nobody    20   0  220m 150m 1836 S  0.0 30.6   0:00.71 httpd                                                                                                                 
32455 root      20   0  146m  34m 1752 S  0.0  7.0   0:11.98 spamd                                                                                                                 
 3371 root      20   0  145m  17m 1524 S  0.0  3.7   0:15.35 spamd                                                                                                                 
12719 root      20   0 63528  13m 1128 S  0.0  2.7   0:09.58 lfd                                                                                                                   
 2846 mysql     20   0  115m 8900 2368 S  0.0  1.8   5:33.98 mysqld                                                                                                                
19988 root      20   0 45924 7152 1276 S  0.0  1.4   0:00.11 leechprotect                                                                                                          
 2810 ntp       20   0 23448 5076 3904 S  0.0  1.0   0:05.24 ntpd                                                                                                                  
 3568 root      20   0 41424 3828 1236 S  0.0  0.8   0:18.88 tailwatchd                                                                                                            
 3551 root      20   0 86544 3612  660 S  0.0  0.7   0:09.19 cpsrvd-ssl                                                                                                            
 3558 root      20   0 37468 2164  636 S  0.0  0.4   0:00.68 queueprocd                                                                                                            
 3527 root      20   0 90464 1992  680 S  0.0  0.4   0:00.00 cpdavd                                                                                                                
 3841 root      20   0 91388 1828 1552 S  0.0  0.4   0:07.47 sshd                                                                                                                  
 3667 named     20   0  233m 1576  456 S  0.0  0.3   0:00.88 named                                                                                                                 
 3412 dovecot   20   0 38232 1472 1376 S  0.0  0.3   0:00.10 pop3-login                                                                                                            
 3411 dovecot   20   0 38232 1428 1332 S  0.0  0.3   0:00.06 pop3-login                                                                                                            
 3414 dovecot   20   0 38244 1372 1268 S  0.0  0.3   0:00.08 imap-login                                                                                                            
 3413 dovecot   20   0 38244 1320 1224 S  0.0  0.3   0:00.13 imap-login                                                                                                            
 3873 root      20   0 10956 1280 1000 S  0.0  0.3   0:00.26 bash                                                                                                                  
25418 root      20   0 12804 1184  860 R  0.0  0.2   0:00.10 top                                                                                                                   
 3399 root      20   0 19104  896  752 S  0.0  0.2   0:01.88 dovecot-auth                                                                                                          
 3422 root      20   0 93424  824  820 S  0.0  0.2   0:00.00 pure-ftpd                                                                                                             
 3569 root      38  18 23480  784  460 S  0.0  0.2   0:00.22 cpanellogd                                                                                                            
 2796 root      20   0 62692  660  512 S  0.0  0.1   0:00.00 sshd                                                                                                                  
 3398 root      20   0 12464  648  516 S  0.0  0.1   0:04.82 dovecot                                                                                                               
 3424 root      20   0 35948  648  648 S  0.0  0.1   0:00.00 pure-authd                                                                                                            
 2822 root      20   0 10824  616  612 S  0.0  0.1   0:00.01 mysqld_safe                                                                                                           
 3432 root      20   0 19720  588  512 S  0.0  0.1   0:00.79 crond                                                                                                                 
 3145 mailnull  20   0 64284  568  468 S  0.0  0.1   0:00.35 exim                                                                                                                  
    1 root      20   0 10412  540  508 S  0.0  0.1   0:01.81 init                                                                                                                  
 3590 root      20   0  3852  416  412 S  0.0  0.1   0:00.00 mingetty                                                                                                              
 2720 root      20   0  5972  388  388 S  0.0  0.1   0:00.61 syslogd                                                                                                               
 1210 root      16  -4 12680  340  336 S  0.0  0.1   0:01.11 udevd                                                                                                                 
 2664 root      20   0  6804  264  160 S  0.0  0.1   0:00.00 dhclient

Look at the apache process memory size. That is obviously wrong and i dont know why but still judging from the free memory i have it is still very big.


free -M:

             total       used       free     shared    buffers     cached
Mem:           490        418         72          0          6        119
-/+ buffers/cache:        291        198
Swap:          511         75        436

lsof:

COMMAND  PID USER   FD   TYPE DEVICE    SIZE   NODE NAME
httpd   5300 root  cwd    DIR  202,0    4096      2 /
httpd   5300 root  rtd    DIR  202,0    4096      2 /
httpd   5300 root  txt    REG  202,0 1344418 468075 /usr/local/apache/bin/httpd
httpd   5300 root  mem    REG  202,0   53880 794725 /lib64/libnss_files-2.5.so
httpd   5300 root  mem    REG  202,0 1173890 471470 /usr/local/apache/modules/mod_security2.so
httpd   5300 root  mem    REG  202,0  653255  73967 /opt/lua/lib/liblua-5.1.3.so
httpd   5300 root  mem    REG  202,0 4157882 163870 /opt/xml2/lib/libxml2.so.2.7.8
httpd   5300 root  mem    REG  202,0   56072 794699 /lib64/libgcc_s-4.1.2-20080825.so.1
httpd   5300 root  mem    REG  202,0  973680   7702 /usr/lib64/libstdc++.so.6.0.8
httpd   5300 root  mem    REG  202,0   60119 468198 /usr/local/apache/modules/mod_suphp.so
httpd   5300 root  mem    REG  202,0    9433 468151 /usr/local/apache/modules/mod_bwlimited.so
httpd   5300 root  mem    REG  202,0  245232   3095 /lib64/libsepol.so.1
httpd   5300 root  mem    REG  202,0   92960   8133 /lib64/libselinux.so.1
httpd   5300 root  mem    REG  202,0   89800 794727 /lib64/libresolv-2.5.so
httpd   5300 root  mem    REG  202,0    7176   3280 /lib64/libkeyutils-1.2.so
httpd   5300 root  mem    REG  202,0   33224   4200 /usr/lib64/libkrb5support.so.0.1
httpd   5300 root  mem    REG  202,0  151176   4186 /usr/lib64/libk5crypto.so.3.1
httpd   5300 root  mem    REG  202,0    7736   3253 /lib64/libcom_err.so.2.1
httpd   5300 root  mem    REG  202,0  611312   4198 /usr/lib64/libkrb5.so.3.3
httpd   5300 root  mem    REG  202,0  188328   4182 /usr/lib64/libgssapi_krb5.so.2.2
httpd   5300 root  mem    REG  202,0 1716720   3295 /lib64/libc-2.5.so
httpd   5300 root  mem    REG  202,0   20424 794721 /lib64/libdl-2.5.so
httpd   5300 root  mem    REG  202,0  142696   3257 /lib64/libpthread-2.5.so
httpd   5300 root  mem    REG  202,0   45728   4144 /lib64/libcrypt-2.5.so
httpd   5300 root  mem    REG  202,0   50288 794728 /lib64/librt-2.5.so
httpd   5300 root  mem    REG  202,0   15288   3185 /lib64/libuuid.so.1.2
httpd   5300 root  mem    REG  202,0  273488 468006 /usr/local/apache/lib/libapr-1.so.0.4.2
httpd   5300 root  mem    REG  202,0  140880   3104 /lib64/libexpat.so.0.5.0
httpd   5300 root  mem    REG  202,0  200852 468052 /usr/local/apache/lib/libaprutil-1.so.0.3.10
httpd   5300 root  mem    REG  202,0  611880 794722 /lib64/libm-2.5.so
httpd   5300 root  mem    REG  202,0  210897 158558 /opt/pcre/lib/libpcre.so.0.0.1
httpd   5300 root  mem    REG  202,0   83344   7646 /usr/lib64/libz.so.1.2.3
httpd   5300 root  mem    REG  202,0 1363952   3316 /lib64/libcrypto.so.0.9.8e
httpd   5300 root  mem    REG  202,0  312344   3314 /lib64/libssl.so.0.9.8e
httpd   5300 root  mem    REG  202,0  137344 794641 /lib64/ld-2.5.so
httpd   5300 root  DEL    REG    0,4          18822 /dev/zero
httpd   5300 root    0r   CHR    1,3            327 /dev/null
httpd   5300 root    1w   CHR    1,3            327 /dev/null
httpd   5300 root    2w   REG  202,0 5066230 322638 /usr/local/apache/logs/error_log
httpd   5300 root    3w   REG  202,0       0 323058 /usr/local/apache/logs/modsec_audit.log
httpd   5300 root    4r   REG  202,0  401441 393583 /var/cpanel/locale/en.gdbm
httpd   5300 root    5w   REG  202,0       0 323059 /usr/local/apache/logs/modsec_debug_log
httpd   5300 root    6u  IPv4  22044            TCP *:http (LISTEN)
httpd   5300 root    7u  IPv4  22050            TCP *:https (LISTEN)
httpd   5300 root    8r  FIFO    0,8         337638 pipe
httpd   5300 root    9w  FIFO    0,8         337638 pipe
httpd   5300 root   10w   REG  202,0 5295941 322639 /usr/local/apache/logs/access_log
httpd   5300 root   11w   REG  202,0    5976 360479 /usr/local/apache/domlogs/**********
httpd   5300 root   12w   REG  202,0    1295 363375 /usr/local/apache/domlogs/*********-bytes_log
httpd   5300 root   13w   REG  202,0    5976 363469 /usr/local/apache/domlogs/*********-ssl_log
httpd   5300 root   14w   REG  202,0       0 361946 /usr/local/apache/domlogs/***********
httpd   5300 root   15w   REG  202,0       0 360458 /usr/local/apache/domlogs/**********-bytes_log
httpd   5300 root   16w   REG  202,0       0 361949 /usr/local/apache/domlogs/**********
httpd   5300 root   17w   REG  202,0       0 361953 /usr/local/apache/domlogs/**********-bytes_log
httpd   5300 root   18w   REG  202,0       0 360478 /usr/local/apache/domlogs/**********
httpd   5300 root   19w   REG  202,0       0 360477 /usr/local/apache/domlogs/************-bytes_log
httpd   5300 root   20w   REG  202,0 2672252 364109 /usr/local/apache/domlogs/********
httpd   5300 root   21w   REG  202,0    8387 361990 /usr/local/apache/domlogs/**********-bytes_log
httpd   5300 root   22w   REG  202,0     522 364110 /usr/local/apache/domlogs/***********
httpd   5300 root   23w   REG  202,0       0 364107 /usr/local/apache/domlogs/*******-bytes_log
httpd   5300 root   24w   REG  202,0       0 363896 /usr/local/apache/domlogs/********
httpd   5300 root   25w   REG  202,0       0 363895 /usr/local/apache/domlogs/*******-bytes_log
httpd   5300 root   26w   REG  202,0       0 361997 /usr/local/apache/domlogs/*******
httpd   5300 root   27w   REG  202,0       0 361958 /usr/local/apache/domlogs/*********-bytes_log
httpd   5300 root   28w   REG  202,0       0 319514 /usr/local/apache/logs/ssl_mutex (deleted)
httpd   5300 root   29r   CHR    1,9           5584 /dev/urandom
httpd   5300 root   31w  FIFO    0,8         339991 pipe
httpd   5300 root   32r  FIFO    0,8         339992 pipe

uname -a:

Linux ******** 2.6.38-x86_64 #1 SMP Thu Apr 28 22:18:47 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

Thank you for your help

[cPanelTristan]

Could you try temporarily switching to DSO to see if the memory usage drops significantly for these processes? I simply want to see if this is a PHP handler issue or Apache in general on the system. You can change the handler in WHM > Apache Configuration > PHP and SuExec Configuration area.

[Sannin]

Switching to DSO and back, did not reduce the memory usage. After the switch php stopped working . I have no output with php -i command. I am rebuilding Apache with basic profile plus some minor changes to see if it will help with memory problems and restore php.


Update: No change with newer settings, still 150 Mbs per process, php is working again.

[cPanelTristan]

When switching to DSO, if PHP sites ceased working entirely, you can simply switch back to suPHP to get them working again without recompiling Apache entirely unless the purpose was to recompile it to see if that would help with the memory consumption. Otherwise, a recompile wasn't needed.

I suggest hiring the services of someone to look over the machine and the sites. If your datacenter, NOC or provider might provide such services, since Apache shouldn't be using such high levels of memory on a machine for each process. Otherwise, we have a catalog of system administrators who provide services:

Dev & Sys Admin Services « Application Catalog

This isn't something that can really be determined via the forum, since it requires logging into the machine to check usage. Even if you add additional logging information, it won't provide any real details on what's happening with Apache. It would be better to simply bite the bullet and hire someone now to look over the machine.

[Sannin]

Something went seriously wrong when i switched to DSO. I couldn' t switch back to suphp because the fields were blank. I am not concerned about that as much as the memory issue.

I did some more research in MPM Prefork. It seems that when apache forks the resulting process shares a large amount of resources with the original process plus some overhead, and that is the reason all my apache processes use seem to use 150mbs ram. Something must have caused this, because i clearly remember that at my first apache builds, each process was at about 29mbs. A possible explanation is that it was built with MPM worker, but the default is MPM prefork and i never touched that setting before.

The problem is that i need to calculate the MaxClients setting based on the apache process memory size and at the moment i can't see how much it is exactly.

I was also thinking about hiring someone, but only for the initial optimization of the VPS.

[cPanelTristan]

Based on the recommendations that I'm seeing online for suPHP and which MPM to use, it likely would be better to have Worker for the MPM over using Prefork. There are some discussions about higher memory consumption with Prefork due to the fact that it launches more processes when there is a spike in traffic. Otherwise, you might try not selecting any MPM at all to have a comparison between using Prefork, using Worker and not having any MPM selected.

[Sannin]

Prefork is the default mpm in linux, and it will be compiled even if i wont select it in easyapache. I can do a test between worker and prefork though.

My current prefork settings are these:

<IfModule prefork.c>
MinSpareServers 3
MaxSpareServers 6
</IfModule>
ServerLimit 10
MaxClients 10
MaxRequestsPerChild 2000
KeepAlive On
KeepAliveTimeout 3
MaxKeepAliveRequests 200

I can't see the settings for worker in whm, so i 'll edit directly httpd.conf.

[Sannin]

Just for the record... I did a minimal centos installation on Virtualbox on my pc with the same amount of ram. I used a demo cpanel license and before i configure anything else, i run easyapache with the exact same profile as in my server.

The resulting original process was 8 mbs and the forked processes 3,3 mbs each.

I don' t know what else can i do to identify to problem, i have even an strace log but i can't make anything out of it. I believe the only solution is a clean reinstall on a new vps and transfer the cpanel accounts, judging from the results on VirtualBox.