Apache childs eating too much memory

[fasdush]

Hello.

We running cPanel 11.15.0-R17853 on CentOS 5 x86_64, Apache 1.3.37 /w mod_fastcgi and ~1000 virtual hosts.

Recently this server began to regulary goes out of memory. The reason are apache childs, ~40mb RSS each.

I`ve tried to recompile Apache with /scripts/easyapache, it get compiled succesfully, but failed during config update (our httpd.conf is heavily self-modified). I was able to replace libs/modules and binaries from 1.3.39 -- memory usage remaines the same. I`ve tried to disable most of the modules, but that gave me just 1-2mb decrease for each process.

There is another server with just the same platform/OS/cPanel/Apache/config -- it uses ten times less memory per child (around 4mb, which is normal for httpd without mod_cgi/mod_php).

Why Apache eating so much memory? What should I do? I do not want to migrate for 2.0 branch for now, because server is heavilly populated.

Sorry for bad english, I`m from Russia.

---------------------------------
# /usr/local/apache/bin/httpd -V
Server version: Apache/1.3.39 (Unix)
Server built: Nov 8 2007 11:48:51
Cpanel::Easy::Apache v3.2.0 rev3044
Server's Module Magic Number: 19990320:18
Server compiled with....
-D EAPI
-D HAVE_MMAP
-D HAVE_SHMGET
-D USE_SHMGET_SCOREBOARD
-D USE_MMAP_FILES
-D HAVE_FCNTL_SERIALIZED_ACCEPT
-D HAVE_SYSVSEM_SERIALIZED_ACCEPT
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D DYNAMIC_MODULE_LIMIT=64
-D HARD_SERVER_LIMIT=256
-D HTTPD_ROOT="/usr/local/apache"
-D SUEXEC_BIN="/usr/local/apache/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/httpd.scoreboard"
-D DEFAULT_LOCKFILE="logs/httpd.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
-D ACCESS_CONFIG_FILE="conf/access.conf"
-D RESOURCE_CONFIG_FILE="conf/srm.conf"
---------------------------------
# /usr/local/apache/bin/httpd -l
Compiled-in modules:
http_core.c
mod_env.c
mod_log_config.c
mod_mime.c
mod_negotiation.c
mod_status.c
mod_include.c
mod_autoindex.c
mod_dir.c
mod_asis.c
mod_imap.c
mod_actions.c
mod_userdir.c
mod_alias.c
mod_rewrite.c
mod_access.c
mod_auth.c
mod_expires.c
mod_headers.c
mod_unique_id.c
mod_so.c
mod_setenvif.c
mod_ssl.c
suexec: enabled; valid wrapper /usr/local/apache/bin/suexec
---------------------------------
# egrep ^"(Load|Add)Module" /etc/httpd/conf/httpd.conf
LoadModule bwlimited_module libexec/mod_bwlimited.so
LoadModule bytes_log_module libexec/mod_log_bytes.so
LoadModule auth_passthrough_module libexec/mod_auth_passthrough.so
LoadModule fastcgi_module libexec/mod_fastcgi.so
AddModule mod_env.c
AddModule mod_log_config.c
AddModule mod_mime.c
AddModule mod_include.c
AddModule mod_dir.c
AddModule mod_asis.c
AddModule mod_actions.c
AddModule mod_alias.c
AddModule mod_rewrite.c
AddModule mod_access.c
AddModule mod_auth.c
AddModule mod_expires.c
AddModule mod_so.c
AddModule mod_bwlimited.c
AddModule mod_log_bytes.c
AddModule mod_auth_passthrough.c
AddModule mod_fastcgi.c
---------------------------------
# ps -U nobody -o rss,vsize,size,cmd|head
RSS VSZ SZ CMD
42248 68484 42652 /usr/local/apache/bin/httpd -DSSL
42028 68416 42600 /usr/local/apache/bin/httpd -DSSL
42428 68616 42800 /usr/local/apache/bin/httpd -DSSL
42468 68632 42816 /usr/local/apache/bin/httpd -DSSL
42436 68716 42820 /usr/local/apache/bin/httpd -DSSL
42500 68672 42856 /usr/local/apache/bin/httpd -DSSL
42172 68504 42688 /usr/local/apache/bin/httpd -DSSL
42432 68596 42780 /usr/local/apache/bin/httpd -DSSL
42392 68600 42784 /usr/local/apache/bin/httpd -DSSL
---------------------------------

[fasdush]

Just found thta if I comment out, say, half of VirtualHost entries the memory usage per child drops (to 25mb in case of half entries). But why that matters? Most of my VirtaulHost entries looks like this:
-----------------------------------
<VirtualHost x.x.x.x:80>
ServerAlias www.server.com
ServerAdmin webmaster@server.com
DocumentRoot /home/user/public_html/

ServerName server.com

#mod_userdir is not used
<IfModule mod_userdir.c>
Userdir disabled
Userdir enabled user
</IfModule>

#mod_php is not used
<IfModule mod_php4.c>
php_admin_value open_basedir "/home/user:/usr/lib/php:/usr/local/lib/php:/tmp"
</IfModule>
<IfModule mod_php5.c>
php_admin_value open_basedir "/home/user:/usr/lib/php:/usr/local/lib/php:/tmp"
</IfModule>

User user
Group user
CustomLog /usr/local/apache/domlogs/server.com combined
ScriptAlias /cgi-bin/ /home/user/public_html/cgi-bin/
</VirtualHost>
-----------------------------------