Server down due to fcgid

[racso]

Hi, I have a problem with my server, when we have a traffic spike the server goes down almost immediately, last time I checked google analytics real time we had 800 people connected and the server stopped responding.

The server is Quad Opteron x8-6128 2000Mhz - 18GB Ram - 4 Physical CPUs - 32 Total Cores - CENTOS 5.9 x86_64 xenhvm

Checking the logs I found that during every spike load the log was FULL of these lines

• [warn] [client ip.ip.ip.ip] mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5, referer: http://externaldomain.com
• Connection reset by peer: mod_fcgid: error reading data

And then this error started to show up

• server reached MaxClient settings, consider raising the MaxClients setting

It seems ever single new connection got a fcgid error and that caused the downtime, could somebody help me to understand whats happening ? my server goes down from time to time


php -v

PHP 5.2.17 (cli) (built: May 16 2013 00:27:20) 
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
    with the ionCube PHP Loader v4.2.2, Copyright (c) 2002-2012, by ionCube Ltd.

/usr/local/cpanel/bin/rebuild_phpconf --current

Available handlers: suphp fcgi cgi none
DEFAULT PHP: 5
PHP4 SAPI: none
PHP5 SAPI: fcgi
SUEXEC: enabled
RUID2: not installed

Home »Service Configuration »Apache Configuration »Include Editor
Pre Main Include

<IfModule worker.c>
ThreadLimit 25
StartServers 2
MaxClients 400
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

Pre VirtualHost Include

<IfModule mod_fcgid.c>
FcgidMaxRequestLen 1073741824
FcgidMaxRequestsPerProcess 1500
FcgidMinProcessesPerClass 0
FcgidMaxProcessesPerClass 120
FcgidConnectTimeout 500
FcgidIOTimeout 1200
FcgidFixPathinfo 1
FcgidIdleTimeout 900
FcgidIdleScanInterval 120
FcgidBusyTimeout 2600
FcgidBusyScanInterval 120
FcgidErrorScanInterval 9
FcgidZombieScanInterval 9
FcgidProcessLifeTime 3600
</IfModule>
<IfModule mod_evasive20.c>
DOSHashTableSize    6151
DOSPageCount        4
DOSSiteCount        100
DOSPageInterval     1
DOSSiteInterval     1
DOSBlockingPeriod   10
</IfModule>
#MaxRequestsPerProcess 1000
#DefaultMaxClassProcessCount 120
#IdleTimeout 60
#MaxProcessCount 2000
#IPCCommTimeout 40
#IPCConnectTimeout 10
#MaxRequestLen 10240000

Post VirtualHost Include

Include /usr/local/lp/configs/httpd/conf.d/*.conf

httpd -l

 core.c
  mod_authn_file.c
  mod_authn_default.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_default.c
  mod_auth_basic.c
  mod_include.c
  mod_filter.c
  mod_deflate.c
  mod_log_config.c
  mod_logio.c
  mod_env.c
  mod_expires.c
  mod_headers.c
  mod_unique_id.c
  mod_setenvif.c
  mod_version.c
  mod_proxy.c
  mod_proxy_connect.c
  mod_proxy_ftp.c
  mod_proxy_http.c
  mod_proxy_scgi.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  mod_ssl.c
  worker.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_info.c
  mod_suexec.c
  mod_cgid.c
  mod_negotiation.c
  mod_dir.c
  mod_actions.c
  mod_userdir.c
  mod_alias.c
  mod_rewrite.c
  mod_so.c

Thank you all!

 

[racso]

Help please

 

[thinkbot]

You should also copy "top" command result executed during spike to see which services takes the most cpu
Please also post /etc/my.cnf contents and mysqltuner result of mysql

Do you have apc or eaccelerator (opcode cache) in php ?

 

[cPanelMichael]

Hello

It's important to keep in mind that cPanel does not recommend an FCGI configuration for your PHP handler. FCGI requires fine tuning of mod_fcgid to ensure that the server does not become overloaded with idle PHP processes. It's typically noted that those without the experience or expertise in fine tuning FCGI should consider an alternate handler for PHP.

Thank you.

 

[racso]

Thank you cpanelmichael, maybe this is the problem with my server because crash with a minimal spike load could you help me to know what must I do in order to resolve this ? I can ask it to my hosting provider (liquid web) who configured this.

THank you

 

[cPanelMichael]

You can check the following log files when your server shuts down to see if there are any specific messages related to the shutdown:

/var/log/dmesg
/var/log/messages

Thank you.