SOLVED Apache crashing when using PHP-FPM

[AtariAge]

Greetings!

I've setup a new server recently and moved a fairly busy forum to it (4.5 million posts, typically 500-1000 users online in any 30 minute period). I'm running Invision 4.4.4, which is reasonably resource intensive. Database (MySQL 5.7) running on one dedicated server, Apache running on another dedicated server. No other sites are running on the web server, and the database server only supports this one site.

While optimizing the server, I tried switching from PHP-CGI to PHP-FPM. Upon initially doing this, Apache would go down pretty quickly, apparently due to exhausting MaxRequestWorkers. I increased MaxClients to 2500 and MaxRequestWorkers to 2000. I also bumped up the PHP-FPM Pool Options a bit (Max Requests to 200, Process Idle Timeout to 10 and Max Children to 10).

This seemed to resolve the problem, but after about 12 hours Apache stopped responding to requests again, and I didn't see any active PHP-FPM processes. Restarting apache didn't seem to help, and it wasn't until I switched back to PHP-CGI that the server started responding to requests again. Moving from PHP-CGI to PHP-FPM is nice, as it dropped the load from above 3.0 to around 1.0, so I'd like to use PHP-FPM if I can. I tried this a second time (switching back to PHP-FPM), but after another 12 hours or so, it went down again adn I had to switch back to PHP-CGI.

I could try increasing MaxClients and MaxRequestWorkers further, but I don't know if that would help--seems like I might just be putting a temporary band-aid on a deeper issue.

I looked through the logs the last time Apache went down (or at least hung) and saw tons of messages like this:

[Wed Jul 03 22:20:16.028372 2019] [proxy_fcgi:error] [pid 12645] (104)Connection reset by peer: [client 110.249.201.217:10762] AH01075: Error dispat
ching request to :
[Wed Jul 03 22:20:16.028452 2019] [proxy_fcgi:error] [pid 12652] [client 111.225.148.7:11198] AH01067: Failed to read FastCGI header
[Wed Jul 03 22:20:16.028491 2019] [proxy_fcgi:error] [pid 12652] (104)Connection reset by peer: [client 111.225.148.7:11198] AH01075: Error dispatch
ing request to :
[Wed Jul 03 22:20:16.028512 2019] [proxy_fcgi:error] [pid 12651] [client 125.209.235.178:37891] AH01067: Failed to read FastCGI header
[Wed Jul 03 22:20:16.028586 2019] [proxy_fcgi:error] [pid 12651] (104)Connection reset by peer: [client 125.209.235.178:37891] AH01075: Error dispat
ching request to :
[Wed Jul 03 22:20:16.028620 2019] [proxy_fcgi:error] [pid 12150] [client 110.249.202.82:63448] AH01067: Failed to read FastCGI header
[Wed Jul 03 22:20:16.028642 2019] [proxy_fcgi:error] [pid 12527] [client 54.36.148.120:38628] AH01067: Failed to read FastCGI header

Running PHP 7.2.19 and Apache 2.4.39. CENTOS 7.6 and WHM v80.0.20. Also running Redis on this server for caching. Machine has 32GB of RAM, and the site is running on a fast NVMe SSD. It's main task is just running the web server for this one site, plus email.

Thanks in advance for any insight and/or debugging advice.

 

[The Old Man]

Hi Atariage,

Yes, it's me from the IPS Community forums. PHP-FPM is definitely a good choice. I read earlier that PHP recommend using it with Apache MPM Prefork on their site, and letting PHP handle the multiple threads. I not been aware aware of any issues myself, but I have small sites.

I've spent a week trying to get PHP-FPM Status working with Apache 2.4 on CentOS 7.6 and it's so difficult but it might help you, especially once your IPS community has finished building its caches. Most of the tutorials online are unfortunately based on using it with Nginx, probably for a good reason, if only to save one's sanity. I really wish it was easier to get working as Netdata needs it for PHP-FPM monitoring.

As you know I've no experience with IPS communities as large as yours, but here's a few suggestions:

• You're probably already using InnoDB, but I'd definitely recommend 'tuning' MariaDB/Mysql, try using MySQLTuner, let the server settle and build its caches after a couple of days at least, then run it and consider the recommendations.
• make sure Opcache is installed and working.
• PHP 7.3 has some performance improvements over 7.2
• make sure your Apache caching is working efficiently well. Redbot is handy for checking http caching headers.
• Redis is excellent, but use a socket to connect instead of an IP4/6 address or localhost connection as it's noticeably nippier being a low level service.
• Check you have the Redis PHP extension installed.
• host as much of your theme, image and attachment content as possible on AWS S3 with Cloudfront which will take a large load off your server as well as speed up loading times for your visitors
• definitely use Cloudflare if you're not doing so already. Their Argo mode is also an effective option.
• Check out the awesome and free Netdata, so you can see what's happening live on your server. Demo sites. It's monitoring software with a tiny footprint, like Munin, but much more modern and improved. Actually it would make a great CPanel plug-in.

Hope this helps with a start.

 

[AtariAge]

Hello, and thanks for the informative response!

I made several changes to the server yesterday, and it seems to be running fine at the moment. At least it hasn't crashed yet. The load is higher than I would like, though, still sitting above 3.0 when the site is busier. Database server is fine, though, not much load on that.

To get PHP-FPM status working, I just modified the AtariAge.com virtual host in httpd.conf directly. I couldn't see any way to do this directly in WHM, and it also seems with EasyApache 4 they got rid of the distiller command to bake in any changes you might make to the httpd.conf. Which means, of course, they were overwritten when I changed Apache to use mpm_event before enabling HTTP/2 (mpm_prefork is apparently not compatible with HTTP/2).

Someone on another forum suggested increasing the PHP-FPM "Max Children" from 10 to 30, which I did. After hunting down the PHP-FPM log (logs, logs, everywhere!), I saw that it was indeed bumping into Max Children when I had it set to 10. I assume it hasn't hit that since I increased it to 30, or I would have known about it. I need to setup the PHP-FPM status page again. Here's what I added after the <FilesMatch \.(phtml|php[0-9]*)$> to define Set Handler for php-fpm:

       <LocationMatch "/(ping|status)">
       order deny,allow
       deny from all
       allow from xx.xx.xx.xx/22
       SetHandler proxy:unix:/opt/cpanel/ea-php72/root/usr/var/run/php-fpm/56919a6cd1265d9d836343a635dea4103fdfa6af7.sock|fcgi://atariage.com
       </LocationMatch>

It would be convenient if I could get that to stick so I when I make changes in EasyApache, this doesn't get clobbered.

I also created a static domain at static.atariage.com that I could use to serve static content (basically everything inside the ACP that you can point to a local directory). That seems to work well, although this morning I noticed a ton of errors:

[Fri Jul 05 11:48:55.291283 2019] [ssl:error] [pid 22560:tid 47352987125504] [client 207.161.204.215:60258] AH02032: Hostname atariage.com provided via SNI and hostname static.atariage.com provided via HTTP have no compatible SSL setup, referer: https://atariage.com/forums/
[Fri Jul 05 11:48:55.294036 2019] [ssl:error] [pid 22560:tid 47352987125504] [client 207.161.204.215:60258] AH02032: Hostname atariage.com provided via SNI and hostname static.atariage.com provided via HTTP have no compatible SSL setup, referer: https://atariage.com/forums/
[Fri Jul 05 11:48:55.295782 2019] [ssl:error] [pid 22560:tid 47352987125504] [client 207.161.204.215:60258] AH02032: Hostname atariage.com provided via SNI and hostname static.atariage.com provided via HTTP have no compatible SSL setup, referer: https://atariage.com/forums/
[Fri Jul 05 11:48:55.298311 2019] [ssl:error] [pid 22560:tid 47352987125504] [client 207.161.204.215:60258] AH02032: Hostname atariage.com provided via SNI and hostname static.atariage.com provided via HTTP have no compatible SSL setup, referer: https://atariage.com/forums/
[Fri Jul 05 11:48:55.298523 2019] [ssl:error] [pid 22560:tid 47352991328000] [client 207.161.204.215:60258] AH02032: Hostname atariage.com provided via SNI and hostname static.atariage.com provided via HTTP have no compatible SSL setup, referer: https://atariage.com/forums/
[Fri Jul 05 11:48:55.300873 2019] [ssl:error] [pid 22560:tid 47352991328000] [client 207.161.204.215:60258] AH02032: Hostname atariage.com provided via SNI and hostname static.atariage.com provided via HTTP have no compatible SSL setup, referer: https://atariage.com/forums/

I haven't looked into this yet, but will do so shortly. I created static.atariage.com as a new A record, pointed it to the IP for the site, and then added the domain in cPanel for the domain, pointing at the site root (atariage.com). From the client side it seems to work well, but obviously I want to fix this ssl issue.

Regarding your suggestions,

• You're probably already using InnoDB, but I'd definitely recommend 'tuning' MariaDB/Mysql, try using MySQLTuner, let the server settle and build its caches after a couple of days at least, then run it and consider the recommendations.
• make sure Opcache is installed and working.
• PHP 7.3 has some performance improvements over 7.2
• make sure your Apache caching is working efficiently well. Redbot is handy for checking http caching headers.
• Redis is excellent, but use a socket to connect instead of an IP4/6 address or localhost connection as it's noticeably nippier being a low level service.
• Check you have the Redis PHP extension installed.
• host as much of your theme, image and attachment content as possible on AWS S3 with Cloudfront which will take a large load off your server as well as speed up loading times for your visitors
• definitely use Cloudflare if you're not doing so already. Their Argo mode is also an effective option.
• Check out the awesome and free Netdata, so you can see what's happening live on your server. Demo sites. It's monitoring software with a tiny footprint, like Munin, but much more modern and improved. Actually it would make a great CPanel plug-in.

I am using InnoDB, which I actually setup on my 3.4.x forum at some point. Invision did discover that some of the tables (in particular, large tables like posts, messages, topics) were somehow MyISAM tables. This must have happened during the upgrade, since it creates new tables. However, the default-storage-engine is set to InnoDB, so no idea how that happened. I immediately converted them when they informed me about this a few days ago.

I am running Opcache.

I have not yet setup PHP 7.3, as there was one bug in particular that was affecting other admins. it's been patched in PHP, but I'm not sure there's a production version available, much less available via EasyApache. I'll move to 7.3 as soon as I know this has been resolved (there's a thread about it on the Invision forum).

I do have Redis running, and it looks like via TCP instead of a socket. I'll look into changing that, thanks. I do have the PHP extension installed and it's configured in the ACP.

I still have to investigate using Cloudfront, or Wasabi and Cloudflare, as described here. No idea how much that will cost yet, as it's a fairly busy site.

That Netdata demo is quite sexy, I definitely need to set that up! Thanks for the links. You're right, that would be a great cPanel plugin.

Thanks again for your post, much appreciated!

..Al

 

[AtariAge]

Just a followup to this thread. Since making the changes to the PHP-FPM "Max Children" from 10 to 30, the server has been stable and I haven't seen Apache crash (or hang).

I need to try running mysqltuner again. I do have it installed, but last time I tried to run it it ended up hanging and caused the database server to become unresponsive. I'm not sure why this would happen, as I've used this tool many times in the past without issues. Need to try it again late at night, take the forum offline, and see what happens.

..Al

 

[cPanelMichael]

Hello @AtariAge,

Thanks for the follow-up. Feel free to open a new thread if you continue to experience issues running the MySQL tuning script and we can help investigate.

Thank you.