php-fpm / PHP 5.6.28 / defunct processes

Did cPanel change something with the way php-fpm works with the PHP 5.6.28 update? Or perhaps PHP did, although I'm not seeing anything in PHP's changelog concerning this.

Is anybody else seeing issues with PHP-FPM after updating to PHP 5.6.28? Am I the only one still using PHP 5.6? I am using cPanel 56, perhaps I'm the only one using PHP-FPM with PHP 5.6 on cPanel 56?

It's like the php-fpm processes aren't properly closing. They just build up and become defunct. It's not uncommon for me to have over 100 php-fpm processes running when they should be between 10 and 20.

Since I'm a bit limited with time, my solution right now is downgrading down to PHP 5.6.27 which causes the problem to go away. I'm just curious if anyone else is seeing this.

 

The only way I have been able to duplicate this is on a production server, where PHP is actually used. Low usage or test servers (because of their lack of real-world access) aren't able to duplicate this.

I'm just really puzzled that nobody else has seen this. I guess it's something that I have done. But I can't for the life of me figure it out.

I ran the numbers on a couple of different servers. To calculate the number php-fpm processes, I used the command:

ps aux | grep "php-fpm.*-php56" | grep -v ^root | wc -l

collecting data every two minutes.

server1:
With PHP 5.6.27 - average number of php-fpm processes is 15 (130 samples)
With PHP 5.6.28 - average number of php-fpm processes is 35 (618 samples)

server2:
With PHP 5.6.27 - average number of php-fpm processes is 18 (602 samples)
With PHP 5.6.28 - average number of php-fpm processes is 92 (95 samples)

Granted, the sample sizes for each version of PHP are off by quite a bit, but I'm just not seeing anything that leads me to believe collecting more samples is going to substantially change the average number of php-fpm proceses.

I do see where one of the patches being applied to cPanel's PHP has to do with php-fpm and ondemand, the php-5.6.26.ondemand_fork.patch patch. Although I'm not seeing where this has any bearings between PHP 5.6.27 and PHP 5.6.28. But I do note that the php-5.6.26.ondemand_fork.patch is stated: "This patch is a major architectural change and is likely to be fragile and require much tender loving care."

I'm also wondering if this might have something to do with cPanel 56. Perhaps cPanel's PHP 5.6.28 was built solely for cPanel 60? Although I'm not seeing where any changes were made between PHP 5.6.27 and PHP 5.6.28 that would concern this. I believe cPanel 60 contains some fundamental changes to the way PHP-FPM operates with cPanel? I haven't really had time to look into all of that. Our plan was to move our production servers to cPanel 58, but this issue with the PHP 5.6.28 update didn't allow us to push out the cPanel 58 update as planned.

Nobody else is seeing this issue?

Anybody else running cPanel's PHP 5.6.28 on their servers with PHP-FPM and ondemand process manager? How many php-fpm processes are you seeing?

ps aux | grep "php-fpm.*-php56" | grep -v ^root | wc -l

How does this compare to what you were seeing with PHP 5.6.27?



I have figured out how to show this issue and duplicate it.

Take any account that is using PHP 5.6.28 with PHP-FPM and using ondemand process management, probably one that is a test account that doesn't get any legitimate hits (if someone else access the account while performing this test it may throw the results off).

1) Create a PHP file on that account, for my purposes I created a phpinfo() page named phpinfo.php

2) Visit that URL - i.e. http://example.tld/phpinfo.php

3) Now immediately on the server run the command:

Code:

ps aux | grep 'php-fpm: pool %user%-php56' | grep -v ^root

(replace %user% with the owner of example.tld, the pool name that example.tld is using)

You should see 1 entry:

Code:

%user%  7871  0.0  0.0 337340 10892 ?  S  15:02  0:00 php-fpm: pool %user%-php56

4) Wait for that entry to expire (This would be the pm.process_idle_timeout value for this pool in /opt/cpanel/ea-php56/root/etc/php-fpm.d/%user%.conf - I have mine set to 10 seconds, yours may be different)

5) After that has expired (i.e. when ps aux | grep 'php-fpm: pool %user%-php56' | grep -v ^root returns nothing), visit that PHP URL again - i.e. http://example.tld/phpinfo.php

6) And immediately run:

Code:

ps aux | grep 'php-fpm: pool %user%-php56' | grep -v ^root

This time you'll get more than one entry:

Code:

%user% 15052  0.0  0.0 337340 10928 ?  S  15:20  0:00 php-fpm: pool %user%-php56
%user% 15053  0.0  0.0 337340  8016 ?  S  15:20  0:00 php-fpm: pool %user%-php56
%user% 15054  0.0  0.0 337340  8016 ?  S  15:20  0:00 php-fpm: pool %user%-php56
%user% 15055  0.0  0.0 337340  8016 ?  S  15:20  0:00 php-fpm: pool %user%-php56
%user% 15056  0.0  0.0 337340  8016 ?  S  15:20  0:00 php-fpm: pool %user%-php56
%user% 15057  0.0  0.0 337340  8016 ?  S  15:20  0:00 php-fpm: pool %user%-php56
%user% 15058  0.0  0.0 337340  8020 ?  S  15:20  0:00 php-fpm: pool %user%-php56

Why? I don't know, but that's the issue. In a production server, where many sites are hosting PHP scripts, getting multiple hits these 7 php-fpm pool processes become more and more and quickly start to add up. It's like its ignoring the ondemand process management.

Downgrade to PHP 5.6.27 and retry all of this, and all of this will work as it should. When you get to step 6 with PHP 5.6.27 only 1 process is returned - as expected. Why are multiple php-fpm pool processes being started in PHP 5.6.28?

Worth Noting: I'm basically following the PHP-FPM instructions from Configure or Disable PHP-FPM with User Pools for EasyApache 4 - cPanel Knowledge Base - cPanel Documentation as I am using cPanel 56 (also seeing this behavior in cPanel 58). Perhaps some of this changed in cPanel 60, I do not know. I don't know how cPanel 60 operates PHP-FPM pools, are they still named %user%-php56? Is configuration for these still maintained in /opt/cpanel/ea-php56/root/etc/php-fpm.d/%user%.conf I suspect cPanel 60 will show the same behavior.

 

I've had a bit of time to look into this a bit more. It seems the issue came about in 5.6.27-2.2.1 it is not present in 5.6.27-1.1.2. This is when the php-5.6.26.ondemand_fork.patch patch was added.

Does this patch create more problems than it solves?

php-5.6.26.ondemand_fork.patch is suppose to fix the bug at PHP :: Bug #69724 :: pm.ondemand forks fewer child workers than it should but I note that that bug is from 2015 and for PHP 5.5. The test for this bug as mentioned at GitHub - bjaspan/goofy: Goofy is an HTTP load testing tool that simulates waves of surfers in a somewhat unusual way says that the experiment was conducted in 2014.

I'm just wondering if this is still a bug in PHP 5.6 and/or if this php-5.6.26.ondemand_fork.patch patch creates more problems than it solves.