PHP FPM not optimal for shared hosting

[walterdevos]

Hi there

A few versions back FPM was introduced and we blindly activated the feature, as usually whm/cpanel works fine out of the box.
This feature however reacts unexpected in a shared hosting setup.

It seems FPM is meant to run one site (one user). So if we use Cpanel (out of the box) for multiple sites, every client (every site) gets it's own pool and if we activate FPM with max_children=20 , every client can expand to 20 processes. If they in a rare case all do so at the same time, the server will run out of memory (and oom kicks in).

The Internet suggests that I should experiment, give some sites like 5 processes, others 2 or put some on "pm=demand" ( not dynamic ) and look at the logs for "you should increase number of childs" to further tune the setup... Is that really the only way?

I found a hint to something like max.processes. This would globally cap the max processes ALL Pools combined. But I can't figure out where to put it. Every configfile is telling me that I should use the "interface".

I could also make all sites run as one user, but I don't think cpanel will support that.

If you know what I can do?

Edit: toned it down a bit

 

[ffeingol]

If you don't like FPM, just switch to a different PHP handler. PHP Handlers | cPanel & WHM Documentation There are obvious trade-offs to switching, but it can be done.

 

[walterdevos]

Hi Again.
I solved it myself. Here is what I found out & what I did.

The PHP-FPM will spawn pool processes for every .conf file in the /opt/cpanel/ea-php{php_version}/root/etc/php-fpm.d directory. These are the domains for which you have php-fpm set to "on" in the user interface under "MultiPHP Manger".
Each domain will basically get the amount of "max children" ( standard 5 ) set in the "System PHP-FPM" tab. Besides the defaults for all domains, you can individually change the settings using the "Edit PHP-FPM" link for each domain.

However, this configuration either sets you for a crash, or leaves many of the server's resources unused. When you set the max children to high, and the sum of all max children of all domains combined is more that your server can handle. It might run smooth, until many domains simultaneously demand more processes. When that happens , you run out of memory and OOM will kill your server, downtime for every client on the server. When you put a number too small, sites will stop working if they reach only a fraction of the load the server could potentially handle.

I also found that when a domain reaches it's max children, the domain becomes unresponsive ( ERR Timeout ). This might be another issue altogether, but it also suggest to raise the max children.

Either way, you're in a bad place.

Here is the list of global php-fpm.conf directives

PHP: Configuration - Manual

www.php.net

One of them is : process.max int
The maximum number of processes FPM will fork. This has been design to control the global number of processes when using dynamic PM within a lot of pools. Use it with caution. Default value: 0.

The WHM user interface does not allow you to change these Global options.
You are also not allowed to manually change the php-fpm.conf file, because it is auto generated.

The source of the configuration for creating the config file is stored in yaml files here:
/var/cpanel/ApachePHPFPM

There will be a system_pool_defaults.yaml , that will contain the setting you entered in the user interface. So no need to change this.
We need system.yaml , This file contains the global settings. In my setup it was missing, so I created it with touch :
touch /var/cpanel/ApachePHPFPM/system.yaml

I then edited the file, so it contained this:

---
process_max: 35

And finally, rebuild the configuration with this command:
/scripts/php_fpm_config --rebuild

After this, the new directive ( Process.max ) was automatically added to the php-fpm.conf by the build process.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; cPanel FPM Configuration ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; NOTICE This file is generated. Please use our WHM User Interface
; to set these values.

include=/opt/cpanel/ea-php72/root/etc/php-fpm.d/*.conf

;;;;;;;;;;;;;;;;;;
; Global Options ;
;;;;;;;;;;;;;;;;;;

[global]
daemonize = no
emergency_restart_interval = 0
emergency_restart_threshold = 0
error_log = /opt/cpanel/ea-php72/root/usr/var/log/php-fpm/error.log
log_level = notice
pid = /opt/cpanel/ea-php72/root/usr/var/run/php-fpm/php-fpm.pid
process_control_timeout = 10
process.max = 35

Still some issues I need to resolve.
Somehow PHP-FPM keeps processes running iddle, but they don't accept new request. Sites go dark after the max children has been reached. I solved this by setting process iddle timeout to 5 ( from 150 ).
If someone might have a clue why this happens?


In conclusion, it might be something CPanel could consider integrating, as I found many issues relating to this. And no documentation for FPM that I could find talks about FPM in a shared hosting situation.

Thanks for your time.

 

[kdean]

Note, the php-fpm.conf file you reference isn't global unless PHP 7.2 is the only version installed.

process.max you mentioned is for use with the dynamic process manager. Cpanel defaults to the ondemand on to use less processes / memory so I do not believe that setting will do anything for you.

Adjusting the idle timeout down is good for your needs since with the previous setting of 150 seconds that left a lot of time keeping a processing waiting for further use and if you have a really active site that could grow the number of processes a lot. So a shorter timeout allows them a better chance to close.

max_requests per process is another way to keep the memory use per process in check for processes that see enough activity to never close. This will make sure they close even if the idle timeout hasn't passed.

Max children may not need a high setting if your sites make use of any caching that delivers cached html pages rather than executing php pages. I have a bunch site that get away with 5 and I haven't set any past 15. You'll just need to set a value and watch the process use.

 

[walterdevos]

@kdean Thanks for your reply.

Indeed. There is one older version of php running. This will be phased out this week. Could that cause the instabilities?

Well , the max_processes setting does trigger when the max processes are reached (saw it once when i stress tested the setup) and it puts a line in the logs , but it also crashes the webserver.

Same goes for max children (per domain). If I get the message "server reached max_children setting (15), consider raising it" also the domain is completely down (times out). I have to reboot the server to get it back. Restart fpm alone doesn't fix. Sometimes restarting fpm AND apache helps.

Most sites work good with 5 and never use more than 2 children. Just that one site is having 5+ all the time.

It's very frustrating to have FPM freeze unexpectedly. And there isn't much to go on, because nobody is talking about using PHP-FPM in a shared environment. The only things I could find were people with similar problems when using it in Cpanel. Lot and lot's of manuals about using it elsewhere, but always servers that serve only one application (or one website), never multiple users. I'm not sure what to think or why Cpanel put it in here when it's not supposed to work this way?

I will try changing OnDemand to Dynamic though (like you pointed), maybe my setting isn't supported and untested.
edit: that was a bad idea XD

If someone can shed a light on why the Apache or FPM is not serving sites anymore after max_children has been reached, it would be great. I would hate it to have to go back to mod_php for stability.

 

[Michael-Inet]

PHP-FPM has always been a bug in my side with cPanel, I was asking much of these same questions 3-4 years ago. Yeah, it’s the new hotness, but cPanel has never really provided any decent documentation on how it either works or how to configure it. And you’ve just verified again that they still knowingly do not providing access to all of its settings.

I’ve not hit any of Walter’s problems, but once it became default, RAM became cheap enough I just doubled the amount of RAM I normally used for a server and did a swag of a double amount of Max Children any site on the box will ever hit and then used that multiplied by the number of sites on the box to set Max Requests. Which is completely stupid, but works okay enough I don’t ever get sites going dark.

Interestingly in CENTOS 7.8, WHM v90.0.15 Process Idle Timeout’s default is now 10.

That seems to suggest cPanel is aware of the crashes the webserver issues...

Frustrated too,
Michael

 

[ffeingol]

As I said in my reply, you can switch handlers. We turn off PHP-FPM and are typically using LSAPI.

 

[cPanelLauren]

This thread has a very good discussion on handlers, my personal preference is LSAPI but PHP-FPM is also suitable for many instances.

@walterdevos I'm glad to see you were able to resolve your issue, please let us know if you have any further issues.

 

[sparek-3]

What do you have set for the pm.process_idle_timeout parameter?

ondemand should work. If you're using ondemand and you run up against max_children and assuming you have a reasonable value for max_children (i.e. not 1 or 2) then I might be a little concerned that the websites themselves are just getting flooded with requests. This can happen a lot with various search engine crawling bots that don't really adhere to any form of rate limiting themselves (and are usually some less that recognizable search engine bots).

 

[Michael-Inet]

@cPanelLaurne,

Where did @walterdevos say his issue was solved? Even before his last edit he clearly states, “but it also crashes the webserver” and “If someone can shed a light on why the Apache or FPM is not serving sites anymore after max_children has been reached”

# # #

Hi All,

For a secure, shared environment (e.g. more than one cPanel account per server) we’re left with FCGI and PHP-FPM. Hence this discussion on how to get FPM to work without fubar-ing websites...

Here’s background to continue this discussion (from [1]):

Note: If I missed a PHP handler that can be installed in WHM/cPanel, please add it.


Usable in shared environment
FCGI
The primary benefit for FastCGI is the ability to be used with SuEXEC like the handler SuPHP. This allows scripts to be processed/executed as the user instead of the default nobody4) user that Apache uses.
With EasyApache4, the FCGI handler can be applied to all versions.
@[2] The mod_fcgid Apache module is not compatible with the mod_ruid2 or mod_mpm_itk Apache modules.


PHP-FPM
Requires in-depth configured
Can cause performance issues if not configured properly
Requires decent amount of server resources available
Not meant for low-end servers


Not sure if usable without CloudLinux?
LSAPI / mod_lsapi
CloudLinux is required to take full advantage of LSAPI with mod_lsapi PRO.
In a Multi-PHP Environment, the LSAPI handler can be applied to all versions.
Improper file permissions can cause a world-readable situation that introduces security risks across websites.

@[2]
cPanel & WHM's version of LSAPI differs from the one that CloudLinux provides. This version disables Checkpoint/Restore in Userspace (CRIU), connection pooling, and opcache preservation.
Note:
To utilize this handler's full functionality, your system must run CloudLinux. For more information, read Cloudlinux's Apache mod_lsapi documentation.


Not generally considered usable in shared environment
CGI
This handler is considered insecure due to non-restrictive permissions
Apache directives cannot be used. Configuration file (php.ini) required

SuPHP
Unable to use any PHP Opcode Caching
No longer actively maintained by it's developers

DSO
In a Multi-PHP Environment (EasyApache 4 or using multiple versions of PHP with DirectAdmin's CustomBuild) the DSO+mod_ruid2 handler can only be applied to one PHP version – DSO cannot be used for multiple versions of PHP at the same time.

Reference:
[1] PHP Handlers [KnownHost Wiki]
[2] PHP Handlers - EasyApache 4 - cPanel Documentation

# # #

@walterdevos,

You’ve done the most (recent) testing with FPM, would you mind filing a bug report with cPanel detailing how to replicate the issues we all have experienced at one point or another?

Buying double RAM isn’t anyone’s long term preferred solution...

Thanks,
Michael

Edit:
Additional References:

PHP :: Bug #74709 :: PHP-FPM process eating 100% CPU attempting to use kill_all_lockers

bugs.php.net

Setting a cronjob to reset PHP-FPM

Our shared servers seem to gradually build up CPU use until we reset PHP-FPM. It then comes down right away. I was thinking of running a weekly or maybe even a daily cronjob to reset it. Do you think this could have any negative precautions? And what would be the best cron job to do this say...

forums.cpanel.net

 

[Michael-Inet]

What do you have set for the pm.process_idle_timeout parameter?

WHM »Home »Software »MultiPHP Manager
Tab: System PHP-FPM Configuration

Edit: I read that as 'Where do you set' opps...

Here's the right answer:

Somehow PHP-FPM keeps processes running iddle, but they don't accept new request. Sites go dark after the max children has been reached. I solved this by setting process iddle timeout to 5 ( from 150 ).

 

[cPanelLauren]

@Michael-Inet

@cPanelLaurne,

Where did @walterdevos say his issue was solved? Even before his last edit he clearly states, “but it also crashes the webserver” and “If someone can shed a light on why the Apache or FPM is not serving sites anymore after max_children has been reached”

It says that here:

Hi Again.
I solved it myself. Here is what I found out & what I did.

The PHP-FPM will spawn pool processes for every .conf file in the /opt/cpanel/ea-php{php_version}/root/etc/php-fpm.d directory. These are the domains for which you have php-fpm set to "on" in the user interface under "MultiPHP Manger".
Each domain will basically get the amount of "max children" ( standard 5 ) set in the "System PHP-FPM" tab. Besides the defaults for all domains, you can individually change the settings using the "Edit PHP-FPM" link for each domain.

The reason for the crash I assumed in this instance was due to reaching the max workers which would actually be normal when pool limits are reached.

But, @walterdevos If you're thinking this issue is related to PHP :: Bug #74709 :: PHP-FPM process eating 100% CPU attempting to use kill_all_lockers I would correct suggestions to open a case and say that you should not open a ticket to cPanel for this as this is a known issue with PHP-FPM which is specifically related to PHP-FPM and while we are aware it exists it is not something we can fix.

The problem specifically being that kill_all_lockers is attempting to kill processes that belong to a different FPM pool. This from every instance I've seen is only occurring when opcache is enabled as well. The workaround that is suggested for this currently is to add kill_all_lockers to disable functions or alternatively disable/remove opcache. The latter not being an option many are able to do. I've discussed this issue on the forums in multiple other threads, the most memorable for me at the time being Remove auto generation htaccess AddHandler application/x-httpd

 

[Michael-Inet]

Pasting more notes:

You must configure Apache to allow enough server instances to handle the maximum number of PHP-FPM child processes.
Set the Start Servers option in WHM’s Global Configuration interface (WHM >> Home >> Service Configuration >> Apache Configuration >> Global Configuration) to a value greater than or equal to the pm_max_children setting in WHM’s MultiPHP Manager interface (WHM >> Home >> Software >> MultiPHP Manager).
From:

Apache vs. PHP-FPM Bottleneck with Child Processes | cPanel & WHM Documentation

Apache servers that use the `prefork` MPM with fewer server instances than the maximum number of PHP-FPM child processes may experience performance issues. This document explains how to resolve this dependency issue.

docs.cpanel.net

Home >> Service Configuration >> Apache Configuration
>> Global Configuration
5 : Start Servers (must be >= than Max Children below)

Home >> Software >> MultiPHP Manager
>> System PHP-FPM Configuration
PHP-FPM Configuration
PHP-FPM Pool Options
20 : Max Requests
pm.max_requests int
The number of requests each child process should execute before respawning. This can be useful to work around memory leaks in 3rd party libraries. For endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS. Default value: 0.

10 : Process Idle Timeout
pm.process_idle_timeout mixed
The number of seconds after which an idle process will be killed. Used only when pm is set to ondemand. Available units: s(econds)(default), m(inutes), h(ours), or d(ays). Default value: 10s.

5 : Max Children
pm.max_children int
The number of child processes to be created when pm is set to static and the maximum number of child processes to be created when pm is set to dynamic. This option is mandatory.
This option sets the limit on the number of simultaneous requests that will be served. Equivalent to the ApacheMaxClients directive with mpm_prefork and to the PHP_FCGI_CHILDREN environment variable in the original PHP FastCGI.


Other settings for FPM:
process.max int
The maximum number of processes FPM will fork. This has been design to control the global number of processes when using dynamic PM within a lot of pools. Use it with caution. Default value: 0.

emergency_restart_threshold int
If this number of child processes exit with SIGSEGV or SIGBUS within the time interval set by emergency_restart_interval, then FPM will restart. A value of 0 means 'Off'. Default value: 0 (Off).

emergency_restart_interval mixed
Interval of time used by emergency_restart_interval to determine when a graceful restart will be initiated. This can be useful to work around accidental corruptions in an accelerator's shared memory. Available Units: s(econds), m(inutes), h(ours), or d(ays). Default Unit: seconds. Default value: 0 (Off).

process_control_timeout mixed
Time limit for child processes to wait for a reaction on signals from master. Available units: s(econds), m(inutes), h(ours), or d(ays) Default Unit: seconds. Default value: 0.

From:

PHP: Configuration - Manual

www.php.net

# # #

@cPanelLauren,

Thanks for the reply!

Would you happen to know where to set the 3 settings (above: emergency_restart_threshold, eri, pct) so they are persistent?

If I'm wrapping my head around all three, then all three combined might mitigate the kill_all_lockers issue. Or if nothing else, worth a try seeing if they'll mitigate
@walterdevos 's "not serving sites anymore after max_children has been reached."

Best,
Michael

Edit:

I’ve not hit any of Walter’s problems, but once it became default, RAM became cheap enough I just doubled the amount of RAM I normally used for a server and did a swag of a double amount of Max Children any site on the box will ever hit and then used that multiplied by the number of sites on the box to set Max Requests. Which is completely stupid, but works okay enough I don’t ever get sites going dark.

I wanted to edit the original post for this, but the forums are locked down so tight I can’t…

To add to my frustration, while the above calculation has been working, it’s not because it was from sound logic. Egg on my face! Max Requests has nothing to do with the maximum request FPM as a whole does per pool before the pool respawns, it’s the “number of requests each child process should execute before respawning.”

 

[cPanelLauren]

Would you happen to know where to set the 3 settings (above: emergency_restart_threshold, eri, pct) so they are persistent?

If I'm wrapping my head around all three, then all three combined might mitigate the kill_all_lockers issue. Or if nothing else, worth a try seeing if they'll mitigate
@walterdevos 's "not serving sites anymore after max_children has been reached."

First I want to note that while it may be worth a try, adding kill_all_lockers to disable_functions is the official workaround from PHP, I would assume that if there were an alternate workaround that didn't involve disabling it they would have suggested that.

These changes can be made in WHM>>Software>>MultiPHP Manager -> System PHP-FPM Configuration which is discussed here: MultiPHP INI Editor for WHM | cPanel & WHM Documentation

Alternatively you can make them over CLI and this is discussed here: How to Manage Your php.ini Directives with PHP FPM | cPanel & WHM Documentation

system-wide modifications can be made here:
/var/cpanel/ApachePHPFPM/system_pool_defaults.yaml

Per VirtualHost modifications can be set here:

/var/cpanel/userdata/[user]/[domain].php-fpm.yaml


For the items you're looking to modify:

process.max int
emergency_restart_threshold int
emergency_restart_interval mixed
process_control_timeout mixed

These are all present in the system_pool_defaults yaml file referenced above. A full list of configuration values can be found in our documentation here: Configuration Values of PHP-FPM | cPanel & WHM Documentation

I wanted to edit the original post for this, but the forums are locked down so tight I can’t…

Forum responses are editable for 24 hours after they're made.