PHP-FPM configuration settings

[digitaliway]

once a day at random times I get notified : The service “apache_php_fpm” appears to be down.
it goes down then recovers in 4 minutes.

I really have no idea where to start and need some help to get this stable, thanks to anyone in advance.

fresh new server just migrated all clients from another cpanel server. server has 32cores 400gb ssd RAID 1 hardware - 96GB Ram - CLOUDLINUX 7.3 x86_64 standard –cPanel & WHM 64.0 (build 33) about 100 sites all running php-fpm with all default pool settings mostly php5.6 and a few php 7.

more error report info below and this image: [removed]

Memory Information
Used 4.85 GB
Available 89.36 GB
Installed 94.21 GB
Load Information 2.06 1.80 1.48

Service Check Raw Output
The subprocess “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm” reported error number 9 when it ended.

 

[Anoop P Alias]

Does this affect any web site?. I think this is cPanel daemon's internal php using php-fpm and don't affect the php-fpm service used by the hosted domains .

 

[digitaliway]

thanks for the reply. I am not sure how to determine if it is site or cpanel, would there be a specific log I could review? It is only down for 4 minutes or less according to the email report and I have yet to check sites in real time while down to see if they are non responsive since the window is so small and seems random.

 

[cPanelMichael]

Hello,

Could you let us know the output from /usr/local/apache/logs/error_log when this happens? Ensure to replace real domain names and IP addresses with examples, and post the output in CODE tags.

Thank you.

 

[digitaliway]

If you let me know how I can run a command to get the entire log as a downloaded file I can post privately or personal message.

I grabbed a few entries that seem problematic.

I am getting thousands of these entries

[Mon Jul 24 08:46:48.395089 2017] [proxy_fcgi:error] [pid 6920:tid 139924917442304] [client 0.0.0.0:50723] AH01071: Got error 'Primary script unknown\n'

[Mon Jul 24 08:47:05.680480 2017] [ssl:error] [pid 6920:tid 139924791551744] AH02032: Hostname 0.0.0.0 provided via SNI and hostname www.domain.com provided via HTTP have no compatible SSL setup

[Mon Jul 24 09:05:55.062773 2017] [mpm_worker:notice] [pid 3142:tid 139925245261952] AH00297: SIGUSR1 received.  Doing graceful restart
[Mon Jul 24 09:05:55.319296 2017] [hostinglimits:notice] [pid 3142:tid 139925245261952] mod_hostinglimits: use Min UID 0
[Mon Jul 24 09:05:55.319331 2017] [hostinglimits:notice] [pid 3142:tid 139925245261952] mod_hostinglimits: version 1.0-32. LVE mechanism enabled
[Mon Jul 24 09:05:55.319340 2017] [hostinglimits:notice] [pid 3142:tid 139925245261952] mod_hostinglimits: found apr extention version 3
[Mon Jul 24 09:05:55.319355 2017] [hostinglimits:notice] [pid 3142:tid 139925245261952] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok
[Mon Jul 24 09:05:55.374054 2017] [ssl:warn] [pid 3142:tid 139925245261952] AH01909: domain.com:443:0 server certificate does NOT include an ID which matches the server name
[Mon Jul 24 09:05:55.375679 2017] [ssl:warn] [pid 3142:tid 139925245261952] AH01909: domain.com:443:0 server certificate does NOT include an ID which matches the server name
[Mon Jul 24 09:05:55.408460 2017] [mpm_worker:notice] [pid 3142:tid 139925245261952] AH00292: Apache/2.4.25 (cPanel) OpenSSL/1.0.2k mod_bwlimited/1.4 configured -- resuming normal operations
[Mon Jul 24 09:05:55.408505 2017] [core:notice] [pid 3142:tid 139925245261952] AH00094: Command line: '/usr/sbin/httpd'

[Mon Jul 24 09:16:44.073968 2017] [core:error] [pid 15769:tid 139924799944448] [client 0.0.0.0:37320] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
[Mon Jul 24 09:16:44.145855 2017] [core:error] [pid 16271:tid 139925022525184] [client 0.0.0.0:37322] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
[Mon Jul 24 09:16:44.217821 2017] [core:error] [pid 15717:tid 139924774766336] [client 0.0.0.0:37324] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
[Mon Jul 24 09:16:45.390174 2017] [core:error] [pid 15769:tid 139924883871488] [client 0.0.0.0:37332] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.

 

[jonh]

Same thing started happening to me about two weeks ago. I get the fail email then few ins later the restart email for php fpm around same time.

 

[cPanelMichael]

Hello,

Feel free to open a support ticket using the link in my signature so we can take a closer look at an affected system to see what's happening.

Thank you.

 

[Nirjonadda]

once a day at random times I get notified : The service “apache_php_fpm” appears to be down.
it goes down then recovers in 4 minutes.

Yes, This same issue happened on my server.

 

[digitaliway]

@cPanelMichael I submitted a ticket ID is: 8748347. please let me know when you are able to take a look at this. the crashes are increasing daily from only one to now two or three per day.

 

[cPanelMichael]

Hello,

To update, internal case CPANEL-13411 is open to address an issue where the service check for apache_php_fpm hangs when a domain reaches the max_children PHP-FPM limit. This leads to an error message like this during the service check:

apache_php_fpm [[check command:-][check command output:
The subprocess “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm” reported error number 9 when it ended.][socket connect:N/A][fail count:1]Restarting apache_php_fpm....
[notify:failed service:apache_php_fpm]]...Done
Service Check Finished

I'll monitor this case and update this thread with more information as it becomes available.

In the meantime, the temporary workaround is to search the PHP-FPM error logs (/opt/cpanel/ea-php$$/root/usr/var/log/php-fpm/error.log) for any domain name's reaching the max_children limit. The offending entries will look like this:

ERROR: unable to read what child say: Bad file descriptor (9)
NOTICE: [pool domain_tld] child 1234 exited with code 0 after 25.12345 seconds from start

Once you find which domain names are reaching the limit, you can increase the max_children value for them using the "Pool Options" feature in "WHM >> MultiPHP Manager".

Thank you.

 

[Nirjonadda]

@cPanelMichael Does this issue still are not fixed? I have set Max Children to 15 from 5, Does this enough valve for Max Children?

 

[cPanelMichael]

@cPanelMichael Does this issue still are not fixed? I have set Max Children to 15 from 5, Does this enough valve for Max Children?

There's no update to report on CPANEL-13411 at this time. I'll continue to monitor the case and update this thread with new information as it becomes available. Increasing the "Max Children" limit for the offending domain name from "5" to "15" should act as a workaround, yes. However, you may want to monitor the PHP-FPM error log after making the change to verify the error messages no longer appear.

Thank you.

 

[PabloC]

Hi,

I am checking my PHP-FPM error logs and I get no "error". Thousands of lines like these, all the same. Is this ok?

[15-Dec-2017 21:30:50] NOTICE: [pool domain_com] child 13438 exited with code 0 after 78.066207 seconds from start
[15-Dec-2017 21:30:50] NOTICE: [pool domain_com] child 13655 started
[15-Dec-2017 21:30:57] NOTICE: [pool domain_com] child 13444 exited with code 0 after 78.084127 seconds from start
[15-Dec-2017 21:30:57] NOTICE: [pool domain_com] child 13657 started
[15-Dec-2017 21:31:26] NOTICE: [pool domain_com] child 13595 exited with code 0 after 79.547892 seconds from start
[15-Dec-2017 21:31:26] NOTICE: [pool domain_com] child 13682 started

 

[cPanelMichael]

Hello @PabloC,

Those are just notices and not actual errors. That should not lead to any actual problems.

Thank you.

 

[LBJ]

Increasing the "Max Children" limit for the offending domain name from "5" to "15" should act as a workaround, yes.

Is there truly no global way to change the pm.max_children value for all users under cPanel's implementation of FPM?

Having to resort to making a manual change to thousands of hosting accounts one-at-a-time over multiple servers is ludicrous.

Although mass updates of php_admin type entries are possible via...

/var/cpanel/ApachePHPFPM/system_pool_defaults.yaml

...such as...

php_admin_flag_allow_url_fopen: { name: 'php_admin_flag[allow_url_fopen]', value: Off }

...entries such as the following are not processed by /scripts/php_fpm_config --rebuild...

pm_max_children: { name: 'pm.max_children', value: 15 }

cPanel's documentation suggests otherwise at Configurations Values of PHP-FPM - Version 68 Documentation - cPanel Documentation but the documentation is either untested or out-of-date.

Am I missing a trick here?

Best regards,

LBJ

 

[cPanelMichael]

Hello @LBJ,

It looks like the entry you added for the max children value is incorrect. You should be able to simply add the following line to the /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml file:

pm_max_children: 20

Then, run the following command:

/usr/local/cpanel/scripts/php_fpm_config --rebuild

Additionally, I encourage you to vote for the following feature request:

PHP-FPM Enhanced Configuration

Thank you.

 

[LBJ]

You should be able to simply add the following line to the /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml file:

pm_max_children: 20

Then, run the following command:

/usr/local/cpanel/scripts/php_fpm_config --rebuild

Nope. That won't do the trick with all underscores in the key. Even though that's not in line with your own documentation, we did try that. The closest you can get is with a rebuild after adding...

pm.max_children: 20

to /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml

However, that still only gives you a conflicting directive as per...

...
pm = ondemand
pm.max_children = 20
pm.max_children = 5
pm.max_requests = 20
pm.max_spare_servers = 5
pm.min_spare_servers = 1
...

The lower directive takes control and you still end up with 5 as the max_children value. The error.log continues to show it that way too.

I'd really encourage you to set up a test system and try this all out for yourself. I think you'd be amazed at the size of the void between what's supposed to happen according to cPanel documentation with FPM and what really does.

Best regards,

LBJ

 

[cPanelMichael]

Hi @LBJ,

I tested that modification before sending my previous response and it worked well. Here's exactly how the /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml looks on the test machine:

# cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml
pm_max_children: 20

Upon running the "/usr/local/cpanel/scripts/php_fpm_config --rebuild" command, here's the output when viewing a user's PHP-FPM configuration file:

# grep children /opt/cpanel/ea-php56/root/etc/php-fpm.d/example.tld.conf
pm.max_children = 20

I just changed "pm_max_children: 20" to "pm_max_children: 25", ran the "/usr/local/cpanel/scripts/php_fpm_config --rebuild" command again, and it now shows 25:

# grep children /opt/cpanel/ea-php56/root/etc/php-fpm.d/example.tld.conf
pm.max_children = 25

It also shows the correct value under "Pool Options" in "WHM >> MultiPHP Manager". Are you sure there are no other entries in your /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml file, or no other PHP-FPM custom configurations are enabled?

Thank you.

 

[LBJ]

OK. We're now entering the realm of magic.

That simply doesn't work on any one of our servers, especially not with all underscores as you've entered.

Here's what happens if we follow your example...

# grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf
pm.max_children = 5

# cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml
pm_max_children: 20
php_admin_value_disable_functions: { name: 'php_admin_value[disable_functions]', value: exec,system,shell_exec,passthru,popen,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,show_source }
php_admin_flag_allow_url_fopen: { name: 'php_admin_flag[allow_url_fopen]', value: Off }
php_admin_value_error_reporting: { name: 'php_admin_value[error_reporting]', value: E_ALL & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT }

# /scripts/php_fpm_config --rebuild

# grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf
pm.max_children = 5

WHM also shows 5

---- Now if we go with pm.max_children: 20 instead of pm_max_children: 20 ---

# grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf
pm.max_children = 5

# cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml
pm.max_children: 20
php_admin_value_disable_functions: { name: 'php_admin_value[disable_functions]', value: exec,system,shell_exec,passthru,popen,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,show_source }
php_admin_flag_allow_url_fopen: { name: 'php_admin_flag[allow_url_fopen]', value: Off }
php_admin_value_error_reporting: { name: 'php_admin_value[error_reporting]', value: E_ALL & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT }


# /scripts/php_fpm_config --rebuild

# grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf
pm.max_children = 20
pm.max_children = 5

WHM still shows 5 and the error.log reports accordingly.

Best regards,

LBJ

Edit performed to mask domain name.

 

[LBJ]

...and just in case having other directives in the defaults yaml somehow makes a difference...

# grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf
pm.max_children = 5

# cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml
pm_max_children: 20

# /scripts/php_fpm_config --rebuild

# grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf
pm.max_children = 5

----

# grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf
pm.max_children = 5

# cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml
pm.max_children: 20

# /scripts/php_fpm_config --rebuild

# grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf
pm.max_children = 20
pm.max_children = 5

Best regards,

LBJ