ERROR: failed to prepare the stderr pipe: Too many open files (24)

[APatchworkBoy]

Probably one for @cPanelMichael - I'm finding PHP-FPM stalling, and on checking /opt/cpanel/ea-php71/root/usr/var/log/php-fpm/error.log I'm seeing "ERROR: failed to prepare the stderr pipe: Too many open files (24)"

Using the {user} as which php-fpm is specifically running, I'm seeing the following:

# ulimit -aSH

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) 200000
scheduling priority             (-e) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) 200000
open files                      (-n) 5000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
real-time priority              (-r) unlimited
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 384
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Within /etc/security/limits.conf I've got...

{user}          hard    nofile          500000
{user}          soft    nofile          500000

And within /etc/security/limits.d/90-nproc.conf I've got...

{user}    soft    nproc     384
{user}    soft    nofile    5000

And running lsof | wc -l for the user in question returns:

870

In /opt/cpanel/ea-php71/root/etc/php-fpm.d/{user}.conf I've got...

[{user}]
catch_workers_output = yes
chdir = /home/{user}
group = "{user}"
listen = /opt/cpanel/ea-php71/root/usr/var/run/php-fpm/1733bd16bede7c482577faa383082e8365142cd2.sock
listen.group = "nobody"
listen.mode = 0660
listen.owner = "{user}"
php_admin_flag[allow_url_fopen] = on
php_admin_flag[log_errors] = on
php_admin_value[disable_functions] = exec,passthru,shell_exec,system
php_admin_value[doc_root] = "/home/{user}/public_html/"
php_admin_value[error_log] = /home/{user}/logs/{user}.php.error.log
php_admin_value[max_input_vars] = 4000
php_admin_value[short_open_tag] = on
php_value[error_reporting] = E_ALL & ~E_NOTICE
;php_value[error_reporting] = E_ALL
ping.path = /ping
pm = ondemand
pm.max_children = 200
pm.max_requests = 300
pm.max_spare_servers = 5
pm.min_spare_servers = 1
pm.process_idle_timeout = 20
pm.start_servers = 10
pm.status_path = /status
security.limit_extensions = .phtml .php .php3 .php4 .php5 .php6 .php7
user = "{user}"

Any ideas/suggestions on how I can diagnose this further? I'm not seeing any mention of hitting max_children in logs or anything else to indicate issues, just this Too Many Files (24) error... previous posts I've looked at for same error have referred to poss fixes for CentOS7, but not found any referring to CentOS6...

CentOS6.10 / WHMv82.0.16
Apache2.4.41 / PHP7.1 / PHP-FPM / MPM: Worker

 

[cPanelLauren]

I've actually seen this previously - what is the output of the following?

cat /proc/`pgrep fpm -u 0`/limits | grep 'Max open files'

I did also notice that you've modified the 90-nproc.conf file as this is the default:

cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited

In a lot of cases this has been really low and the services were inheriting the system values. I've seen this resolved in cases for cPanel PHP-FPM on CentOS 6 and Apache PHP-FPM on CentOS 7 but not for Apache PHP-FPM on CentOS 6. On CentOS 7 the resolution was to add some systemd configuration drop-in files. For CentOS 6 with cPanel PHP-FPM the resolution was to create a new .conf include in /var/cpanel/php-fpm.d/

# cat /var/cpanel/php-fpm.d/rlimit.conf
[global]
rlimit_files = 4096

This wouldn't be where you'd add an include for Apache PHP-FPM though, this would need to be added to /opt/cpanel/ea-php71/root/etc/php-fpm.d/ as follows:

[[email protected] php-fpm.d]# cat /opt/cpanel/ea-php71/root/etc/php-fpm.d/rlimit.conf
[global]
rlimit_files = 4096

since that's where ALL of the conf files for ea-php71 reside and the master conf file includes from here:

[[email protected] etc]# grep include /opt/cpanel/ea-php71/root/etc/php-fpm.conf
include=/opt/cpanel/ea-php71/root/etc/php-fpm.d/*.conf

Then restart the service with the following:

/scripts/restartsrv_apache_php_fpm --stop
/scripts/restartsrv_apache_php_fpm --start

Ultimately though, I can't help but wonder if this is not occurring due to some configuration changes that have been made on the system, you might try the proposed resolution and let me know if that works, I'm not 100% certain on it though it does appear that it should work.

 

[APatchworkBoy]

Thanks @cPanelLauren - shall give that a whirl on Monday and let you know. It’s all being caused by our net connection being maxed out and a custom web app handling it badly... problem only lasts a few minutes - I just need to get the server to survive those few minutes until I get chance to add some suitable timeout/retry-middleware to thenetworg’s oauth2-client. In meantime the app just keeps trying, as do the users, and more and more pools fire up until our connection clears. Bad coding on my part generally!

 

[cPanelLauren]

Ahh, well at the end of the day, at least you know the underlying cause. I'll await your update Monday!