locador

Member
Nov 8, 2018
5
0
1
São Paulo, Brazil
cPanel Access Level
Root Administrator
Hello,

I'm going crazy over tailwatchd. It thinks that PHP-FPM is not running, but it is, and it keeps sending notifications constantly.

I do have a small modification but I don't believe this should interfere with how the tailwatchd check the services. I'm using a custom compiled "php-fpm" and "php-cgi" that I've replaced within the usual cpanel/easyapache installation dir. The main reason is that some features that my customers requires are not supported by EasyApache. So, my only option is to compile a custom PHP-CGI/FPM and replace the binaries within the original installation dir.

And everything was working just fine for years until some days ago, when tailwatchd started to send tons of messages telling that "apache_php_fpm" is down. But it's not, they are running fine and websites are working normally.

Here's some useful information:

- As you can see, the services are running and websites are functional:

Code:
[email protected] [/tmp]# ps axf | grep -i fpm | grep -i master
 9818 ?        Ss     0:01 php-fpm: master process (/usr/local/cpanel/etc/php-fpm.conf)
28856 ?        Ss     0:00 php-fpm: master process (/opt/php56/etc/php-fpm.conf)
28861 ?        Ss     0:01 php-fpm: master process (/opt/php54/etc/php-fpm.conf)
- Systemctl reports services are up and running:

Code:
[email protected] [/tmp]# /sbin/service ea-php56-php-fpm status
Redirecting to /bin/systemctl status ea-php56-php-fpm.service
● ea-php56-php-fpm.service - The PHP FastCGI Process Manager 5.6
   Loaded: loaded (/usr/lib/systemd/system/ea-php56-php-fpm.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 10min ago
 Main PID: 28856 (php-fpm)
   CGroup: /system.slice/ea-php56-php-fpm.service
           └─28856 php-fpm: master process (/opt/php56/etc/php-fpm.conf)

Feb 25 11:38:39 [***HIDEN***] systemd[1]: Started The PHP FastCGI Process Manager 5.6.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend Guard Loader requires Zend Engine API version 220100525.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend OPcache requires Zend Engine API version 220100525.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache.
Code:
[email protected] [/tmp]# /sbin/service ea-php56-php-fpm status
Redirecting to /bin/systemctl status ea-php56-php-fpm.service
● ea-php56-php-fpm.service - The PHP FastCGI Process Manager 5.6
   Loaded: loaded (/usr/lib/systemd/system/ea-php56-php-fpm.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 10min ago
 Main PID: 28856 (php-fpm)
   CGroup: /system.slice/ea-php56-php-fpm.service
           └─28856 php-fpm: master process (/opt/php56/etc/php-fpm.conf)

Feb 25 11:38:39 [***HIDEN***][1]: Started The PHP FastCGI Process Manager 5.6.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend Guard Loader requires Zend Engine API version 220100525.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend OPcache requires Zend Engine API version 220100525.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer.
Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache.
[email protected] [/tmp]# /sbin/service ea-php54-php-fpm status
Redirecting to /bin/systemctl status ea-php54-php-fpm.service
● ea-php54-php-fpm.service - The PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/ea-php54-php-fpm.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 11min ago
 Main PID: 28861 (php-fpm)
   CGroup: /system.slice/ea-php54-php-fpm.service
           ├─25124 php-fpm: pool [***HIDEN***]
           ├─25316 php-fpm: pool [***HIDEN***]
           └─28861 php-fpm: master process (/opt/php54/etc/php-fpm.conf)

Feb 25 11:38:39 [***HIDEN***] systemd[1]: Started The PHP FastCGI Process Manager.
- But restartsrv_apache_php_fpm (invoked by tailwatchd to check the service) thinks it's disabled:

Code:
[email protected] [/tmp]# /usr/local/cpanel/scripts/restartsrv_apache_php_fpm --check
(XID mc2hd4) O serviço “Apache PHP-FPM 56” está desativado.
The “Apache PHP-FPM 56” service is down.
(XID sqp8uk) O serviço “Apache PHP-FPM 54” está desativado.
The “Apache PHP-FPM 54” service is down.
- And as I don't have a clue about how this compiled script works, I've tried debugging it with strace to try to see what it does:

Code:
strace -ff -o trace.txt /usr/local/cpanel/scripts/restartsrv_apache_php_fpm --check
- And I can see traces that it reads or stats all the configuration files and also checks the running process.

Code:
[email protected] [/tmp]# grep fpm trace.txt.27471
execve("/usr/local/cpanel/scripts/restartsrv_apache_php_fpm", ["/usr/local/cpanel/scripts/restar"..., "--check"], [/* 26 vars */]) = 0
stat("/var/cpanel/perl/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", 0x7fff5e54cd18) = -1 ENOENT (No such file or directory)
stat("/usr/local/cpanel/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", {st_mode=S_IFREG|0644, st_size=5171, ...}) = 0
open("/usr/local/cpanel/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", O_RDONLY|O_CLOEXEC) = 3
open("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", O_WRONLY|O_CREAT|O_EXCL|O_NONBLOCK|O_CLOEXEC, 0644) = 3
link("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", "/var/run/restartsrv_apache_php_fpm.lock") = 0
lstat("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0
unlink("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440") = 0
stat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0
open("/var/run/restartsrv_apache_php_fpm", O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC, 0666) = 4
stat("/opt/cpanel/ea-php55/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
stat("/opt/cpanel/ea-php55/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4429360, ...}) = 0
stat("/opt/cpanel/ea-php56/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
stat("/opt/cpanel/ea-php56/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=15447136, ...}) = 0
stat("/opt/cpanel/ea-php70/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
stat("/opt/cpanel/ea-php70/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4244848, ...}) = 0
stat("/opt/cpanel/ea-php71/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
stat("/opt/cpanel/ea-php71/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4405128, ...}) = 0
stat("/opt/cpanel/ea-php54/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
stat("/opt/cpanel/ea-php54/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=16455640, ...}) = 0
read(6, "php-fpm: master process (/usr/lo"..., 262144) = 76
read(6, "9818 (cpanel_php_fpm) S 1 9818 9"..., 262144) = 351
read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
read(6, "25345 (cpanel_php_fpm) S 9818 98"..., 262144) = 345
read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
read(6, "26871 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
read(6, "26887 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
read(6, "26891 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
read(6, "26898 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
read(6, "26934 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
read(6, "27145 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
read(6, "php-fpm: pool [***HIDEN***]\0\0"..., 262144) = 57
read(6, "27385 (php-fpm) S 28861 28861 28"..., 262144) = 340
read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57
read(6, "27392 (php-fpm) S 28861 28861 28"..., 262144) = 342
read(6, "php-fpm: pool [***HIDEN***]\0\0\0\0\0"..., 262144) = 57
read(6, "27433 (php-fpm) S 28861 28861 28"..., 262144) = 338
read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57
read(6, "27435 (php-fpm) S 28861 28861 28"..., 262144) = 341
read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57
read(6, "27439 (php-fpm) S 28861 28861 28"..., 262144) = 340
read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57
read(6, "27444 (php-fpm) S 28861 28861 28"..., 262144) = 339
read(6, "php-fpm: master process (/opt/ph"..., 262144) = 57
read(6, "28856 (php-fpm) S 1 28856 28856 "..., 262144) = 341
read(6, "php-fpm: master process (/opt/ph"..., 262144) = 57
read(6, "28861 (php-fpm) S 1 28861 28861 "..., 262144) = 350
openat(AT_FDCWD, "/opt/cpanel/ea-php55/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
openat(AT_FDCWD, "/opt/cpanel/ea-php56/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
openat(AT_FDCWD, "/opt/cpanel/ea-php70/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/opt/cpanel/ea-php71/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/opt/cpanel/ea-php54/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
lstat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0
lstat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0
unlink("/var/run/restartsrv_apache_php_fpm.lock") = 0
- But still, it thinks that the services are disabled.

Right now I've have deleted the chkservd files for "apache_php_fpm" to temporarily stop the false-positive notification messages.

It would be easier to know exactly how exactly "restartsrv_apache_php_fpm" works...
So, how does it really work? What does it check? Etc...
What can I do to make sure it sees my php-fpm running?

Thanks a lot.
 
Last edited by a moderator:

locador

Member
Nov 8, 2018
5
0
1
São Paulo, Brazil
cPanel Access Level
Root Administrator
OOPS! Sorry, I think I've duplicated my message ;) I usually cut and paste back the message before I click on submit to avoid getting an error from the forum and having to write it all again... So, copied twice I think...
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,229
463
Hello @locador,

Can you try stopping the master process for the PHP-FPM service that's reported as down, and then starting it directly from the binary? EX:

Code:
/opt/cpanel/ea-php70/root/usr/sbin/php-fpm
Starting it from the binary should output more information (and may list error messages).

Note that internal case CPANEL-19951 is open to request more intuitive error messages when Chkservd detects that one or more PHP-FPM services fail to start.

Thank you.
 

locador

Member
Nov 8, 2018
5
0
1
São Paulo, Brazil
cPanel Access Level
Root Administrator
Hi Michael,

I'm not using php70 on any websites, so this is expected:

Code:
[email protected] [~]# /opt/cpanel/ea-php70/root/usr/sbin/php-fpm
[27-Feb-2019 15:14:12] WARNING: Nothing matches the include pattern '/opt/cpanel/ea-php70/root/etc/php-fpm.d/*.conf' from /opt/cpanel/ea-php70/root/etc/php-fpm.conf at line 8.
[27-Feb-2019 15:14:12] ERROR: No pool defined. at least one pool section must be specified in config file
[27-Feb-2019 15:14:12] ERROR: failed to post process the configuration
[27-Feb-2019 15:14:12] ERROR: FPM initialization failed
However, 54 and 56 are ok, complete shell log:

Code:
[email protected] [~]# /opt/cpanel/ea-php54/root/usr/sbin/php-fpm
[27-Feb-2019 15:14:47] ERROR: An another FPM instance seems to already listen on /opt/cpanel/ea-php54/root/usr/var/run/php-fpm/e7bcdf970ea206fc5d2c5d09b2e18e869b44430b.sock
[27-Feb-2019 15:14:47] ERROR: FPM initialization failed
[email protected] [~]# pkill -f php-fpm
[email protected] [~]# /opt/cpanel/ea-php54/root/usr/sbin/php-fpm
[27-Feb-2019 15:15:06] NOTICE: fpm is running, pid 19519
[27-Feb-2019 15:15:06] NOTICE: ready to handle connections
^C[27-Feb-2019 15:15:10] NOTICE: Terminating ...
[27-Feb-2019 15:15:10] NOTICE: exiting, bye-bye!
[email protected] [~]# /opt/cpanel/ea-php56/root/usr/sbin/php-fpm
Zend Guard Loader requires Zend Engine API version 220100525.
The Zend Engine API version 220131226 which is installed, is newer.
Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader.

Zend OPcache requires Zend Engine API version 220100525.
The Zend Engine API version 220131226 which is installed, is newer.
Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache.
                                                                                                                                                                                            
[27-Feb-2019 15:15:14] NOTICE: fpm is running, pid 19906                                                                                                                                    
[27-Feb-2019 15:15:14] NOTICE: ready to handle connections                                                                                                                                  
[27-Feb-2019 15:15:14] NOTICE: systemd monitor interval set to 10000ms                                                                                                                      
^C[27-Feb-2019 15:15:17] NOTICE: Terminating ...                                                                                                                                            
[27-Feb-2019 15:15:17] NOTICE: exiting, bye-bye!
After that pkill, manual start for you to see the output and my CRTL-C, I've restarted both using systemctl, and they are running fine, websites working, etc:

Code:
[email protected] [~]# ps axf | grep -i fpm
18984 pts/0    S+     0:00  |       \_ grep --color=auto -i fpm
 9818 ?        Ss     0:11 php-fpm: master process (/usr/local/cpanel/etc/php-fpm.conf)
14389 ?        S      0:00  \_ php-fpm: pool cpanelroundcube
14827 ?        S      0:00  \_ php-fpm: pool cpanelroundcube
28856 ?        Ss     0:04 php-fpm: master process (/opt/php56/etc/php-fpm.conf)
28861 ?        Ss     0:45 php-fpm: master process (/opt/php54/etc/php-fpm.conf)
18913 ?        Sl     0:00  \_ php-fpm: pool [**HIDDEN**]_com_br
18918 ?        S      0:00  \_ php-fpm: pool [**HIDDEN**]_com_br
18933 ?        S      0:00  \_ php-fpm: pool [**HIDDEN**]_org
Right now I'm keeping the apache_php_fpm file from chkservd deleted because the error still persists and I've written my own script to monitor this service (by accessing a phpinfo on websites and then restarting if there's any error).

Thanks,
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,229
463
Hello @locador,

It's possible the custom modifications you've made to PHP-FPM are resulting in this issue. Can you can open a support ticket so we can take a closer look at the system? You can post the ticket number here and I'll link this thread to it.

Thank you.