Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Chkservd thinks PHP-FPM is down

Discussion in 'EasyApache' started by locador, Feb 25, 2019.

  1. locador

    locador Member

    Joined:
    Nov 8, 2018
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    São Paulo, Brazil
    cPanel Access Level:
    Root Administrator
    Hello,

    I'm going crazy over tailwatchd. It thinks that PHP-FPM is not running, but it is, and it keeps sending notifications constantly.

    I do have a small modification but I don't believe this should interfere with how the tailwatchd check the services. I'm using a custom compiled "php-fpm" and "php-cgi" that I've replaced within the usual cpanel/easyapache installation dir. The main reason is that some features that my customers requires are not supported by EasyApache. So, my only option is to compile a custom PHP-CGI/FPM and replace the binaries within the original installation dir.

    And everything was working just fine for years until some days ago, when tailwatchd started to send tons of messages telling that "apache_php_fpm" is down. But it's not, they are running fine and websites are working normally.

    Here's some useful information:

    - As you can see, the services are running and websites are functional:

    Code:
    root@hexer [/tmp]# ps axf | grep -i fpm | grep -i master
     9818 ?        Ss     0:01 php-fpm: master process (/usr/local/cpanel/etc/php-fpm.conf)
    28856 ?        Ss     0:00 php-fpm: master process (/opt/php56/etc/php-fpm.conf)
    28861 ?        Ss     0:01 php-fpm: master process (/opt/php54/etc/php-fpm.conf)
    
    - Systemctl reports services are up and running:

    Code:
    root@hexer [/tmp]# /sbin/service ea-php56-php-fpm status
    Redirecting to /bin/systemctl status ea-php56-php-fpm.service
    ● ea-php56-php-fpm.service - The PHP FastCGI Process Manager 5.6
       Loaded: loaded (/usr/lib/systemd/system/ea-php56-php-fpm.service; enabled; vendor preset: disabled)
       Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 10min ago
     Main PID: 28856 (php-fpm)
       CGroup: /system.slice/ea-php56-php-fpm.service
               └─28856 php-fpm: master process (/opt/php56/etc/php-fpm.conf)
    
    Feb 25 11:38:39 [***HIDEN***] systemd[1]: Started The PHP FastCGI Process Manager 5.6.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend Guard Loader requires Zend Engine API version 220100525.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend OPcache requires Zend Engine API version 220100525.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache.
    
    Code:
    root@hexer [/tmp]# /sbin/service ea-php56-php-fpm status
    Redirecting to /bin/systemctl status ea-php56-php-fpm.service
    ● ea-php56-php-fpm.service - The PHP FastCGI Process Manager 5.6
       Loaded: loaded (/usr/lib/systemd/system/ea-php56-php-fpm.service; enabled; vendor preset: disabled)
       Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 10min ago
     Main PID: 28856 (php-fpm)
       CGroup: /system.slice/ea-php56-php-fpm.service
               └─28856 php-fpm: master process (/opt/php56/etc/php-fpm.conf)
    
    Feb 25 11:38:39 [***HIDEN***][1]: Started The PHP FastCGI Process Manager 5.6.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend Guard Loader requires Zend Engine API version 220100525.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend OPcache requires Zend Engine API version 220100525.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer.
    Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache.
    root@hexer [/tmp]# /sbin/service ea-php54-php-fpm status
    Redirecting to /bin/systemctl status ea-php54-php-fpm.service
    ● ea-php54-php-fpm.service - The PHP FastCGI Process Manager
       Loaded: loaded (/usr/lib/systemd/system/ea-php54-php-fpm.service; enabled; vendor preset: disabled)
       Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 11min ago
     Main PID: 28861 (php-fpm)
       CGroup: /system.slice/ea-php54-php-fpm.service
               ├─25124 php-fpm: pool [***HIDEN***]
               ├─25316 php-fpm: pool [***HIDEN***]
               └─28861 php-fpm: master process (/opt/php54/etc/php-fpm.conf)
    
    Feb 25 11:38:39 [***HIDEN***] systemd[1]: Started The PHP FastCGI Process Manager.
    
    - But restartsrv_apache_php_fpm (invoked by tailwatchd to check the service) thinks it's disabled:

    Code:
    root@hexer [/tmp]# /usr/local/cpanel/scripts/restartsrv_apache_php_fpm --check
    (XID mc2hd4) O serviço “Apache PHP-FPM 56” está desativado.
    The “Apache PHP-FPM 56” service is down.
    (XID sqp8uk) O serviço “Apache PHP-FPM 54” está desativado.
    The “Apache PHP-FPM 54” service is down.
    
    - And as I don't have a clue about how this compiled script works, I've tried debugging it with strace to try to see what it does:

    Code:
    strace -ff -o trace.txt /usr/local/cpanel/scripts/restartsrv_apache_php_fpm --check
    
    - And I can see traces that it reads or stats all the configuration files and also checks the running process.

    Code:
    root@hexer [/tmp]# grep fpm trace.txt.27471
    execve("/usr/local/cpanel/scripts/restartsrv_apache_php_fpm", ["/usr/local/cpanel/scripts/restar"..., "--check"], [/* 26 vars */]) = 0
    stat("/var/cpanel/perl/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", 0x7fff5e54cd18) = -1 ENOENT (No such file or directory)
    stat("/usr/local/cpanel/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", {st_mode=S_IFREG|0644, st_size=5171, ...}) = 0
    open("/usr/local/cpanel/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", O_RDONLY|O_CLOEXEC) = 3
    open("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", O_WRONLY|O_CREAT|O_EXCL|O_NONBLOCK|O_CLOEXEC, 0644) = 3
    link("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", "/var/run/restartsrv_apache_php_fpm.lock") = 0
    lstat("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0
    unlink("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440") = 0
    stat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0
    open("/var/run/restartsrv_apache_php_fpm", O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC, 0666) = 4
    stat("/opt/cpanel/ea-php55/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
    stat("/opt/cpanel/ea-php55/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4429360, ...}) = 0
    stat("/opt/cpanel/ea-php56/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
    stat("/opt/cpanel/ea-php56/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=15447136, ...}) = 0
    stat("/opt/cpanel/ea-php70/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
    stat("/opt/cpanel/ea-php70/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4244848, ...}) = 0
    stat("/opt/cpanel/ea-php71/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
    stat("/opt/cpanel/ea-php71/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4405128, ...}) = 0
    stat("/opt/cpanel/ea-php54/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0
    stat("/opt/cpanel/ea-php54/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=16455640, ...}) = 0
    read(6, "php-fpm: master process (/usr/lo"..., 262144) = 76
    read(6, "9818 (cpanel_php_fpm) S 1 9818 9"..., 262144) = 351
    read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
    read(6, "25345 (cpanel_php_fpm) S 9818 98"..., 262144) = 345
    read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
    read(6, "26871 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
    read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
    read(6, "26887 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
    read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
    read(6, "26891 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
    read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
    read(6, "26898 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
    read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
    read(6, "26934 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
    read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76
    read(6, "27145 (cpanel_php_fpm) S 9818 98"..., 262144) = 344
    read(6, "php-fpm: pool [***HIDEN***]\0\0"..., 262144) = 57
    read(6, "27385 (php-fpm) S 28861 28861 28"..., 262144) = 340
    read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57
    read(6, "27392 (php-fpm) S 28861 28861 28"..., 262144) = 342
    read(6, "php-fpm: pool [***HIDEN***]\0\0\0\0\0"..., 262144) = 57
    read(6, "27433 (php-fpm) S 28861 28861 28"..., 262144) = 338
    read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57
    read(6, "27435 (php-fpm) S 28861 28861 28"..., 262144) = 341
    read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57
    read(6, "27439 (php-fpm) S 28861 28861 28"..., 262144) = 340
    read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57
    read(6, "27444 (php-fpm) S 28861 28861 28"..., 262144) = 339
    read(6, "php-fpm: master process (/opt/ph"..., 262144) = 57
    read(6, "28856 (php-fpm) S 1 28856 28856 "..., 262144) = 341
    read(6, "php-fpm: master process (/opt/ph"..., 262144) = 57
    read(6, "28861 (php-fpm) S 1 28861 28861 "..., 262144) = 350
    openat(AT_FDCWD, "/opt/cpanel/ea-php55/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
    openat(AT_FDCWD, "/opt/cpanel/ea-php56/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
    openat(AT_FDCWD, "/opt/cpanel/ea-php70/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
    openat(AT_FDCWD, "/opt/cpanel/ea-php71/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
    openat(AT_FDCWD, "/opt/cpanel/ea-php54/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
    lstat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0
    lstat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0
    unlink("/var/run/restartsrv_apache_php_fpm.lock") = 0
    
    - But still, it thinks that the services are disabled.

    Right now I've have deleted the chkservd files for "apache_php_fpm" to temporarily stop the false-positive notification messages.

    It would be easier to know exactly how exactly "restartsrv_apache_php_fpm" works...
    So, how does it really work? What does it check? Etc...
    What can I do to make sure it sees my php-fpm running?

    Thanks a lot.
     
    #1 locador, Feb 25, 2019
    Last edited by a moderator: Feb 27, 2019
  2. locador

    locador Member

    Joined:
    Nov 8, 2018
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    São Paulo, Brazil
    cPanel Access Level:
    Root Administrator
    OOPS! Sorry, I think I've duplicated my message ;) I usually cut and paste back the message before I click on submit to avoid getting an error from the forum and having to write it all again... So, copied twice I think...
     
  3. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,273
    Likes Received:
    2,154
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @locador,

    Can you try stopping the master process for the PHP-FPM service that's reported as down, and then starting it directly from the binary? EX:

    Code:
    /opt/cpanel/ea-php70/root/usr/sbin/php-fpm
    Starting it from the binary should output more information (and may list error messages).

    Note that internal case CPANEL-19951 is open to request more intuitive error messages when Chkservd detects that one or more PHP-FPM services fail to start.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. locador

    locador Member

    Joined:
    Nov 8, 2018
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    São Paulo, Brazil
    cPanel Access Level:
    Root Administrator
    Hi Michael,

    I'm not using php70 on any websites, so this is expected:

    Code:
    root@hexer [~]# /opt/cpanel/ea-php70/root/usr/sbin/php-fpm
    [27-Feb-2019 15:14:12] WARNING: Nothing matches the include pattern '/opt/cpanel/ea-php70/root/etc/php-fpm.d/*.conf' from /opt/cpanel/ea-php70/root/etc/php-fpm.conf at line 8.
    [27-Feb-2019 15:14:12] ERROR: No pool defined. at least one pool section must be specified in config file
    [27-Feb-2019 15:14:12] ERROR: failed to post process the configuration
    [27-Feb-2019 15:14:12] ERROR: FPM initialization failed
    
    However, 54 and 56 are ok, complete shell log:

    Code:
    root@hexer [~]# /opt/cpanel/ea-php54/root/usr/sbin/php-fpm
    [27-Feb-2019 15:14:47] ERROR: An another FPM instance seems to already listen on /opt/cpanel/ea-php54/root/usr/var/run/php-fpm/e7bcdf970ea206fc5d2c5d09b2e18e869b44430b.sock
    [27-Feb-2019 15:14:47] ERROR: FPM initialization failed
    root@hexer [~]# pkill -f php-fpm
    root@hexer [~]# /opt/cpanel/ea-php54/root/usr/sbin/php-fpm
    [27-Feb-2019 15:15:06] NOTICE: fpm is running, pid 19519
    [27-Feb-2019 15:15:06] NOTICE: ready to handle connections
    ^C[27-Feb-2019 15:15:10] NOTICE: Terminating ...
    [27-Feb-2019 15:15:10] NOTICE: exiting, bye-bye!
    root@hexer [~]# /opt/cpanel/ea-php56/root/usr/sbin/php-fpm
    Zend Guard Loader requires Zend Engine API version 220100525.
    The Zend Engine API version 220131226 which is installed, is newer.
    Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader.
    
    Zend OPcache requires Zend Engine API version 220100525.
    The Zend Engine API version 220131226 which is installed, is newer.
    Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache.
                                                                                                                                                                                                
    [27-Feb-2019 15:15:14] NOTICE: fpm is running, pid 19906                                                                                                                                    
    [27-Feb-2019 15:15:14] NOTICE: ready to handle connections                                                                                                                                  
    [27-Feb-2019 15:15:14] NOTICE: systemd monitor interval set to 10000ms                                                                                                                      
    ^C[27-Feb-2019 15:15:17] NOTICE: Terminating ...                                                                                                                                            
    [27-Feb-2019 15:15:17] NOTICE: exiting, bye-bye!                                                                                                                    
    After that pkill, manual start for you to see the output and my CRTL-C, I've restarted both using systemctl, and they are running fine, websites working, etc:

    Code:
    root@hexer [~]# ps axf | grep -i fpm
    18984 pts/0    S+     0:00  |       \_ grep --color=auto -i fpm
     9818 ?        Ss     0:11 php-fpm: master process (/usr/local/cpanel/etc/php-fpm.conf)
    14389 ?        S      0:00  \_ php-fpm: pool cpanelroundcube
    14827 ?        S      0:00  \_ php-fpm: pool cpanelroundcube
    28856 ?        Ss     0:04 php-fpm: master process (/opt/php56/etc/php-fpm.conf)
    28861 ?        Ss     0:45 php-fpm: master process (/opt/php54/etc/php-fpm.conf)
    18913 ?        Sl     0:00  \_ php-fpm: pool [**HIDDEN**]_com_br
    18918 ?        S      0:00  \_ php-fpm: pool [**HIDDEN**]_com_br
    18933 ?        S      0:00  \_ php-fpm: pool [**HIDDEN**]_org
    
    Right now I'm keeping the apache_php_fpm file from chkservd deleted because the error still persists and I've written my own script to monitor this service (by accessing a phpinfo on websites and then restarting if there's any error).

    Thanks,
     
  5. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,273
    Likes Received:
    2,154
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @locador,

    It's possible the custom modifications you've made to PHP-FPM are resulting in this issue. Can you can open a support ticket so we can take a closer look at the system? You can post the ticket number here and I'll link this thread to it.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice