[EA-9845 ] - WHM turned off PHP-FPM overnight for all domains, brought them ALL down, "File is writeable by group"

[adamreece.webbox]

Overnight all of our WHM servers (and a few 3rd party ones we manage) decided that all of their websites will suddenly return HTTP 500. Inspecting the Apache error log at "/usr/local/apache/logs/error_log" I found huge volumes of this:

SoftException in Application.cpp:267: File "/home/example/public_html/app.php" is writeable by group, referer: https://www.example.com/

I immediately checked the PHP files, which were either 775 or 664 as usual, however noticed that PHP-FPM had been switched off for every domain.

The quick answer to this was to turn on PHP-FPM again by: Software > MultiPHP Manager > PHP-FPM > Enable On All Domains

Question: Why did PHP-FPM turn itself off overnight for all sites without warning?

Interestingly this did not impact two of our WHM servers of which also use CloudLinux. Only the pure CentOS 7 installations (about 10 of them) were impacted.

 

[Martin Hinrichsen]

I can confirm.. All sites down this morning cause: All accounts FPM where switched off.

 

[infiniteeye]

Same issue today across multiple servers. thanks for posting this thread.

cPanel - sorry but this is simply not good enough

 

[Matt.R]

Same here, PHP-FPM seems have disabled for all sites (60) on the server at some point during the night (GMT). Also on Centos 7 (cPanel v96.0.9)

We do backups during the night with JetBackup, just checking times to see if any correlation.

 

[PMD]

Same. Thank you for this thread as I partially got a site working by altering permissions before I found this and was scratching my head for everything else.

The backups don't seem correlated. From my looking at various logs and the cpanel updates - the moment apache restarted after the cpanel update is when everything started throwing 500 errors.

 

[Matt.R]

The backups don't seem correlated.

After checking further I agree.

 

[anton_latvia]

I can confirm the same. Even though in my opinion correct permissions are 755 and 644 - still - several servers got this problem that has caused a lot of stress. Turning all to PHP-FPM solved the problem.

But I am sure - most of sites on these affected servers had PHP-FPM enabled, but today everything was off.

We can not have it like this! There was an update, but update should never change anything in an unappropriate way, so that sites stop working. We are not hosting some dog-and-cat sites - people run businesses, they pay and expect some service - while we pay and expect - that servers will be working. When such things happens, especially without any actions from our side - how does this look like? This has already caused financial loss for customers and us!

+ even servers with old Centos6 were affected.

 

[rs200]

Same for me, unfortunately turning again on PHP-FPM didn't resolve immediately the issue, because i use domain.tld.php-fpm.yaml files inside /var/cpanel/userdata directory for change the root directory of my clients' applications.

Fortunately i had few files to modify and the applications are all up again!

 

[adamreece.webbox]

We can not have it like this! There was an update, but update should never change anything in an unappropriate way, so that sites stop working. We are not hosting some dog-and-cat sites - people run businesses, they pay and expect some service - while we pay and expect - that servers will be working. When such things happens, especially without any actions from our side - how does this look like? This has already caused financial loss for customers and us!

I concur. This is the third gigantic screw up on cPanel's part we've experienced this year. We pay our monthly license fees very consistently but don't get any compensation back.

 

[Matt.R]

Suspect there will a substantial number adding to this thread as the US wakes up this morning....

 

[Qadisha]

Same here, PHP-FPM seems have disabled for all sites (60) on the server at some point during the night (GMT). Also on Centos 7 (cPanel v96.0.9)

We do backups during the night with JetBackup, just checking times to see if any correlation.

Same here and it seems the backup are not affected.
As per our findings, everything started around 20.45 CEST. Have you any evidence about timing or process triggering this?
The lastlog show something around 0030 CEST, but i'm not sure is related.

wp-toolkit Thu Jun 10 00:36:52 +0200 2021


Thanks

 

[Reado]

Any word from cPanel yet as to what caused this? One of my servers had PHP-FPM disabled, whereas the other server says this for all domains:

"You must install the PHP-FPM package in order to use PHP-FPM."

What has cPanel done to EasyApache to cause it to either disable PHP-FPM or remove it entirely?!...

 

[Reado]

Any word from cPanel yet as to what caused this? One of my servers had PHP-FPM disabled, whereas the other server says this for all domains:

"You must install the PHP-FPM package in order to use PHP-FPM."

What has cPanel done to EasyApache to cause it to either disable PHP-FPM or remove it entirely?!...

OK after further investigation it turns out EA4 listed PHP-FPM has "installed" despite MultiPHP Manager claiming otherwise. Restarting the PHP-FPM service allowed me to enable PHP-FPM on all domains.

The server that had the "you must install" message did not show any HTTP 500 messages - all websites were actually running fine. It's a shame that this server was not customer facing and that the server that did displaying HTTP 500 was indeed customer facing! But hey, it's resolved for us now, though I would like an explanation from cPanel as to what caused this in the first place.

 

[JoeHabit]

All of our users php fpm was also disabled.
It seems to me that the latest commit to the 100-phpfpm_cleanup.pl file could be related to this problem:
https://github.com/CpanelInc/ea-apa...f0532fc7995160a03fc61f54ee2b397037174aec1696f

 

[Stephen Evans]

We've also had the same issue on three server so far this morning and are actively checking others. All CentOS 7.

Once was actually pretty unstable with high CPU usage. Apache and PHP-FPM Service restarts were not enough and the server had to be rebooted.

Stephen

 

[JoeHabit]

Also from the updatelog file:

[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:10 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 71 since we are removing the package.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:12 2021] [100-phpfpm_cleanup] All domains that were on ea-php71-php-fpm should be back to system defaults.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:12 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 72 since we are removing the package.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:13 2021] [100-phpfpm_cleanup] All domains that were on ea-php72-php-fpm should be back to system defaults.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:13 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 56 since we are removing the package.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:14 2021] [100-phpfpm_cleanup] All domains that were on ea-php56-php-fpm should be back to system defaults.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:14 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 70 since we are removing the package.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:14 2021] [100-phpfpm_cleanup] No domains were on ea-php70-php-fpm.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:15 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 74 since we are removing the package.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:15 2021] [100-phpfpm_cleanup] All domains that were on ea-php74-php-fpm should be back to system defaults.
[2021-06-10 03:58:16 +0200]      [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:15 2021] [100-phpfpm_cleanup] Restarting Apache

 

[MikeDVB]

The cPanel QA we know and love. Now with more downtime-inducing bugs as of VC ownership and massive price increases.

Maybe CentOS is the minority these days compared to CloudLinux and they forgot to test on base CentOS?

Phew. Thank the lord we are paying so much more than we were last year.

 

[masterross]

I have a similar issue.
All my site were down and the log:

[Thu Jun 10 10:20:40.736751 2021] [:error] [pid 19791:tid 47292916741888] [client 66.249.66.19:50671] SoftException in Application.cpp:648: Directory /home is not owned by ceramic1
[Thu Jun 10 10:20:40.736852 2021] [core:error] [pid 19791:tid 47292916741888] [client 66.249.66.19:50671] End of script output before headers: index.php
[Thu Jun 10 10:20:40.783966 2021] [:error] [pid 19991:tid 47292800931584] [client 46.148.79.91:36656] SoftException in Application.cpp:648: Directory /home is not owned by mobilneforum
[Thu Jun 10 10:20:40.784072 2021] [core:error] [pid 19991:tid 47292800931584] [client 46.148.79.91:36656] End of script output before headers: mobiquo.php
[Thu Jun 10 10:20:40.787667 2021] [:error] [pid 19991:tid 47292805134080] [client 46.148.79.91:36656] SoftException in Application.cpp:648: Directory /home is not owned by mobilneforum
[Thu Jun 10 10:20:40.787736 2021] [core:error] [pid 19991:tid 47292805134080] [client 46.148.79.91:36656] End of script output before headers: mobiquo.php
[Thu Jun 10 10:20:41.337658 2021] [:error] [pid 19991:tid 47292805134080] [client 46.148.79.91:36656] SoftException in Application.cpp:648: Directory /home is not owned by mobilneforum
[Thu Jun 10 10:20:41.337741 2021] [core:error] [pid 19991:tid 47292805134080] [client 46.148.79.91:36656] End of script output before headers: mobiquo.php
[Thu Jun 10 10:20:41.361356 2021] [:error] [pid 19991:tid 47292805134080] [client 157.55.39.8:63232] SoftException in Application.cpp:648: Directory /home is not owned by mobimart
[Thu Jun 10 10:20:41.361428 2021] [core:error] [pid 19991:tid 47292805134080] [client 157.55.39.8:63232] End of script output before headers: showthread.php
[Thu Jun 10 10:23:35.497504 2021] [core:error] [pid 19991:tid 47292800931584] [client 185.167.35.105:42928] End of script output before headers: showthread.php

So I rebuild easy apache with default settings and all was OK except for sites with PHP 5.6 - 7.1.
They continue to give error:

[Thu Jun 10 16:05:29.009167 2021] [core:error] [pid 26557:tid 47017672058624] [client 157.55.39.77:11328] End of script output before headers: index.php
[Thu Jun 10 16:05:31.024846 2021] [:error] [pid 26468:tid 47017751488256] [client 157.55.39.98:56768] SoftException in Application.cpp:648: Directory /home is not owned by linkdecode
[Thu Jun 10 16:05:31.024930 2021] [core:error] [pid 26468:tid 47017751488256] [client 157.55.39.98:56768] End of script output before headers: index.php
[Thu Jun 10 16:05:31.037466 2021] [:error] [pid 26468:tid 47017751488256] [client 157.55.39.98:56768] SoftException in Application.cpp:648: Directory /home is not owned by linkdecode
[Thu Jun 10 16:05:31.037676 2021] [core:error] [pid 26468:tid 47017751488256] [client 157.55.39.98:56768] End of script output before headers: index.php

Any idea?

 

[kennysamuerto]

We have encountered some servers with this problem.

This is not the way, cPanel. We can't waste days of work fixing things that break the panel. Each time, the product is more expensive, and offers lower quality.

 

[anton_latvia]

Phew. Thank the lord we are paying so much more than we were last year.

Yes, indeed! We are happy too. But - I wanted to mention CloudLinux / Centos. My personal experience shows that sites on Centos are faster. That resource management, that comes with CloudLinux does add quite a bit of delays. But we love extra fee, though.