SOLVED [UPS-184] OPCache issue after recent EA4 update

[kdean]

Anyone having problems with Opcache no longer caching scripts/keys after the 7.3.9 update? Still works fine on 7.2 and earlier but not at all with the latest 7.3.9 even though phpinfo says Opcode Caching is Up and Running and Optimization is Enabled. Previous version of 7.3 was working.

Had to set all my php 7.3 sites back to 7.2 for now until I have more time to look or if others have the same problem and an update/fix is released.

Also updated from cPanel 82.0.12 to 82.0.14 at the same time if it makes a difference on Centos 6.10

 

[cPanelLauren]

Hello @kdean

Can you show me the output of the following?

rpm -qa |grep opcache

 

[kdean]

ea-php73-php-opcache-7.3.9-1.1.3.cpanel.x86_64
ea-php72-php-opcache-7.2.22-1.1.3.cpanel.x86_64
ea-php71-php-opcache-7.1.32-1.1.3.cpanel.x86_64
ea-php70-php-opcache-7.0.33-9.9.5.cpanel.x86_64

 

[cPanelLauren]

Hello,


Thanks for that, I ultimately just wanted to ensure you had opcache present for php 7.3. I've not seen any recent reports of widespread issues with opcache on php 7.3. Can you tell me about your configuration?

- Are there new/different extensions installed for PHP 7.3 than what you had for 7.2? You can grab the extensions for a specific PHP version by running something like:

/opt/cpanel/ea-php72/root/usr/bin/php -m

/opt/cpanel/ea-php72/root/usr/bin/php -m

- Are you running PHP-FPM? Is anything noted in the PHP-FPM logs in regard to opcache if you are?

/opt/cpanel/ea-php72/root/usr/var/log/php-fpm/error.log

/opt/cpanel/ea-php73/root/usr/var/log/php-fpm/error.log

 

[kdean]

Nothing new. Have been running previous versions of 7.3 with no issue that I had noticed. I rebooted after the update so can't confirm if it was working prior to the reboot but definitely not afterward.

Running as PHP-FPM. Nothing of note in the php-fpm error logs.

The only difference that I see between the 2 extensions/modules is PHP7.2 has
bz2
calendar

and for some reason PHP7.3 does not. I usually match the configs so I don't know why those aren't there. I'll be sure to add them in to see if it makes any difference. Although looking at previously saved EasyApache build profiles, they weren't installed in previous versions either. php -m outputs further down below.


I even enabled
opcache.log_verbosity_level=4

and routed an error log:
opcache.error_log=/var/log/opcache.error_log

but nothing appears. I have one test domain where I through some php in addition to a phpinfo file and a OPcache status file but still shows no scripts cached. I'll need to try placing a copy of a full site there just to be definitive there should be some scripts.

The only php.ini differences that aren't path related is it appears PHP73 defaults to:
register_argc_argv = On

while my php72 is set to:
register_argc_argv = Off

Setting it to off doesn't seem to make any difference.

There just doesn't seem to be any logical reason for it to have all of sudden stopped working for just 7.3 while 7.2, 7.1,7.0,5.6 are all fine.

I may need to try another reboot in case there's something fundamentally messed up that's running. Or I may try removing 7.3 entirely and then re-provisioning it to see if that makes any difference. Likely will need to wait until the weekend to mess with that.

Here's the Opcache section of PHPinfo for 7.3 and yes it's generated from within the opcache.restrict_api path I've set.

/opt/cpanel/ea-php72/root/usr/bin/php -m

[PHP Modules]
bcmath
brotli
bz2
calendar
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
gettext
hash
iconv
imagick
imap
ionCube Loader
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
Reflection
session
SimpleXML
soap
sockets
SPL
sqlite3
standard
tidy
tokenizer
wddx
xml
xmlreader
xmlrpc
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
the ionCube PHP Loader + ionCube24

/opt/cpanel/ea-php73/root/usr/bin/php -m

[PHP Modules]
bcmath
brotli
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
gettext
hash
iconv
imagick
imap
ionCube Loader
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
Reflection
session
SimpleXML
soap
sockets
SPL
sqlite3
standard
tidy
tokenizer
wddx
xml
xmlreader
xmlrpc
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
the ionCube PHP Loader + ionCube24

 

[cPanelLauren]

Everything seems to be in order with opcache, I'm not able to replicate the issue you're having though.

Did you have the following set to "On" on php 7.2?

opcache.enable_file_override boolean
When enabled, the opcode cache will be checked for whether a file has already been cached when file_exists(), is_file() and is_readable() are called. This may increase performance in applications that check the existence and readability of PHP scripts, but risks returning stale data if opcache.validate_timestamps is disabled.

Also is there a specific reason you're restricting it to /home/rhubarb/phpstats? Is that the site you're testing with? If you remove the restriction does the behavior change?

 

[kdean]

The restriction is there so users can't view a list of cached scripts and especially prevents them from clearing the opcache, because it dumps it for the entire server which if done during the day causes a lot of cache slam slowdown with many requests trying to rebuild opcache at the same time over different sites. This is the same configuration for 7.2 and earlier but I could test to see if it makes any difference. It shouldn't since the whole point of that setting is to allow scripts to be cached but limit their access.

 

[kdean]

Removing API restriction makes no difference.

 

[cPanelLauren]

Hi @kdean

Thanks for checking that out for me, I definitely believe there's an issue with your specific configuration as I had opcache enabled for 15 minutes and already had several cached scripts/keys as you can see in the SS below (I had this account on ea-php72 w/OPcache and PHP-FPM then moved it to ea-php73 with the same configuration when I was testing) :



I'm leaning toward some configuration issue but also I don't think it's going to be able to be addressed easily through the forums. I'd advise opening a ticket at this point, if you do open one please update this thread with the ticket ID and I'll follow up here with the resolution so it might help others who run into a similar issue.


Thanks!

 

[kdean]

Was your test with 7.3.9 and are you running Centos 6 or 7 since I suppose their could be a difference in RPMs there.

In a last ditch effort before opening a support ticket, I uninstalled PHP 7.3 via Easy Apache and deleted the /opt/cpanel/ea-php73/ folder.

Reinstalled PHP 7.3 and extensions. Opcache was still not working. Rebooted. Still not working. Reapplied my 7.2 settings to 7.3 php.ini and opcache since it doesn't seem to having any bearing on the problem and so my settings are as they were for when it hopefully works again. I do not have any user level php/user.ini files set to disable opcache or anything like that. All the sites that weren't caching when switched down to 7.2 are caching fine.

Decided to test specific opcache functions to see what happens if it would throw any error.

I created a test php file that runs:

opcache_compile_file('phpinfo.php');

So PHP is basically telling opcache to cache my phpinfo.php file.

That works. I run it and the phpinfo.php file is listed as a cached script. Cached Scripts increments and Cache Misses increments (as expected).

Now if I load the phpinfo.php file directly it does not cause Cache Hits to increment so PHP-FPM is not even checking the cache. However if I run my test php file with the compile function in it, the Cache Hits does increment. So that function triggers a cache check as well.

So it appears somehow the automatic checking of the OPCache and Compiling of uncached scripts is just not happening anymore and only works with the specific php commands. I can't even find how or if there is even a configuration option for that. I wonder if it's not some php-fpm option that changed just for 7.3 since again this all worked before and still works 7.2 and earlier. Perhaps a problem with ea-php73-php-fpm.x86_64 or ea-php73-php-opcache.x86_64 for Centos 6.

This is just so weird. So, any last ideas before I resort to a support ticket?

 

[vacancy]

A similar problem exists in the directadmin control panel.

The problem is caused by a bug in php 7.3.9.

Opcache is installed but cache keys, cache misses values are null.

PHP 7.3 - OPcache not cachings scripts

PHP 7.3 - OPcache not caching scripts When testing with the following configuration: - Debian 9 - PHP 7.3.9 (php-fpm) - OPCache OPCache doesn't seem to be caching any scripts. Downgrading to PHP 7.2 fixes the issue. I've tested this on 2 different servers, using the following OPcache...

forum.directadmin.com

 

[kdean]

Guess it's good I'm not the only one after all and will likely just need to wait for PHP to fix.

 

[cPanelLauren]

Hi Guys,

I've seen a few more reports of this filter in, it looks like it's replicable on Litespeed servers as well as CloudLinux servers running Apache, which explains why I wasn't able to replicate when I attempted to on CentOS 7.

I did see one suggestion that indicated switching to DSO as a potential solution though, unfortunately, the tickets I found didn't have any followup confirming this did, in fact, resolve the issue.

Looking at PHP's reported bugs related to opcache I'm not seeing anything either: PHP :: Bugs :: Search

But I do see there is a litespeed thread for this open currently: Zend OPcache not working on PHP 7.3


@kdean are you running CloudLinux or Litespeed?

I'm waiting on a CloudLinux install to complete so I can attempt to replicate this again as well.

Thanks!

 

[kdean]

I'm running Apache/2.4.41, MPM: event, PHP-FPM, Centos 6.10 - Linux 2.6.32-754.18.2.el6.x86_64
No CloudLinux or Litespeed.
Multiple versions of PHP are required so no DSO.

 

[cPanelLauren]

Interesting, I've been testing everything on CentOS 7 w/Apache 2.4.1 and Prefork with and without PHP-FPM. , and I can't replicate. This changes my theory a bit, I'll try on CentOS 6 w/Event

 

[vacancy]

The problem is also present on the centos 7 servers.

How do you test the problem, lauren?

Opcache is installed in php 7.3 and appears active in php info, but no cache data is available.

 

[cPanelLauren]

Tested on the following:

[[email protected] ~]# uname -a
Linux server.mydomain.com 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Server Version: Apache/2.4.41 (cPanel) OpenSSL/1.0.2t mod_bwlimited/1.4

   Server MPM: prefork

With the following installed:

[[email protected] ~]# rpm -qa |egrep 'ea-php73-php-opcache|ea-php73-7.3'
ea-php73-php-opcache-7.3.9-2.2.1.cpanel.x86_64
ea-php73-7.3.9-1.1.1.cpanel.x86_64

Installed a WP site and I can see it working - per the screenshot's I showed previously.

I also tested using lsapi, cgi and with/without php-fpm

 

[cPanelLauren]

Also @kdean

Multiple versions of PHP are required so no DSO.

For the sole purpose of clarification, you can have multiple PHP versions with DSO but you can only have DSO as the handler for one of them.

 

[vacancy]

What do the opcache "cache hits" "cache misses" "cache script" "cache keys" values show in php info?

 

[cPanelLauren]

Hi @vacancy

You can see them in my screenshots above for reference, but here are today's:

Here's the first one of the day:




Then a bit later:


And another few minutes later:



As you can see the cache values are different in every example.