Servers updated to 110.0.2 from 108.0.15 (Broken Pipe) [Edit: Not proven, just speculation, read entire thread]

[Steini Petur]

Hi,

Not sure if related but quite a coincidence, all servers that are

v110.0.2

They now have broken pipes on everything

[root@xxx ~]#  cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5 | md5sum | awk {'print $1'}
tr: write error: Broken pipe
tr: write error
cat: write error: Broken pipe
d9df68393da0b9ffc963c2209418ed84

[root@yyy ~]# cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5 | md5sum | awk {'print $1'}
tr: write error: Broken pipe
tr: write error
83922d16d1a6eb2f0aebd6fe456ab98e
cat: write error: Broken pipe

Any server that is still not at v110.0.2 and is such as v108.0.15

[root@zzz ~]# cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5 | md5sum | awk {'print $1'}
5aba211b19b9a527b983cc3f5e9a0cf9

What I am about to do is I am about to update to 110.0.2 on the last server which is still pending, all other servers are done

[2023-04-11 04:20:26 +0000]   Completed update 11.108.0.15 -> 11.110.0.2

As shown on 11th, all our servers 5 from one and 1 from another brand updated to 11.110.0.2 and caused Broken pipes, and the Professional Spamexpert plugin to start throwing errors like

[14-Apr-2023 11:00:17 Atlantic/Reykjavik] PHP Fatal error:  Uncaught TypeError: array_key_exists(): Argument #2 ($array) must be of type array, Zend_Registry given in /usr/local/prospamfilter/library/Zend/Registry.php:211
Stack trace:
#0 /usr/local/prospamfilter/library/Zend/Registry.php(145): Zend_Registry->offsetExists('logger')
#1 /usr/local/prospamfilter/library/SpamFilter/Core.php(131): Zend_Registry::get('logger')
#2 /usr/local/prospamfilter/library/SpamFilter/Core.php(113): SpamFilter_Core::getPanelType()
#3 /usr/local/prospamfilter/application/bootstrap.php(174): SpamFilter_Core::isCpanel()
#4 /usr/local/prospamfilter/interfaces/admin.php(46): require_once('/usr/local/pros...')
#5 /usr/local/prospamfilter/frontend/whm/prospamfilter.php(36): require_once('/usr/local/pros...')
#6 {main}
  thrown in /usr/local/prospamfilter/library/Zend/Registry.php on line 211

I tie this exactly to the updates because every server that hasn't updated in the fold does not show these symptoms.

The only server left is our zzz, and I am about to let it update to 110.0.2 to prove this.. starting now below, once this update finishes, I will run again the cat command..

=> Log opened from /usr/local/cpanel/scripts/updatenow (1692002) at Fri Apr 14 11:27:01 2023
[2023-04-14 11:27:01 +0000]   Running version '11.108.0.15' of updatenow.
[2023-04-14 11:27:01 +0000]   Detected version '11.108.0.15' from version file.
[2023-04-14 11:27:01 +0000]   Successfully verified signature for cpanel (key types: release).
[2023-04-14 11:27:01 +0000]   Target version set to '11.110.0.2'
[2023-04-14 11:27:01 +0000]   Switching to version 11.110.0.2 of updatenow to determine if we can reach that version without failure.
[2023-04-14 11:27:01 +0000]   Retrieving and staging /cpanelsync/11.110.0.2/cpanel/scripts/updatenow.static.bz2
[2023-04-14 11:27:01 +0000]   Using mirror '185.15.22.168' for host 'httpupdate.cpanel.net'.
[2023-04-14 11:27:02 +0000]   Become an updatenow.static for version: 11.110.0.2
.......
[2023-04-14 11:28:22 +0000]      [/usr/local/cpanel/scripts/find_and_fix_rpm_issues]   Preparing...                          ########################################
[2023-04-14 11:28:22 +0000]      [/usr/local/cpanel/scripts/find_and_fix_rpm_issues]   Updating / installing...
[2023-04-14 11:28:22 +0000]      [/usr/local/cpanel/scripts/find_and_fix_rpm_issues]   rpm_is_working-1.0-0                  ########################################

.....MUCH LATER

=> Log opened from cPanel Update (upcp) - Slave (1820317) at Fri Apr 14 12:02:34 2023
[2023-04-14 12:02:34 +0000]   Pre Maintenance completed successfully
[2023-04-14 12:02:34 +0000]   Running /usr/local/cpanel/scripts/postupcp
[2023-04-14 12:02:57 +0000] Completed all updates
=> Log closed Fri Apr 14 12:02:57 2023

The output of the cat command before at v108 was

[root@zzz diskar]# cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5 | md5sum | awk {'print $1'}
5aba211b19b9a527b983cc3f5e9a0cf9

After v108 to v110

[root@zzz diskar]# cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5 | md5sum | awk {'print $1'}
5aba211b19b9a527b983cc3f5e9a0cf9

It's okay still on this server but "Professional SpamFilter" failed.. now its showing 500 error throwing array_key_exists right after the update, so now all our SpamExpert plugins fail on all WHM servers, although the cat command still doesnt show broken pipe.

System is CloudLinux v8.7.0 STANDARD standard on zzz which is still okay regarding pipe broken, but plugin fails post update
System is CloudLinux v8.5.0 and v8.6.0 STANDARD on xxx yyy which are broken pipes and plugin fails post update

EDIT:
I checked and in terminal on xxx yyy zzz the pipe isnt broken, only on xxx yyy the pipe broken in SSH but not terminal and not in SSH and terminal on zzz which i just updated. Maybe the ssh server update? but all our servers now have broken plugins for SpamExperts..

 

[mtindor]

No idea if any of this is related to your problem but:
1. WHM 110 Release Notes say that Internal cPanel PHP is updated from PHP 7.4 to 8.1 (which could affect 3rd party applications)
2. Some of your servers are running outdated versions of CL8 - why? Some problems may go away by simply making sure all servers are running CL 8.7
3. If you rarely reboot your machines (because you depend upon kernelcare to allow you to not have to reboot so often), you should still consider rebooting

You might find that being on CL 8.7 across the board and rebooting machines more often will prevent some of the issues that you see.

 

[Steini Petur]

Hi @mtindor,

You have some good points, before opening this ticket, I actually did "php -v" but perhaps that isnt the binary that runs WHM? so maybe this is two issues for me

a) The updates to 110, would have broken the PHP application because internally (some other bin is updated that I wasnt checking) so its just breaking my plugin cuz of it
b) The servers CL8.5 CL8.6 are simply the versions that were available at the time, I was not aware of options to upgrade CL 8.5 to 8.7? I was not aware you can do that safely
c) Yes the Kernelcare is the one that I rely on but perhaps rebooting will resolve a few things

I think based on your message that the a) is unrelated, and that what I thought was the "version" that was running, is not running the actual show, just the clients..must be some other PHP Binary that I missed..

It's 2separate issues.

 

[Steini Petur]

@mtindor, I want to point out that some of our 8.7.0 also showed this issue, but the difference between xxx and zzz are kernel differences, not the minor release differences of the CL. I am running update kernel on the

xxx = 4.18.0-348.lve.el8.x86_64
zzz = 4.18.0-425.3.1.lve.3.el8.x86_64

Lets see if updating the kernel to the latest, and 110 the broken pipes stop, but this is not gonna fix the PLugin, I need to Alert SpamExperts of those developments.

 

[mtindor]

Hi @mtindor,

You have some good points, before opening this ticket, I actually did "php -v" but perhaps that isnt the binary that runs WHM? so maybe this is two issues for me

a) The updates to 110, would have broken the PHP application because internally (some other bin is updated that I wasnt checking) so its just breaking my plugin cuz of it
b) The servers CL8.5 CL8.6 are simply the versions that were available at the time, I was not aware of options to upgrade CL 8.5 to 8.7? I was not aware you can do that safely
c) Yes the Kernelcare is the one that I rely on but perhaps rebooting will resolve a few things

I think based on your message that the a) is unrelated, and that what I thought was the "version" that was running, is not running the actual show, just the clients..must be some other PHP Binary that I missed..

It's 2separate issues.

CL 8.5 or CL 8.6 to CL 8.7 are simply done by running something like this (assuming an RHEL-based system of course):

dnf clean all; dnf check-update (that won't update anything -- just give you a list of things needing updated)
dnf clean all; dnf update (will give you the option to update anything outstanding)

It simply sounds like you are way behind on keeping the OS itself up to date. I would have expected it to automatically do that as part of the nightly UPCP, but I can't remember. I am always manually interacting with my systems and making sure the OS is up to date.

You don't need to post it here, but just run a dnf clean all; dnf check update and see if you have a boatload of RPMS to update on your CL 8.5 / 8.6 boxes. I imagine you do. And maybe they aren't updating automatically because either you have them set to not auto update, or maybe they aren't auto-updating because of some conflicts (which you would be able to see if you do a check update).

 

[mtindor]

@mtindor, I want to point out that some of our 8.7.0 also showed this issue, but the difference between xxx and zzz are kernel differences, not the minor release differences of the CL. I am running update kernel on the

xxx = 4.18.0-348.lve.el8.x86_64
zzz = 4.18.0-425.3.1.lve.3.el8.x86_64

Lets see if updating the kernel to the latest, and 110 the broken pipes stop, but this is not gonna fix the PLugin, I need to Alert SpamExperts of those developments.

You're right in that doing any of this is unlikely to fix the plugin issue. But, I'd be really surprised if you are able to update JUST the kernel to 4.18.0-425.3.1.lve.3 on your CL 8.5/8.6 boxes without actually updating the boxes to CL 8.7. You really should consider updating everything that needs to be updated to bring you up to CL 8.7. I have never had an issue updating minor point releases of CL, Almalinux, CentOS, etc. Of course, they are your machines and I would always urge you to be careful. Just do a check update (see previous messages) to see just how out of date things are / how many RPMs are available to update. If you aren't running an RHEL-based OS but instead are running Ubuntu, dont listen to anything I have to say. I don't run CL on Ubuntu.

 

[Steini Petur]

Hi @mtindor,

Of course, my boxes are all RHEL based, Alma converted to CL for Shared hosting needs, I have just never had issues withmy 8.5 and 8.6 boxes, the 8.7 was the latest when I installed last box.

I was under the impression the Kernel Care was suppose to ensure thingswere up to code security wise on the kernels.

I will run those updates soon, Its just it's a live environment, and I need to schedule these reboots, can't just disconnect 800-1000 websites for even the reboot period without so much as a notification about it, we have some anal clients.

With that said, the PHP is the issue, the file Zend/Registry.php line uses array_key_exists, which needs to be replaced with property_exists or similar function as 8 removed what was deprecated, I have notified SpamExperts team about the development.

As suspected, these are 2 separate issues, but even with a full on update, I am not sure if it fixes the broken pipe, but right now 8.7 box has broken pipe anyway after the 110 update,


[root@xxx etc]# cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5 | md5sum | awk {'print $1'}
tr: write error: Broken pipe
tr: write error
cat: write error: Broken pipe
dac9b53b6092f3c4b8d49982b8dd942d

And its a 8.7 box, so not really sure about the broken pipe, how I can resolve those, it seems that I can't link it directly to 110, since zzz is 8.7 with no broken pipe after 110 update, I think its time to do some dnf update check as you mentioned, start by getting up to code and see if it resovles the pipe, and maybe something in 110 breaks the older version of the kernel? Who knows, god Iove software, always so reliable =)

 

[mtindor]

Can't help you on the broken pipe issue. My 110 boxes do not have that issue. I'm running up to date CL 8.7 / kernel / etc on all of them and do reboot at least once a month.

I realize that rebooting is not trivial, especially on production boxes with high numbers of users from all over the world and who like to whine. I've got customers like that. I'm just saying that you will want to / should want to always make sure the OS is completely up to date. And that simply isn't the case if you are running some machines with 8.5 / 8.6.

kernelcare does nothing to keep the OS up to date. Kernelcare just attempts to make sure that whatever kernel you are running is patched for any known vulnerabilities. You still need to / should do regular OS updates to keep thigns up to date. If you keep the OS up to date, you don't always have to reboot afterwards because kernelcare will keep the kernel patched. But I have seen many many funky things happen over the years on servers that haven't been updated or that have been updated but not rebooted for months.

Thanks for sharing that you are having an issue with the SpamExperts plugin. I don't use it, but i have friends who run a hosting company that do -- and I'm going to make them aware of that issue. Fortunately, they are on STABLE and thus likely not updated to 110 yet.

 

[Steini Petur]

Hi @mtindor, That maybe is advisable, I think I will set our boxes to STABLE from RELEASE, although Release has never really been an issue for me since their issue a year or so back with the MariaDB breaking.. Thank you for all of this, I actually try to reboot the boxes but I usually do it when there is "more on the agenda" such as I create a Maintenance window, use it to update the Bios firmware and tackle something more than just a OS update reboot. However you have given me ALOT to think about, I think what I will start by doing now is run all those tests on the "lowest box" on our second brand, with only < 50 on it, and do the update checks, update, reboot see if the pipe resolves on it, if it does, then I know what I need to do on all boxes.

 

[Steini Petur]

@mtindor,

This server is mostly up to date he had

4.18.0-348.lve.el8.x86_64

Could be updated to

kernel.x86_64            1:4.18.0-425.13.1.lve.el8    cloudlinux-x86_64-server-8
kernel-core.x86_64       1:4.18.0-425.13.1.lve.el8    cloudlinux-x86_64-server-8
kernel-modules.x86_64    1:4.18.0-425.13.1.lve.el8    cloudlinux-x86_64-server-8
kmod-libs.x86_64         2:25-19.el8.cloudlinux.3     cloudlinux-x86_64-server-8
kmod-lve.x86_64          1:2.0-50.el8                 cloudlinux-x86_64-server-8

And is 8.7 but threw pipe, after the update, reboot.. I can confirm that

[root@xxx ~]# cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5 | md5sum | awk {'print $1'}
tr: write error: Broken pipe
tr: write error
28ad0896620c7c03159d7b6ffd2216dd
cat: write error: Broken pipe

Sadly, 8.7 latest, all 100% up to date all packages, does not resolve my tr/cat broken pipes, but seems that it may not affect the server functionality to a degree where things arent functioning, something happened and it happened to all our servers that went 110 across the board except 1 server, not sure what it was or how it came about but it globally hit all our servers basically, but NOTE, this is in a SHELL not in a TERMINAL, in a TERMINAL this is not an issue, so this is somehow related to the SHELL and as such, annoying yes, but detrimental to server functionality. no...

But the spamexperts have to come up with a solution quick and they should have ages ago, its been deprecated for the longest of times, right now the frontend and backend plugin is broken.. the SE still works as it should so no actual issues with delivery, but the UI is kaputzkie and we've had 3 tickets already..

 

[mtindor]

@mtindor,

This server is mostly up to date he had

4.18.0-348.lve.el8.x86_64

Could be updated to

kernel.x86_64            1:4.18.0-425.13.1.lve.el8    cloudlinux-x86_64-server-8
kernel-core.x86_64       1:4.18.0-425.13.1.lve.el8    cloudlinux-x86_64-server-8
kernel-modules.x86_64    1:4.18.0-425.13.1.lve.el8    cloudlinux-x86_64-server-8
kmod-libs.x86_64         2:25-19.el8.cloudlinux.3     cloudlinux-x86_64-server-8
kmod-lve.x86_64          1:2.0-50.el8                 cloudlinux-x86_64-server-8

And is 8.7 but threw pipe, after the update, reboot.. I can confirm that

[root@xxx ~]# cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5 | md5sum | awk {'print $1'}
tr: write error: Broken pipe
tr: write error
28ad0896620c7c03159d7b6ffd2216dd
cat: write error: Broken pipe

Sadly, 8.7 latest, all 100% up to date all packages, does not resolve my tr/cat broken pipes, but seems that it may not affect the server functionality to a degree where things arent functioning, something happened and it happened to all our servers that went 110 across the board except 1 server, not sure what it was or how it came about but it globally hit all our servers basically, but NOTE, this is in a SHELL not in a TERMINAL, in a TERMINAL this is not an issue, so this is somehow related to the SHELL and as such, annoying yes, but detrimental to server functionality. no...

But the spamexperts have to come up with a solution quick and they should have ages ago, its been deprecated for the longest of times, right now the frontend and backend plugin is broken.. the SE still works as it should so no actual issues with delivery, but the UI is kaputzkie and we've had 3 tickets already..

Thanks for that info. I don't usually have a shell available for any person. I enabled shell access to one of my personal accounts on a 110 server and then did an "su -l <username>" and attempted to run the command you did. I did not have any issues. Also SSH'd in as that user with shell enabled and it worked fine. This is on a CL 8.7 box / WHM 110.

mike

 

[cPRex]

Did you already submit a ticket to us about this issue?

 

[Steini Petur]

Did you already submit a ticket to us about this issue?

I haven't done that, because I am trying to understand the issue, I know it occurred after 110 from 108 in those servers, but it could be a coincidence. I can see that if I do something like "ls /etc | head -c5" output is fine, but

[root@xxx diskar]# cat /dev/urandom | tr -dc A-Za-z0-9 | head -c5

Please note that this command is inside the SpamExperts installer so I just used that command, the pipe is breaking urandom spits out a large list of characters, then head -c5 is just not working, I have checked though and seems like the system is okay, it seems to be just "shell" and some specific long filenames..

Honestly cPRex, If I could click something like "Close ticket" I would, because at this point It's a nuisance but not affecting system functionality overall, Terminal doesn't do this, and only some random large scale files trying to cat them and -head c5 throws this..

 

[Steini Petur]

@mtindor

This was from the SpamExpert team

Hello,
Thank you for the report. We have been made aware of this issue and can confirm the SpamExperts plugin is not compatible with cPanel version 110 for the reasons you indicated.
Our engineers are working on updating the plugin to accommodate for the changes made by cPanel as part of their current development sprint so this is being looked at with some priority. I do not currently have an estimated date for a release, but we will keep your ticket active and advise further as soon as possible.
As SpamExperts does not rely on the plugin to function, you and your clients can still access the interface in the normal standard way at xxx where you can perform all the same functions and access the logs, quarantine etc.
Kind regards,
Andrew McCabe
Technical Support Representative, Intermediate

So we're looking at some days/weeks for this, probably need to do backend access for quite a while, no client access.

 

[mtindor]

@mtindor

This was from the SpamExpert team




So we're looking at some days/weeks for this, probably need to do backend access for quite a while, no client access.

Thank you. That's unfortunate. I will let my buddies know so that they can possibly avoid updating to 110. Hopefully SpamExperts fixes the plugin before WHM users on Stable are forced to update to 110.

Mike

 

[Steini Petur]

Thank you. That's unfortunate. I will let my buddies know so that they can possibly avoid updating to 110. Hopefully SpamExperts fixes the plugin before WHM users on Stable are forced to update to 110.

Mike

Yeah, wish I would have had it set to STABLE, its by default set to RELEASE..Overall RELEASE is usually stable for production but whats done is done, I can't reverse this I see in the /3rdparty/php that 74 exists and 81, i wonder if cPanel can help me adjust the internal cPanel to use 74 for now.. I might slip em a ticket.

 

[cPRex]

No, there is not a way to roll back the internal PHP, so we wouldn't be able to help with that.

 

[Steini Petur]

@cPRex sadly, that was what cPanel ended up telling me, I was hoping there was an unsupported trick up their sleeve but I guess not, on inspecting the 74 folder its not full of bin files anyway, it has remnant files but its obvious 81 is the only way, so the only option would be Feature request but again, by the time that became a Feature I would literally have this solved by the developers, but this does put me in a pause, I should change the RELEASE (default option) to STABLE.

 

[gvard]

N-able SpamExperts: Integrations Update – cPanel

Posted on April 19, 2023 by dorinatoba


We’ve updated cPanel integrations to work with the latest upstream versions.
cPanel Stable

• Controlpanel: cPanel v11.108.0.15
• PHP version: 7.4.33
• Addon version: 3.0.96617

cPanel Release

• Controlpanel: cPanel v11.110.0.2
• PHP version: 8.1.16
• Addon version: 3.0.96617

Note: In case the update is not possible using UI, please use the script:
wget -N https://download.seinternal.com/integration/installers/cpanel/installer.sh && bash installer.sh
Please consider the documentation for more details: https://documentation.n-able.com/spamexperts/userguide/Content/Integration/cpanel-addon.htm
----

It seems that the new update works for CL7/CL8, however we need to install it manually.

There is another issue now, we have some legacy CL6 servers which throw an error after the update (which was tried to be made automatically AFAIK). I believe they will let us know that they no longer support CL6, however I would like to know if someone has the previous -working- version of the SpamExperts plugin in order for us to downgrade to it (and disable auto updates).

 

[gvard]

For CentOS6 users, the issue can be resolved following 2 steps:

1) Install the previous SpamExperts addon:

wget https://raw.githubusercontent.com/spamexperts/cpanel-addon/master/bin/installer/old_installer.sh
chmod +x old_installer.sh
./old_installer.sh 3.0.96604

2) Disable automatic updates (from SpamExperts configuration tab).

However the issue with CloudLinux6 remains. CL6 uses PHP 8.1.16 and the plugin displays the following when accessing it: