System Integrity checking detected a modified system file

[BlueSteam]

Hello,

This morning I was greeted with the following lfd log output check from CSF.

Time: Fri Mar 18 02:20:28 2022 +0200

The following list of files have FAILED the md5sum comparison test. This means that the file has been changed in some way. This could be a result of an OS update or application upgrade. If the change is unexpected it should be investigated:

/usr/bin/ab: FAILED
/usr/bin/bootctl: FAILED
/usr/bin/busctl: FAILED
/usr/bin/coredumpctl: FAILED
/usr/bin/ea-php73: FAILED
/usr/bin/ea-php74: FAILED
/usr/bin/ea-php80: FAILED
/usr/bin/ex: FAILED
/usr/bin/gencat: FAILED
/usr/bin/getconf: FAILED
/usr/bin/getent: FAILED
/usr/bin/hostnamectl: FAILED
/usr/bin/htdbm: FAILED
/usr/bin/htdigest: FAILED
/usr/bin/htpasswd: FAILED
/usr/bin/httxt2dbm: FAILED
/usr/bin/iconv: FAILED
/usr/bin/journalctl: FAILED
/usr/bin/locale: FAILED
/usr/bin/localectl: FAILED
/usr/bin/localedef: FAILED
/usr/bin/loginctl: FAILED
/usr/bin/logresolve: FAILED
/usr/bin/makedb: FAILED
/usr/bin/pldd: FAILED
/usr/bin/resolvectl: FAILED
/usr/bin/rvi: FAILED
/usr/bin/rview: FAILED
/usr/bin/rvim: FAILED
/usr/bin/sprof: FAILED
/usr/bin/systemctl: FAILED
/usr/bin/systemd-analyze: FAILED
/usr/bin/systemd-ask-password: FAILED
/usr/bin/systemd-cat: FAILED
/usr/bin/systemd-cgls: FAILED
/usr/bin/systemd-cgtop: FAILED
/usr/bin/systemd-delta: FAILED
/usr/bin/systemd-detect-virt: FAILED
/usr/bin/systemd-escape: FAILED
/usr/bin/systemd-firstboot: FAILED
/usr/bin/systemd-hwdb: FAILED
/usr/bin/systemd-inhibit: FAILED
/usr/bin/systemd-machine-id-setup: FAILED
/usr/bin/systemd-mount: FAILED
/usr/bin/systemd-notify: FAILED
/usr/bin/systemd-path: FAILED
/usr/bin/systemd-resolve: FAILED
/usr/bin/systemd-run: FAILED
/usr/bin/systemd-socket-activate: FAILED
/usr/bin/systemd-stdio-bridge: FAILED
/usr/bin/systemd-sysusers: FAILED
/usr/bin/systemd-tmpfiles: FAILED
/usr/bin/systemd-tty-ask-password-agent: FAILED
/usr/bin/systemd-umount: FAILED
/usr/bin/timedatectl: FAILED
/usr/bin/udevadm: FAILED
/usr/bin/vi: FAILED
/usr/bin/view: FAILED
/usr/bin/vim: FAILED
/usr/bin/vimdiff: FAILED
/usr/bin/xmlcatalog: FAILED
/usr/bin/xmllint: FAILED
/usr/bin/xxd: FAILED
/usr/sbin/build-locale-archive: FAILED
/usr/sbin/dmeventd: FAILED
/usr/sbin/dmfilemapd: FAILED
/usr/sbin/dmsetup: FAILED
/usr/sbin/dmstats: FAILED
/usr/sbin/fcgistarter: FAILED
/usr/sbin/halt: FAILED
/usr/sbin/htcacheclean: FAILED
/usr/sbin/httpd: FAILED
/usr/sbin/iconvconfig: FAILED
/usr/sbin/init: FAILED
/usr/sbin/ldconfig: FAILED
/usr/sbin/lvchange: FAILED
/usr/sbin/lvconvert: FAILED
/usr/sbin/lvcreate: FAILED
/usr/sbin/lvdisplay: FAILED
/usr/sbin/lvextend: FAILED
/usr/sbin/lvm: FAILED
/usr/sbin/lvmconfig: FAILED
/usr/sbin/lvmdevices: FAILED
/usr/sbin/lvmdiskscan: FAILED
/usr/sbin/lvmpolld: FAILED
/usr/sbin/lvmsadc: FAILED
/usr/sbin/lvmsar: FAILED
/usr/sbin/lvreduce: FAILED
/usr/sbin/lvremove: FAILED
/usr/sbin/lvrename: FAILED
/usr/sbin/lvresize: FAILED
/usr/sbin/lvs: FAILED
/usr/sbin/lvscan: FAILED
/usr/sbin/modsec-sdbm-util: FAILED
/usr/sbin/nscd: FAILED
/usr/sbin/poweroff: FAILED
/usr/sbin/pvchange: FAILED
/usr/sbin/pvck: FAILED
/usr/sbin/pvcreate: FAILED
/usr/sbin/pvdisplay: FAILED
/usr/sbin/pvmove: FAILED
/usr/sbin/pvremove: FAILED
/usr/sbin/pvresize: FAILED
/usr/sbin/pvs: FAILED
/usr/sbin/pvscan: FAILED
/usr/sbin/reboot: FAILED
/usr/sbin/resolvconf: FAILED
/usr/sbin/rotatelogs: FAILED
/usr/sbin/runlevel: FAILED
/usr/sbin/shutdown: FAILED
/usr/sbin/suexec: FAILED
/usr/sbin/telinit: FAILED
/usr/sbin/udevadm: FAILED
/usr/sbin/vgcfgbackup: FAILED
/usr/sbin/vgcfgrestore: FAILED
/usr/sbin/vgchange: FAILED
/usr/sbin/vgck: FAILED
/usr/sbin/vgconvert: FAILED
/usr/sbin/vgcreate: FAILED
/usr/sbin/vgdisplay: FAILED
/usr/sbin/vgexport: FAILED
/usr/sbin/vgextend: FAILED
/usr/sbin/vgimport: FAILED
/usr/sbin/vgimportclone: FAILED
/usr/sbin/vgimportdevices: FAILED
/usr/sbin/vgmerge: FAILED
/usr/sbin/vgmknodes: FAILED
/usr/sbin/vgreduce: FAILED
/usr/sbin/vgremove: FAILED
/usr/sbin/vgrename: FAILED
/usr/sbin/vgs: FAILED
/usr/sbin/vgscan: FAILED
/usr/sbin/vgsplit: FAILED
/usr/sbin/zdump: FAILED
/usr/sbin/zic: FAILED
/bin/ab: FAILED
/bin/bootctl: FAILED
/bin/busctl: FAILED
/bin/coredumpctl: FAILED
/bin/ea-php73: FAILED
/bin/ea-php74: FAILED
/bin/ea-php80: FAILED
/bin/ex: FAILED
/bin/gencat: FAILED
/bin/getconf: FAILED
/bin/getent: FAILED
/bin/hostnamectl: FAILED
/bin/htdbm: FAILED
/bin/htdigest: FAILED
/bin/htpasswd: FAILED
/bin/httxt2dbm: FAILED
/bin/iconv: FAILED
/bin/journalctl: FAILED
/bin/locale: FAILED
/bin/localectl: FAILED
/bin/localedef: FAILED
/bin/loginctl: FAILED
/bin/logresolve: FAILED
/bin/makedb: FAILED
/bin/pldd: FAILED
/bin/resolvectl: FAILED
/bin/rvi: FAILED
/bin/rview: FAILED
/bin/rvim: FAILED
/bin/sprof: FAILED
/bin/systemctl: FAILED
/bin/systemd-analyze: FAILED
/bin/systemd-ask-password: FAILED
/bin/systemd-cat: FAILED
/bin/systemd-cgls: FAILED
/bin/systemd-cgtop: FAILED
/bin/systemd-delta: FAILED
/bin/systemd-detect-virt: FAILED
/bin/systemd-escape: FAILED
/bin/systemd-firstboot: FAILED
/bin/systemd-hwdb: FAILED
/bin/systemd-inhibit: FAILED
/bin/systemd-machine-id-setup: FAILED
/bin/systemd-mount: FAILED
/bin/systemd-notify: FAILED
/bin/systemd-path: FAILED
/bin/systemd-resolve: FAILED
/bin/systemd-run: FAILED
/bin/systemd-socket-activate: FAILED
/bin/systemd-stdio-bridge: FAILED
/bin/systemd-sysusers: FAILED
/bin/systemd-tmpfiles: FAILED
/bin/systemd-tty-ask-password-agent: FAILED
/bin/systemd-umount: FAILED
/bin/timedatectl: FAILED
/bin/udevadm: FAILED
/bin/vi: FAILED
/bin/view: FAILED
/bin/vim: FAILED
/bin/vimdiff: FAILED
/bin/xmlcatalog: FAILED
/bin/xmllint: FAILED
/bin/xxd: FAILED
/sbin/build-locale-archive: FAILED
/sbin/dmeventd: FAILED
/sbin/dmfilemapd: FAILED
/sbin/dmsetup: FAILED
/sbin/dmstats: FAILED
/sbin/fcgistarter: FAILED
/sbin/halt: FAILED
/sbin/htcacheclean: FAILED
/sbin/httpd: FAILED
/sbin/iconvconfig: FAILED
/sbin/init: FAILED
/sbin/ldconfig: FAILED
/sbin/lvchange: FAILED
/sbin/lvconvert: FAILED
/sbin/lvcreate: FAILED
/sbin/lvdisplay: FAILED
/sbin/lvextend: FAILED
/sbin/lvm: FAILED
/sbin/lvmconfig: FAILED
/sbin/lvmdevices: FAILED
/sbin/lvmdiskscan: FAILED
/sbin/lvmpolld: FAILED
/sbin/lvmsadc: FAILED
/sbin/lvmsar: FAILED
/sbin/lvreduce: FAILED
/sbin/lvremove: FAILED
/sbin/lvrename: FAILED
/sbin/lvresize: FAILED
/sbin/lvs: FAILED
/sbin/lvscan: FAILED
/sbin/modsec-sdbm-util: FAILED
/sbin/nscd: FAILED
/sbin/poweroff: FAILED
/sbin/pvchange: FAILED
/sbin/pvck: FAILED
/sbin/pvcreate: FAILED
/sbin/pvdisplay: FAILED
/sbin/pvmove: FAILED
/sbin/pvremove: FAILED
/sbin/pvresize: FAILED
/sbin/pvs: FAILED
/sbin/pvscan: FAILED
/sbin/reboot: FAILED
/sbin/resolvconf: FAILED
/sbin/rotatelogs: FAILED
/sbin/runlevel: FAILED
/sbin/shutdown: FAILED
/sbin/suexec: FAILED
/sbin/telinit: FAILED
/sbin/udevadm: FAILED
/sbin/vgcfgbackup: FAILED
/sbin/vgcfgrestore: FAILED
/sbin/vgchange: FAILED
/sbin/vgck: FAILED
/sbin/vgconvert: FAILED
/sbin/vgcreate: FAILED
/sbin/vgdisplay: FAILED
/sbin/vgexport: FAILED
/sbin/vgextend: FAILED
/sbin/vgimport: FAILED
/sbin/vgimportclone: FAILED
/sbin/vgimportdevices: FAILED
/sbin/vgmerge: FAILED
/sbin/vgmknodes: FAILED
/sbin/vgreduce: FAILED
/sbin/vgremove: FAILED
/sbin/vgrename: FAILED
/sbin/vgs: FAILED
/sbin/vgscan: FAILED
/sbin/vgsplit: FAILED
/sbin/zdump: FAILED
/sbin/zic: FAILED
/usr/local/bin/ea-php73: FAILED
/usr/local/bin/ea-php74: FAILED
/usr/local/bin/ea-php80: FAILED

So immediately I thought, there must have been a cPanel update that executed during the night. So I proceed to check the /var/cpanel/updatelogs/summary.log file and see that the last update ran on 1 March 2022 which was 17 days ago.

[[email protected]#### updatelogs]# tail -5 summary.log
[2021-12-03 02:15:08 +0200] Completed update 11.100.0.4 -> 11.100.0.5
[2022-01-21 02:29:57 +0200] Completed update 11.100.0.5 -> 11.100.0.7
[2022-02-04 02:15:00 +0200] Completed update 11.100.0.7 -> 11.100.0.9
[2022-02-23 02:14:22 +0200] Completed update 11.100.0.9 -> 11.100.0.10
[2022-03-01 02:14:17 +0200] Completed update 11.100.0.10 -> 11.100.0.11

Then I thought, maybe it was my almalinux that has auto update enabled for it, so I logged in to my almalinux and found that auto updates are off.

However, Almalinux wants to restart a bunch of services as though something did update and of these services, a bunch of them are cPanel services. So why did cPanel do a bunch of changes that has not been logged???



So now I am left wondering what the heck is going on. The above lfd log looks like almost the entire servers file integrity has changed somehow. Can someone give me any ideas of why cPanel made changes and didn't log it in the updates summary ??

 

[BlueSteam]

So looking at the WHM interface I was faced with this notice:



but for some strange reason, there was nothing in my updates summary.log

hmmm....server reboot commencing

 

[Ahmed Shibani]

Hi;

Can you check what were the last packages updated on the server using `yum history` and then `yum history info ID`, replace ID with the top most result from the previous command

 

[Spirogg]

Hello,

This morning I was greeted with the following lfd log output check from CSF.



So immediately I thought, there must have been a cPanel update that executed during the night. So I proceed to check the /var/cpanel/updatelogs/summary.log file and see that the last update ran on 1 March 2022 which was 17 days ago.



Then I thought, maybe it was my almalinux that has auto update enabled for I logged in to my almalinux web interface for the server and found that auto updates are off.

However, Almalinux wants to restart a bunch of services as though something did update and of these services, a bunch of the are cPanel services. So why did cPanel do a bunch of changes that has not been logged???

View attachment 76333

So now I am left wondering what the heck is going on. The above lfd log looks like almost the entire servers file integrity has changed somehow. Can someone give me any ideas of why cPanel made changes and didn't log it in the updates summary ??

This is most likely an update. there are nightly updates that happen like RPM updates and yum updates via cron from cPanel
I just received for 2 server one was around 1;51 am the email and the other was at 4am for the other server.

if your running Almalinux the log file is at /var/log/dnf.log and dnf.rpm.log
other centOS should be at /var/log/yum.log

here is a way to check each one that shows FAILED.

example: to check /usr/bin/htpasswd and see what pkg it belongs too:

run the command below you can do this for each one to make sure.
find the file paths that failed in your email - log from LFD

So one of them I have is
/usr/bin/htpasswd
I then run the code below

[[email protected] ~]# rpm -qf /usr/bin/htpasswd
ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64

--
So now I know it's in ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64
So to check when it updated I use this below grep filename /path/to/log.log

[[email protected] ~]# grep ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64 /var/log/dnf.rpm.log
2022-03-16T17:57:47-0500 SUBDEBUG Upgrade: ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64
2022-03-18T01:50:09-0500 SUBDEBUG Upgraded: ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64

--
Another example:

[[email protected] ~]# rpm -qf /usr/sbin/suphp
ea-apache24-mod_suphp-0.7.2-30.34.9.cpanel.x86_64

--
then check when it's been updated last:

[[email protected] ~]# grep ea-apache24-mod_suphp-0.7.2-30.34.9.cpanel.x86_64 /var/log/dnf.rpm.log
2022-03-16T17:57:53-0500 SUBDEBUG Upgrade: ea-apache24-mod_suphp-0.7.2-30.34.9.cpanel.x86_64
2022-03-18T01:50:09-0500 SUBDEBUG Upgraded: ea-apache24-mod_suphp-0.7.2-30.34.9.cpanel.x86_64

you can also use this pkg rpm checker from cPanel this will check if anything is wrong with any pkgs. I just found this a read a little on it.

The check_cpanel_pkgs Script | cPanel & WHM Documentation

This script scans every cPanel-managed package on your server for problems.

docs.cpanel.net

please read the [OPTIONS] section for more details before running any checks

- there is a way to check md5sum
- but I am not sure where to get the latest from cPanel to verify md5sum.

I'm sure there is a way I just can't find it.

maybe @cPRex can give us some inside info on where we can double check if we want the MD5 checksum for each pkg when we get this email notification.


hope this helps ease your mind or not

Spiro

 

[Spirogg]

Also I forgto to check md5 checksum
you can run this command for each failed

example

[[email protected] ~]# md5sum /usr/bin/htpasswd
e648faf395affeec7ce227d55604df3a  /usr/bin/htpasswd

another example

[[email protected] ~]# md5sum /usr/sbin/suphp
b0067b3b2f6f9542fe518f6c12329e2e  /usr/sbin/suphp

as I mentioned above maybe @cPRex can tell us where we can double check these to make sure there the same as cPanel's

have great morning.

Spiro

 

[cPRex]

Currently the best way is to manually pull the file from our mirror and compare the MD5. We have some more thoughts on that here:

FAILED the md5sum comparison test - how to know when updates occur?

I am receiving pretty regular warnings from LFD/CSF that files fail integrity checks, sometimes after LFD has blocked ip's from foreign countries. How does an admin tell when an update occurs in Cpanel/WHM so we can know if this warning is indeed related to an update?

forums.cpanel.net