System Integrity checking detected a modified system file

BlueSteam

Well-Known Member
Feb 21, 2013
129
21
68
cPanel Access Level
Reseller Owner
Hello,

This morning I was greeted with the following lfd log output check from CSF.

Time: Fri Mar 18 02:20:28 2022 +0200

The following list of files have FAILED the md5sum comparison test. This means that the file has been changed in some way. This could be a result of an OS update or application upgrade. If the change is unexpected it should be investigated:

/usr/bin/ab: FAILED
/usr/bin/bootctl: FAILED
/usr/bin/busctl: FAILED
/usr/bin/coredumpctl: FAILED
/usr/bin/ea-php73: FAILED
/usr/bin/ea-php74: FAILED
/usr/bin/ea-php80: FAILED
/usr/bin/ex: FAILED
/usr/bin/gencat: FAILED
/usr/bin/getconf: FAILED
/usr/bin/getent: FAILED
/usr/bin/hostnamectl: FAILED
/usr/bin/htdbm: FAILED
/usr/bin/htdigest: FAILED
/usr/bin/htpasswd: FAILED
/usr/bin/httxt2dbm: FAILED
/usr/bin/iconv: FAILED
/usr/bin/journalctl: FAILED
/usr/bin/locale: FAILED
/usr/bin/localectl: FAILED
/usr/bin/localedef: FAILED
/usr/bin/loginctl: FAILED
/usr/bin/logresolve: FAILED
/usr/bin/makedb: FAILED
/usr/bin/pldd: FAILED
/usr/bin/resolvectl: FAILED
/usr/bin/rvi: FAILED
/usr/bin/rview: FAILED
/usr/bin/rvim: FAILED
/usr/bin/sprof: FAILED
/usr/bin/systemctl: FAILED
/usr/bin/systemd-analyze: FAILED
/usr/bin/systemd-ask-password: FAILED
/usr/bin/systemd-cat: FAILED
/usr/bin/systemd-cgls: FAILED
/usr/bin/systemd-cgtop: FAILED
/usr/bin/systemd-delta: FAILED
/usr/bin/systemd-detect-virt: FAILED
/usr/bin/systemd-escape: FAILED
/usr/bin/systemd-firstboot: FAILED
/usr/bin/systemd-hwdb: FAILED
/usr/bin/systemd-inhibit: FAILED
/usr/bin/systemd-machine-id-setup: FAILED
/usr/bin/systemd-mount: FAILED
/usr/bin/systemd-notify: FAILED
/usr/bin/systemd-path: FAILED
/usr/bin/systemd-resolve: FAILED
/usr/bin/systemd-run: FAILED
/usr/bin/systemd-socket-activate: FAILED
/usr/bin/systemd-stdio-bridge: FAILED
/usr/bin/systemd-sysusers: FAILED
/usr/bin/systemd-tmpfiles: FAILED
/usr/bin/systemd-tty-ask-password-agent: FAILED
/usr/bin/systemd-umount: FAILED
/usr/bin/timedatectl: FAILED
/usr/bin/udevadm: FAILED
/usr/bin/vi: FAILED
/usr/bin/view: FAILED
/usr/bin/vim: FAILED
/usr/bin/vimdiff: FAILED
/usr/bin/xmlcatalog: FAILED
/usr/bin/xmllint: FAILED
/usr/bin/xxd: FAILED
/usr/sbin/build-locale-archive: FAILED
/usr/sbin/dmeventd: FAILED
/usr/sbin/dmfilemapd: FAILED
/usr/sbin/dmsetup: FAILED
/usr/sbin/dmstats: FAILED
/usr/sbin/fcgistarter: FAILED
/usr/sbin/halt: FAILED
/usr/sbin/htcacheclean: FAILED
/usr/sbin/httpd: FAILED
/usr/sbin/iconvconfig: FAILED
/usr/sbin/init: FAILED
/usr/sbin/ldconfig: FAILED
/usr/sbin/lvchange: FAILED
/usr/sbin/lvconvert: FAILED
/usr/sbin/lvcreate: FAILED
/usr/sbin/lvdisplay: FAILED
/usr/sbin/lvextend: FAILED
/usr/sbin/lvm: FAILED
/usr/sbin/lvmconfig: FAILED
/usr/sbin/lvmdevices: FAILED
/usr/sbin/lvmdiskscan: FAILED
/usr/sbin/lvmpolld: FAILED
/usr/sbin/lvmsadc: FAILED
/usr/sbin/lvmsar: FAILED
/usr/sbin/lvreduce: FAILED
/usr/sbin/lvremove: FAILED
/usr/sbin/lvrename: FAILED
/usr/sbin/lvresize: FAILED
/usr/sbin/lvs: FAILED
/usr/sbin/lvscan: FAILED
/usr/sbin/modsec-sdbm-util: FAILED
/usr/sbin/nscd: FAILED
/usr/sbin/poweroff: FAILED
/usr/sbin/pvchange: FAILED
/usr/sbin/pvck: FAILED
/usr/sbin/pvcreate: FAILED
/usr/sbin/pvdisplay: FAILED
/usr/sbin/pvmove: FAILED
/usr/sbin/pvremove: FAILED
/usr/sbin/pvresize: FAILED
/usr/sbin/pvs: FAILED
/usr/sbin/pvscan: FAILED
/usr/sbin/reboot: FAILED
/usr/sbin/resolvconf: FAILED
/usr/sbin/rotatelogs: FAILED
/usr/sbin/runlevel: FAILED
/usr/sbin/shutdown: FAILED
/usr/sbin/suexec: FAILED
/usr/sbin/telinit: FAILED
/usr/sbin/udevadm: FAILED
/usr/sbin/vgcfgbackup: FAILED
/usr/sbin/vgcfgrestore: FAILED
/usr/sbin/vgchange: FAILED
/usr/sbin/vgck: FAILED
/usr/sbin/vgconvert: FAILED
/usr/sbin/vgcreate: FAILED
/usr/sbin/vgdisplay: FAILED
/usr/sbin/vgexport: FAILED
/usr/sbin/vgextend: FAILED
/usr/sbin/vgimport: FAILED
/usr/sbin/vgimportclone: FAILED
/usr/sbin/vgimportdevices: FAILED
/usr/sbin/vgmerge: FAILED
/usr/sbin/vgmknodes: FAILED
/usr/sbin/vgreduce: FAILED
/usr/sbin/vgremove: FAILED
/usr/sbin/vgrename: FAILED
/usr/sbin/vgs: FAILED
/usr/sbin/vgscan: FAILED
/usr/sbin/vgsplit: FAILED
/usr/sbin/zdump: FAILED
/usr/sbin/zic: FAILED
/bin/ab: FAILED
/bin/bootctl: FAILED
/bin/busctl: FAILED
/bin/coredumpctl: FAILED
/bin/ea-php73: FAILED
/bin/ea-php74: FAILED
/bin/ea-php80: FAILED
/bin/ex: FAILED
/bin/gencat: FAILED
/bin/getconf: FAILED
/bin/getent: FAILED
/bin/hostnamectl: FAILED
/bin/htdbm: FAILED
/bin/htdigest: FAILED
/bin/htpasswd: FAILED
/bin/httxt2dbm: FAILED
/bin/iconv: FAILED
/bin/journalctl: FAILED
/bin/locale: FAILED
/bin/localectl: FAILED
/bin/localedef: FAILED
/bin/loginctl: FAILED
/bin/logresolve: FAILED
/bin/makedb: FAILED
/bin/pldd: FAILED
/bin/resolvectl: FAILED
/bin/rvi: FAILED
/bin/rview: FAILED
/bin/rvim: FAILED
/bin/sprof: FAILED
/bin/systemctl: FAILED
/bin/systemd-analyze: FAILED
/bin/systemd-ask-password: FAILED
/bin/systemd-cat: FAILED
/bin/systemd-cgls: FAILED
/bin/systemd-cgtop: FAILED
/bin/systemd-delta: FAILED
/bin/systemd-detect-virt: FAILED
/bin/systemd-escape: FAILED
/bin/systemd-firstboot: FAILED
/bin/systemd-hwdb: FAILED
/bin/systemd-inhibit: FAILED
/bin/systemd-machine-id-setup: FAILED
/bin/systemd-mount: FAILED
/bin/systemd-notify: FAILED
/bin/systemd-path: FAILED
/bin/systemd-resolve: FAILED
/bin/systemd-run: FAILED
/bin/systemd-socket-activate: FAILED
/bin/systemd-stdio-bridge: FAILED
/bin/systemd-sysusers: FAILED
/bin/systemd-tmpfiles: FAILED
/bin/systemd-tty-ask-password-agent: FAILED
/bin/systemd-umount: FAILED
/bin/timedatectl: FAILED
/bin/udevadm: FAILED
/bin/vi: FAILED
/bin/view: FAILED
/bin/vim: FAILED
/bin/vimdiff: FAILED
/bin/xmlcatalog: FAILED
/bin/xmllint: FAILED
/bin/xxd: FAILED
/sbin/build-locale-archive: FAILED
/sbin/dmeventd: FAILED
/sbin/dmfilemapd: FAILED
/sbin/dmsetup: FAILED
/sbin/dmstats: FAILED
/sbin/fcgistarter: FAILED
/sbin/halt: FAILED
/sbin/htcacheclean: FAILED
/sbin/httpd: FAILED
/sbin/iconvconfig: FAILED
/sbin/init: FAILED
/sbin/ldconfig: FAILED
/sbin/lvchange: FAILED
/sbin/lvconvert: FAILED
/sbin/lvcreate: FAILED
/sbin/lvdisplay: FAILED
/sbin/lvextend: FAILED
/sbin/lvm: FAILED
/sbin/lvmconfig: FAILED
/sbin/lvmdevices: FAILED
/sbin/lvmdiskscan: FAILED
/sbin/lvmpolld: FAILED
/sbin/lvmsadc: FAILED
/sbin/lvmsar: FAILED
/sbin/lvreduce: FAILED
/sbin/lvremove: FAILED
/sbin/lvrename: FAILED
/sbin/lvresize: FAILED
/sbin/lvs: FAILED
/sbin/lvscan: FAILED
/sbin/modsec-sdbm-util: FAILED
/sbin/nscd: FAILED
/sbin/poweroff: FAILED
/sbin/pvchange: FAILED
/sbin/pvck: FAILED
/sbin/pvcreate: FAILED
/sbin/pvdisplay: FAILED
/sbin/pvmove: FAILED
/sbin/pvremove: FAILED
/sbin/pvresize: FAILED
/sbin/pvs: FAILED
/sbin/pvscan: FAILED
/sbin/reboot: FAILED
/sbin/resolvconf: FAILED
/sbin/rotatelogs: FAILED
/sbin/runlevel: FAILED
/sbin/shutdown: FAILED
/sbin/suexec: FAILED
/sbin/telinit: FAILED
/sbin/udevadm: FAILED
/sbin/vgcfgbackup: FAILED
/sbin/vgcfgrestore: FAILED
/sbin/vgchange: FAILED
/sbin/vgck: FAILED
/sbin/vgconvert: FAILED
/sbin/vgcreate: FAILED
/sbin/vgdisplay: FAILED
/sbin/vgexport: FAILED
/sbin/vgextend: FAILED
/sbin/vgimport: FAILED
/sbin/vgimportclone: FAILED
/sbin/vgimportdevices: FAILED
/sbin/vgmerge: FAILED
/sbin/vgmknodes: FAILED
/sbin/vgreduce: FAILED
/sbin/vgremove: FAILED
/sbin/vgrename: FAILED
/sbin/vgs: FAILED
/sbin/vgscan: FAILED
/sbin/vgsplit: FAILED
/sbin/zdump: FAILED
/sbin/zic: FAILED
/usr/local/bin/ea-php73: FAILED
/usr/local/bin/ea-php74: FAILED
/usr/local/bin/ea-php80: FAILED
So immediately I thought, there must have been a cPanel update that executed during the night. So I proceed to check the /var/cpanel/updatelogs/summary.log file and see that the last update ran on 1 March 2022 which was 17 days ago.

[[email protected]#### updatelogs]# tail -5 summary.log
[2021-12-03 02:15:08 +0200] Completed update 11.100.0.4 -> 11.100.0.5
[2022-01-21 02:29:57 +0200] Completed update 11.100.0.5 -> 11.100.0.7
[2022-02-04 02:15:00 +0200] Completed update 11.100.0.7 -> 11.100.0.9
[2022-02-23 02:14:22 +0200] Completed update 11.100.0.9 -> 11.100.0.10
[2022-03-01 02:14:17 +0200] Completed update 11.100.0.10 -> 11.100.0.11
Then I thought, maybe it was my almalinux that has auto update enabled for it, so I logged in to my almalinux and found that auto updates are off.

However, Almalinux wants to restart a bunch of services as though something did update and of these services, a bunch of them are cPanel services. So why did cPanel do a bunch of changes that has not been logged???

Screenshot_1.png

So now I am left wondering what the heck is going on. The above lfd log looks like almost the entire servers file integrity has changed somehow. Can someone give me any ideas of why cPanel made changes and didn't log it in the updates summary ??
 
Last edited:

BlueSteam

Well-Known Member
Feb 21, 2013
129
21
68
cPanel Access Level
Reseller Owner
So looking at the WHM interface I was faced with this notice:

Screenshot_2.png

but for some strange reason, there was nothing in my updates summary.log

hmmm....server reboot commencing
 

Spirogg

Well-Known Member
Feb 21, 2018
696
151
43
chicago
cPanel Access Level
Root Administrator
Hello,

This morning I was greeted with the following lfd log output check from CSF.



So immediately I thought, there must have been a cPanel update that executed during the night. So I proceed to check the /var/cpanel/updatelogs/summary.log file and see that the last update ran on 1 March 2022 which was 17 days ago.



Then I thought, maybe it was my almalinux that has auto update enabled for I logged in to my almalinux web interface for the server and found that auto updates are off.

However, Almalinux wants to restart a bunch of services as though something did update and of these services, a bunch of the are cPanel services. So why did cPanel do a bunch of changes that has not been logged???

View attachment 76333

So now I am left wondering what the heck is going on. The above lfd log looks like almost the entire servers file integrity has changed somehow. Can someone give me any ideas of why cPanel made changes and didn't log it in the updates summary ??
This is most likely an update. there are nightly updates that happen like RPM updates and yum updates via cron from cPanel
I just received for 2 server one was around 1;51 am the email and the other was at 4am for the other server.

if your running Almalinux the log file is at /var/log/dnf.log and dnf.rpm.log
other centOS should be at /var/log/yum.log

here is a way to check each one that shows FAILED.

example: to check /usr/bin/htpasswd and see what pkg it belongs too:

run the command below you can do this for each one to make sure.
find the file paths that failed in your email - log from LFD

So one of them I have is
/usr/bin/htpasswd
I then run the code below
Code:
[[email protected] ~]# rpm -qf /usr/bin/htpasswd
ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64
--
So now I know it's in ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64
So to check when it updated I use this below grep filename /path/to/log.log
Code:
[[email protected] ~]# grep ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64 /var/log/dnf.rpm.log
2022-03-16T17:57:47-0500 SUBDEBUG Upgrade: ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64
2022-03-18T01:50:09-0500 SUBDEBUG Upgraded: ea-apache24-tools-2.4.53-1.1.1.cpanel.x86_64
--
Another example:
Code:
[[email protected] ~]# rpm -qf /usr/sbin/suphp
ea-apache24-mod_suphp-0.7.2-30.34.9.cpanel.x86_64
--
then check when it's been updated last:
Code:
[[email protected] ~]# grep ea-apache24-mod_suphp-0.7.2-30.34.9.cpanel.x86_64 /var/log/dnf.rpm.log
2022-03-16T17:57:53-0500 SUBDEBUG Upgrade: ea-apache24-mod_suphp-0.7.2-30.34.9.cpanel.x86_64
2022-03-18T01:50:09-0500 SUBDEBUG Upgraded: ea-apache24-mod_suphp-0.7.2-30.34.9.cpanel.x86_64
you can also use this pkg rpm checker from cPanel this will check if anything is wrong with any pkgs. I just found this a read a little on it.

please read the [OPTIONS] section for more details before running any checks

- there is a way to check md5sum
- but I am not sure where to get the latest from cPanel to verify md5sum.

I'm sure there is a way I just can't find it.

maybe @cPRex can give us some inside info on where we can double check if we want the MD5 checksum for each pkg when we get this email notification.


hope this helps ease your mind or not :)

Spiro
 
  • Like
Reactions: kodeslogic

Spirogg

Well-Known Member
Feb 21, 2018
696
151
43
chicago
cPanel Access Level
Root Administrator
Also I forgto to check md5 checksum
you can run this command for each failed

example
Code:
[[email protected] ~]# md5sum /usr/bin/htpasswd
e648faf395affeec7ce227d55604df3a  /usr/bin/htpasswd
another example
Code:
[[email protected] ~]# md5sum /usr/sbin/suphp
b0067b3b2f6f9542fe518f6c12329e2e  /usr/sbin/suphp
as I mentioned above maybe @cPRex can tell us where we can double check these to make sure there the same as cPanel's

have great morning.

Spiro
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
10,360
1,631
363
cPanel Access Level
Root Administrator
Currently the best way is to manually pull the file from our mirror and compare the MD5. We have some more thoughts on that here: