/etc directory mucked...lots of residual effects

gsbe

Active Member
Jul 27, 2003
39
0
156
Nashville, TN
I am experiencing a problem in the /etc directory on a CPanel server. The system has been running sluggishly, Apache has difficulty restarting, and errors are popping up in the emailed logs. The following files are suspected to have been overwritten accidently with another system's files in the /etc directory on Aug 12 at approximately 17:15 CST:

Code:
DIR_COLORS
aliases
antivirus.exim
bashrc*
exim.conf
exim.pl
group
host.conf
inputrc
ld.so.cache
ld.so.conf
localdomains
localtime
man.config
mtab
my.cnf
nsswitch.conf
passwd
profile
profile.d/
protocols
resolv.conf
services
shadow
termcap
userdomains
History:
  1. files listed above in /etc directory copied over accidently on Aug 12 at approximately 17:15. Source of files copied from off-site.
  2. clients begin notifying us of problems logging into CPanel and email at approximately 18:15 on Aug 12
  3. I am able to restore etc/exim.conf from a backup via WHM in the early morning of August 13 which gets email functioning. I also restore a backup copy of /etc/vmail that morning to solve additional problems logging into CPanel and errors checking email accounts.
  4. NOC techs step in when I can't get Apache to start (resolved 06:28 on Aug 13). NOC techs found "more than a couple VirtualHost entries containing Group directives for groups that didn't exist on the box." To solve this problem, the NOC techs copied the existing /etc/group to /etc/group.sb and replaced the existing /etc/group with a backup called /etc/group-. This got Apache restarted and they ran /scripts/fixeverything which found a "number of problems" and fixed them.
Known issues:
  1. Quotas are not working at all within WHM/CPanel. The quotas per user are stored somewhere because when you try to reset them via WHM it shows the correct values but they are returned to unlimited when you choose "List All Accounts" in WHM. Although /etc/fstab doesn't appear in the list above I see that 'usrquota' could be set incorrectly there.
  2. The machine is definitely sluggish. exim and spamd frequently use all of the machine's cpu power when monitoring with top.
  3. We received an alert from olemiss.edu that they were adding us to their blocked IP spam list because their Cisco Intrusion Detection System found that our server was the root of MyDoom Virus Activity and was going to be blocking our IP for 24-hours.
  4. daily emails from the server with "run-parts /etc/cron.daily" in the subject says that bindshell is infected. This has been showing up for months in this email log and a quick Google search proves this to most likely be a false positive. Exact line from email is:
    Code:
    `bindshell'... INFECTED (PORTS:  465)
  5. daily security emails from the server with the subject "[hackcheck] fileutils failed checksum test" says that /etc/DIR_COLORS has been modified as suggested above:
    Code:
    	IMPORTANT: Do not ignore this email.
    	This message is to inform you that the rpm 
    	package fileutils did not match the expected checksum.  This could mean that 
    	your system was compromised (OwN3D). The offending files have been removed 
    	and replaced with the OS default.  To be safe you should verify that your 
    	system has not be compromised.
    	
    	Modified Files:
    	S.5....T c /etc/DIR_COLORS
  6. daily script shows errors in email from server with "/scripts/upcp" in the subject saying:
    Code:
    	error: failed to stat /home2/godesign: No such file or directory
    	Preparing...                ##################################################
    	fileutils                   ##################################################
    	httpd                       #warning: /etc/httpd/conf/httpd.conf created as /etc/httpd/conf/httpd.conf.rpmnew
    	warning: /etc/httpd/conf/magic created as /etc/httpd/conf/magic.rpmnew
    	#################################################
    	error: unpacking of archive failed on file /etc/httpd/logs: cpio: rename failed - Invalid argument
  7. When investigating the differences between /etc and /etcbaker (a backup of /etc from Aug 13 at 00:30) I come across differences in these files, many are likely due to new domains being added:
    Code:
    	 /etc/cpbackup.conf
    	 /etc/exim.conf (restored from backup in History step 3)
    	 /etc/fstab.quotas (related to Known Issue #1?)
    	 /etc/group (restored from backup in History step 4)
    	 /etc/localdomains (likely due to new domains being added to system)
    	 /etc/proftpd.conf.tmpeditlib (?)
    	 /etc/quota.conf (related to Known Issue #1?)
    	 /etc/relayhosts (likely due to new domains being added to system)
    	 /etc/relayhostsusers (likely due to new domains being added to system)
    	 /etc/userdomains (likely due to new domains being added to system)
I've tried to be as complete as possible here. I've also sent this information to my NOC techs so that they can help determine our problems. I'm hoping that they may have a default install for their CPanel machines and they can replace the offending files and we'll be on our way (yeah right!). I'm concerned that some of these files that may have been changed may keep the machine from restarting properly. The user data is backed up on a remote machine, so we are covered there. No backup snapshots of our /etc directory, unfortunately!

WHM 9.4.0
cPanel 9.4.1-S65
RedHat 8.0
WHM X v3.1.0

Let me know if you need any more information and I'll provide it here. Any help would be much appreciated.