Server Intrusion: The login shell has changed from '/usr/local/cpanel/bin/jailshell' to '/bin/bash'

[dragonsway]

Within 24 hours of purchasing my first cPanel/whm license to begin my webhosting adventure, I was of course, hacked. :-/ Oh well, gotta learn, backups are good

I 03/06/2021, my server showed symptoms of being hacked. In my rkhunter.log file I have the following warning:


Warning: Changes found in the passwd file for user 'username':
[03:47:48] The login shell has changed from '/usr/local/cpanel/bin/jailshell' to '/bin/bash'

This warning came just after, I imported my cpanel backup from hostgator, which was "infected to gills" when I used the cpanel "virus scan to get rid of all the nastiness.

I assume that this was a malicious file change. Question: what is the best method of undoing this change? ( I used yum reinstall passwd. Just verifying that I am correct)

At the same time, I received a warning that /usr/local/cpanel/bin/jail_safe_passwd had been changed as well. The sha256sum that I currently have is:
73cfe7fd96f2a266c6d68c414ba5128558282f2735c57280b3380ca2b1422961

Is there a repository of the cpanel checksums that I can readily access from the web as to do my own comparisons?
If the above hash is not correct.. How do I replace the file with the correct file?


Other symptoms of the hack are / was as follows:

url spoofing (tried to spoof me to extort bitcoin, funny). In my DNS Zone Manager, I found added SPF entries containing:

"v=spf1 include:spf.mailgun.com +a +mx +ip-MYIP- +ip4:108.167.157.86 +ip4:199.250.207.223 ~all"

Question: I also saw / see many different DKIM records. Is it safe / best practice to delete all of domain keys that I don't recognize? (meaning, will I break something if I delete all of the domain keys, -EXCEPT- the domain key from mailgun that I recognize?

(I have since enable dnssec entries / transfer protections with my registrar.. and blocked the two ip addresses via csf)


Quite annoyingly, I started get rapid fire email alerts from cpanel monitoring, without any body information, just a subject line:
[hostname] FAILED : cpanel-ccs (MYIP)

So I, temporarily disabled the email alerts (enabled now), and ran # maldet -a found and cleaned:

LMD GNU nano 2.3.1 File: /usr/local/maldetect/sess/session.210309-0732.621

HOST: hostname
SCAN ID: 210309-0732.621
STARTED: Mar 9 2021 07:32:21 +0800
COMPLETED: Mar 9 2021 08:39:02 +0800

FILE HIT LIST:
{HEX}php.base64.inject.180 : /home/user/logs/modsec2_user_Mar_2021.gz
===============================================
Linux Malware Detect v1.6.4 < [email protected] >

And, the the deluge of warning emails from cpanel monitoring all stopped.

I also follow the instructions here for cleaning up infected websites.
- I had 2 outdated wp sites and two old drupal sites. I updated and change passwords for the wp sites and outright deleted the drupal sites since I keep clean offline backups of sites.
- I don't have any ftp accounts beyond the special accounts.


Finally, I went through and changed all of my passwords (cpanel, emails, etc.)

Are there any other best practices and/or guides for responses to an intrusion on your server that I should know?

 

[andrew.n]

Have you used restricted restore when you transferred the account?
Do you have any other accounts on the server?

In WHM under ImmunifyAV you will also see if there is any infected files found under the account you transferred.

I do not believe that an infected account could compromise the full server in a very short time and I believe that's the case here. The DKIM entries and such could be the results of the transfer and those records could have been there on the HostGator side as well. I suggest you to reach out to a cPanel Certified Professional from System Administration Services who can have a closer look at your server and see what changes has been made exactly.

 

[cPRex]

Hey there! It sounds like the compromises may have been present in the backup you restored, but it's normal to get notifications about shell or other settings changing when an account is restored. The passwd files would also update as the users are created and modified, so that seems normal to me as well.

If the original backup was infected, it may be better to upload clean copies of the site files to the server instead of restoring the account, but working with a professional system administrator would be the best way to confirm there are no additional problems lingering in the domain's content.

I would recommend deleting any of the DKIM or DNS records that you don't recognize, such as additional IPs in the SPF record, although multiple IPs can be present there as part of the migration/restore.

It really sounds like most of the issues were present in the backup, although after it was restored that certainly could have allowed the attacker access to the new machine. By default, cPanel is as secure as possible from the time it is installed, so I don't have any specific recommendations on that part on my end.

I think the best thing to do would be to work with a security professional if you think there could be any evidence of the compromise on the new account.

 

[dragonsway]

thanks for the replies. @andrew.n Thanks, but the entire reason I purchased a solo license was to give myself time to learn the basics of protecting a webserver with being dependent on a 3rd party, every time the wind blows becuase obviously that gets expensive. i keep good offline backups for specifically this reason. @cPRex, your advice about deleting DKIM that I don't recognize makes sense. thx.

Yesterday I went through and cleaned everything as mentioned above, but off course the intruder is back. It that the intruder has ready SSH access and took a moment to turn my SSH access. I am doing a quick server rollback, but after I do a server rollback and clean everything. What is the best way to verify that only I have SSH access?

Also FYI, I restore the cpanel backup as restricted and immediately cleaned everything and ran maldet. The intrusion began after I installed softaculous. So I am rolling back to before softaculous cleaning everything again.

 

[dragonsway]

Question: if I change the port number and enable 2fa in whm.. that should take care of my intruder problem, correct?

 

[andrew.n]

If your server has been compromised you should reinstall the full server to make sure there is no other exploits on the system and then start from hardening the server and once you are done you can try to transfer the account and restore it using the restricted restore option. Maybe even before it would be good to run a malware scan on the account to ensure nothing harmful is being restored. Immunify360 or even ImmunifyAV is a good way to do the scan.

If your server has already been compromised 2FA and SSH port change won't do much I assume but it's hard to say how well the server is compromised without having a closer look

 

[cPRex]

Is the SSH port number the default? If so, changing the SSH port number to something other than 22 is the best way to keep that service secure.

If the server has been root compromise, the only sure way to get it clean is to reinstall the entire OS and start fresh.

 

[dragonsway]

I may be new to webhosting, but I am good at figuring things out... As a safeguard, I built a test server with virtualbox same centos / same cpanel&whm. And I also installed rkhunter on test & prod. This way I can know if any files have actually been tampered with or whether any hidden directories or ports have been added. Happy to say, that everything checks out 100%. I am fairly sure that deleting the hacked site in it entirety, all my ftp accounts, and changing my port number again (right from I didn't use a default port for root, I'm paranoid) essentially took care of my invader problem. I already ran maldet multiple times and cleaned <{HEX}php.base64.inject.180 : /home/user/logs/modsec2_user_Mar_2021.gz> I am changing all my email passwords and cleaning dns records now. Anything else you fellas might advise? FYI... I expected the cpanel file to be infected coming from hostgator, so I restored with the restricted option... still had this mess. This is why I hardened the server according to documentation prior to the install and setup the test server.

 

[dragonsway]

One last general thing... just want to double-check.

Can someone confirm if the following files/apps are installed by cpanel & whm

/usr/sbin/httpd
/usr/sbin/mysqld
/usr/local/cpanel/libexec/cpdavd-dormant
/usr/sbin/dovecot
/usr/libexec/dovecot/pop3-login
/usr/libexec/dovecot/imap-login
/usr/local/cpanel/3rdparty/sbin/p0f <--- not sure about that
usr/bin/postgres

 

[cPRex]

Yup - those are all us and part of the default system.

 

[dragonsway]

Ok... something strange.

Can you confirm that this harmless? In my paranoia. I installed and ran chkroot. I got a well confirmed false positive for /usr/bin/passwd and the following in the chkroot results:

Checking `chkutmp'... The tty of the following user process(es) were not found
in /var/run/utmp !
! RUID PID TTY CMD
! 29793 t.jar s+ /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDatchkutmp: nothing deleted
Checking `OSX_RSPLUG'... not tested


saw a similar thread here, but for lack of privileges couldn't reply in the thread:

Chkrootkit:Checking `chkutmp'... The tty of the following user process(es) were not

Hi, Today i notice that my chkrootkit test result give me strange message : Checking `chkutmp'... The tty of the following user process(es) were not found in /var/run/utmp ! ! RUID PID TTY CMD ! root 29541 pts/0 /usr/bin/perl /scripts/realperlinstaller version...

forums.cpanel.net

 

[dragonsway]

Also, quite consistently the visits of my intruder cocincide with alerts about cpanel_ccs errors (link below). Perhaps the intruder is somehow exploiting this plugin? Also, each time the intruder visits and the cpanel-css errors start, /etc/sysconfig/network-scripts/ifcfg-eth0 also gets changed to FROM static to dhcp, so I have been manually putting it back to static. At this point, I have compared & contrasted rkhunter & chkrootkit on test & prod. I am fairly sure that there is nothing malicious on my system. I just don't understand the consistent access when I am changing ssh ports, deleted all ftp and websites, changed passwords..

cpanel_ccs Service down and not launching

I think I've narrowed down the issue, which I've bolded, but everytime I try to run restartsrv_cpanel_ccs, the following error occurs. Waiting for “cpanel_ccs” to start ……info [restartsrv_cpanel_ccs] systemd failed to start the service “cpanel-ccs” (The “/usr/bin/systemctl restart...

forums.cpanel.net

Also can you provide a dumb-downed version of the above instructions on how to repair this problem? thanks

 

[dragonsway]

Since, the cpanel_css is experimental and seems to be vulnerable, how vital is it? could I replace it with something like nextcloud?

 

[cPRex]

I think the chkutmp is also a false positive.

As far as CCS, you could disable the plugin with the details here:

Calendar and Contacts Server | cPanel & WHM Documentation

The Calendar and Contacts Server (CCS) allows your cPanel users to manage their CalDAV (calendar) and CardDAV (contacts) accounts from your cPanel & WHM server.

docs.cpanel.net

It still sounds like something odd is happening on the machine. What type of notices are you receiving about this intruder?