Server Hacked - Found out how

[ramzex]

Hello,

We just got on of our servers hacked.
Seems that first hackers used XSS exploits to upload some scripts.
We secured that with latest mod_sec rules from gotroot.com
Unfortunately we did not find those uploaded scripts.
And now when they cannot use XSS anymore they used those scripts to find the users and passwords from the server.
They listed users from /var/mail and changed the passwords of account.
Then they connected to ftp and uploaded/deleted files from the other accounts.
Also they inserted iframes in others.

I am installing suhosin now and put php in safe_mode for now and disabled functions: exec, popen, pclose, ini_set

Also they have a perl script that can make symlinks to other accounts: they used the function symlink() from perl.
How can I disable that for perl?

I will update you on how it's going and you are welcome to let me know some tips on how to secure it better

PS: the script name is EgY SpIdEr ShElL

[sparek-3]

Was the server hacked (i.e. root compromised / entire server hacked) or is this an account-level hack, where just one account or a handful of accounts were hacked?

[ramzex]

I do not think root was compromised.
I checked the cpanel login logs and there were thousands of failed logins.
Seems that they actually used brute-force.
But since the logins were attempted from localhost Brute-Force protection from cpanel didn;t banned them as they are automatically whitelisted or it does not even go trough it.

[sparek-3]

The account that was hacked was probably running an old script, such as an old version of Wordpress or an old version of Joomla, which malicious users exploited to gain access.

Or the owner of this account has a virus/trojan/keylogger installed on their computer which is stealing their username and password and sending that information to hacker groups.

[ramzex]

Originally Posted by sparek-3

The account that was hacked was probably running an old script, such as an old version of Wordpress or an old version of Joomla, which malicious users exploited to gain access.

Or the owner of this account has a virus/trojan/keylogger installed on their computer which is stealing their username and password and sending that information to hacker groups.

Yes. It most likely.
Seems that they hacked about 50 websites with those scripts.
It is unbelievable how many things you can do with php running in suphp, with open_basedir in effect and other security.
It's like having no security at all.

[ramzex]

I have installed suhosin but it did not did protect the server against php vulnerabilities.

You can still see and edit files from /etc.
I have disabled these functions in php:

exec,popen,pclose,ini_set,php_eval,safe_dir,zend,g lob,root,chdir,ftok,posix_access,egy_perl,symlink, set_time_limit,ini_restore, shell_exec, passthru, error_log, ini_alter, dl, openlog, syslog, readlink, symlink, link, leak, popen, escapeshellcmd,proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, escapeshellarg, pcntl_exec, exec, passthru, popen, wscript, curl_exec,apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, eval, exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_alter, ini_get_all, ini_restore, inject_code, mysql_pconnect, openlog, passthru, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, popen, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec, syslog, system, xmlrpc_entity_decode,realpath

Also it seems that php can include files from /etc.... and other dirs.
Why? phpsuexec and suphp are enabled. Also open basedir from cpanel security is enabled.

Why and how can I prevent that?

[Spiral]

Ramzex,

I am getting ready to head out the door so I cannot stay and chat
much at the moment but I definitely want to get back to you and
discuss your current situation. In the meantime, I invite you to look
at my previous posts because there is another user on here (sphost)
I just responded to not 5 minutes ago and I think you need to read
that entire thread as you are in a very similar situation and I may be
able to help you out quite a bit with your situation as well.

Regarding everything you listed in your post above, you are definitely
making all the right moves but I'm not seeing a number of things in the
your list of things you did to secure your server and I want to ask you
about some of those as you may have secured your server well in some
areas while leaving big gaps in other areas.

Please contact me by private message and we'll chat more when
I get back online in a couple of hours and I'll try to help you with
your situation and recovering from this mess.

-Spiral

PS: You may be just a little bit overkill on the "disable_functions" there
and regarding open_basedir, it doesn't work the same way under SuPHP
so just enabling it in Cpanel won't really help much there.

[ramzex]

We have investigated this issue and found the following:

Problem:

1. A php shell script (which contain numerous php/apache/zend vulnerabilities) has been uploaded trough a XSS attack.

2. Script has been used to gather usernames from the servers.

3. Script has modified the passwords of the accounts located in /etc/passwd

4. Hackers connected from different IPs to the FTP accounts and uploaded/deleted files.

Solution:

1. Upgrade to Apache 2.2 with latest PHP versions (currently 5.2.10)! (a must)! and compile with suhosin, suphp, suexec!

2. Install mod_Security from cpanel addons!

3. Install mod_security rules from gotroot.com (they have a free rules download also).

4. Install clamv addon from cpanel.

5. Forbid the following functions in php:

exec,popen,pclose,ini_set,php_eval,safe_dir,zend,g lob,root,chdir,ftok,posix_access,egy_perl,symlink, set_time_limit,ini_restore, shell_exec, passthru, error_log, ini_alter, dl, openlog, syslog, readlink, symlink, link, leak, popen, escapeshellcmd,proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, escapeshellarg, pcntl_exec, exec, passthru, popen, wscript, curl_exec,apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, eval, exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_alter, ini_get_all, ini_restore, inject_code, mysql_pconnect, openlog, passthru, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, popen, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec, syslog, system, xmlrpc_entity_decode,realpath

Please note that some functions like realpath or chdir may be used by some websites.

5. Enable FTP TLS Encryption Support as Required!

6. Change your SSH port to something else.

7. Enable Brute-Force protection.

8. Install firewall.

9. Configure SUHOSIN so it will disable eval function (note that this is a good protection but many legit scripts use this function and could cause issues. but you can whiteliste and blacklist accounts that can use this function - see suhosin docs)

10. Update kernel to latest version. Seems that Linux Kernel 2.6.18-128.1.6 which we had on CentOS 5.3 is vulnerable.

We found that the shell scripts uploaded were base64 encoded.

Use this search command in ssh to find files that are base64 encoded and take a look at them as they may be backdoors:

find /home -type f -print0 -name "*.php*" | xargs -0 grep -l "eval(gzinflate(base64_decode"

Replace "/home" with your path.

Also find files that are using php command: "posix_getpwuid" as this is how they list the server's usernames!

There are other vulnerabilities with zend also!
Even if you enable Safe Mode in PHP they can still list /etc/passwd or any other system file even though Open_basedir restriction is enabled.
We are still investigating this and I will update you as soo as we have a solution.

Also we found another Perl script that came with the shell code above.
It uses the symlink() function to create symlink into vulnerable account to any other account or directory in server. this way they have access to everything.

If someone has more ideas how to secure the server againts these vulnerabilities please let us know.

I will also keep you updated.

Thanks.

PS: I can provide the shell scripts used to hack the server. They contain latest exploited vulnerabilities. Just PM me.

[jols]

Thanks for this. Quick questions:

If you include "zend" in the disable list, won't this mean that zend could no longer be used?

Same question for the above regarding using cURL and disabling "curl_exec"?

NOTE: "ini_restore" is in your list twice, as is "popen", and "exec", and "passthru", and "proc_open", and "symlink" and probably a few others. (popen was in there at least three times).

Also, I have heard that the following should also be in the disable list:

show_source, phpinfo, allow_url_fopen

Here's this finished list, with a few things taken out (that may possibly disrupted legit scripts), and the few things added in from the list just above:

disable_functions = phpinfo, allow_url_fopen, exec, popen, pclose, ini_set, php_eval, safe_dir, g lob, root, ftok, posix_access, egy_perl, symlink, set_time_limit, ini_restore, shell_exec, passthru, ini_alter, dl, openlog, syslog, readlink, link, leak, escapeshellcmd, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, pcntl_exec, wscript, curl_exec, apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_get_all, inject_code, mysql_pconnect, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_terminate, system, xmlrpc_entity_decode



REMOVED:
zend
eval
error_log
curl_exec
realpath
chdir
and most, or all of the duplicates.

ADDED:
show_source
phpinfo
allow_url_fopen

Comments? Please correct me if I'm wrong with any of this. Thanks.

[hidonet]

There is a solution

http://forums.cpanel.net/f7/solution...tml#post552453

[Specks]

Don't forget to disable root logins from SSH. That could be a big help.

[mambovince]

Does disabling shell_exec, and installing Suhusin help to stop shell scripts from running?
Thanks,

- Vince

[mambovince]

Anyone?
Thanks,

- Vince

[userenabled]

Originally Posted by ramzex

I do not think root was compromised.
I checked the cpanel login logs and there were thousands of failed logins.
Seems that they actually used brute-force.
But since the logins were attempted from localhost Brute-Force protection from cpanel didn;t banned them as they are automatically whitelisted or it does not even go trough it.

I've also had issues with brute force attacks. But never from the inside.

[hidonet]

if you don't have CSF, Try it. Blocks all bruteforcers...