Server Hacked - Found out how

ramzex · Jul 9, 2009

Hello,

We just got on of our servers hacked.
Seems that first hackers used XSS exploits to upload some scripts.
We secured that with latest mod_sec rules from gotroot.com
Unfortunately we did not find those uploaded scripts.
And now when they cannot use XSS anymore they used those scripts to find the users and passwords from the server.
They listed users from /var/mail and changed the passwords of account.
Then they connected to ftp and uploaded/deleted files from the other accounts.
Also they inserted iframes in others.

I am installing suhosin now and put php in safe_mode for now and disabled functions: exec, popen, pclose, ini_set

Also they have a perl script that can make symlinks to other accounts: they used the function symlink() from perl.
How can I disable that for perl?

I will update you on how it's going and you are welcome to let me know some tips on how to secure it better

PS: the script name is EgY SpIdEr ShElL

sparek-3 · Jul 9, 2009

Was the server hacked (i.e. root compromised / entire server hacked) or is this an account-level hack, where just one account or a handful of accounts were hacked?

ramzex · Jul 9, 2009

I do not think root was compromised.
I checked the cpanel login logs and there were thousands of failed logins.
Seems that they actually used brute-force.
But since the logins were attempted from localhost Brute-Force protection from cpanel didn;t banned them as they are automatically whitelisted or it does not even go trough it.

sparek-3 · Jul 9, 2009

The account that was hacked was probably running an old script, such as an old version of Wordpress or an old version of Joomla, which malicious users exploited to gain access.

Or the owner of this account has a virus/trojan/keylogger installed on their computer which is stealing their username and password and sending that information to hacker groups.

ramzex · Jul 9, 2009

sparek-3 said: ↑

The account that was hacked was probably running an old script, such as an old version of Wordpress or an old version of Joomla, which malicious users exploited to gain access.

Or the owner of this account has a virus/trojan/keylogger installed on their computer which is stealing their username and password and sending that information to hacker groups.
Click to expand...

Yes. It most likely.
Seems that they hacked about 50 websites with those scripts.
It is unbelievable how many things you can do with php running in suphp, with open_basedir in effect and other security.
It's like having no security at all.

ramzex · Jul 9, 2009

I have installed suhosin but it did not did protect the server against php vulnerabilities.

You can still see and edit files from /etc.
I have disabled these functions in php:

exec,popen,pclose,ini_set,php_eval,safe_dir,zend,glob,root,chdir,ftok,posix_access,egy_perl,symlink,set_time_limit,ini_restore, shell_exec, passthru, error_log, ini_alter, dl, openlog, syslog, readlink, symlink, link, leak, popen, escapeshellcmd,proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, escapeshellarg, pcntl_exec, exec, passthru, popen, wscript, curl_exec,apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, eval, exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_alter, ini_get_all, ini_restore, inject_code, mysql_pconnect, openlog, passthru, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, popen, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec, syslog, system, xmlrpc_entity_decode,realpath
Click to expand...

Also it seems that php can include files from /etc.... and other dirs.
Why? phpsuexec and suphp are enabled. Also open basedir from cpanel security is enabled.

Why and how can I prevent that?

Spiral · Jul 9, 2009

Ramzex,

I am getting ready to head out the door so I cannot stay and chat
much at the moment but I definitely want to get back to you and
discuss your current situation. In the meantime, I invite you to look
at my previous posts because there is another user on here (sphost)
I just responded to not 5 minutes ago and I think you need to read
that entire thread as you are in a very similar situation and I may be
able to help you out quite a bit with your situation as well.

Regarding everything you listed in your post above, you are definitely
making all the right moves but I'm not seeing a number of things in the
your list of things you did to secure your server and I want to ask you
about some of those as you may have secured your server well in some
areas while leaving big gaps in other areas.

Please contact me by private message and we'll chat more when
I get back online in a couple of hours and I'll try to help you with
your situation and recovering from this mess.

-Spiral

PS: You may be just a little bit overkill on the "disable_functions" there
and regarding open_basedir, it doesn't work the same way under SuPHP
so just enabling it in Cpanel won't really help much there.

ramzex · Jul 15, 2009

We have investigated this issue and found the following:

Problem:

1. A php shell script (which contain numerous php/apache/zend vulnerabilities) has been uploaded trough a XSS attack.

2. Script has been used to gather usernames from the servers.

3. Script has modified the passwords of the accounts located in /etc/passwd

4. Hackers connected from different IPs to the FTP accounts and uploaded/deleted files.

Solution:

1. Upgrade to Apache 2.2 with latest PHP versions (currently 5.2.10)! (a must)! and compile with suhosin, suphp, suexec!

2. Install mod_Security from cpanel addons!

3. Install mod_security rules from gotroot.com (they have a free rules download also).

4. Install clamv addon from cpanel.

5. Forbid the following functions in php:

exec,popen,pclose,ini_set,php_eval,safe_dir,zend,g lob,root,chdir,ftok,posix_access,egy_perl,symlink, set_time_limit,ini_restore, shell_exec, passthru, error_log, ini_alter, dl, openlog, syslog, readlink, symlink, link, leak, popen, escapeshellcmd,proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, escapeshellarg, pcntl_exec, exec, passthru, popen, wscript, curl_exec,apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, eval, exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_alter, ini_get_all, ini_restore, inject_code, mysql_pconnect, openlog, passthru, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, popen, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec, syslog, system, xmlrpc_entity_decode,realpath
Click to expand...

Please note that some functions like realpath or chdir may be used by some websites.

5. Enable FTP TLS Encryption Support as Required!

6. Change your SSH port to something else.

7. Enable Brute-Force protection.

8. Install firewall.

9. Configure SUHOSIN so it will disable eval function (note that this is a good protection but many legit scripts use this function and could cause issues. but you can whiteliste and blacklist accounts that can use this function - see suhosin docs)

10. Update kernel to latest version. Seems that Linux Kernel 2.6.18-128.1.6 which we had on CentOS 5.3 is vulnerable.

We found that the shell scripts uploaded were base64 encoded.

Use this search command in ssh to find files that are base64 encoded and take a look at them as they may be backdoors:

find /home -type f -print0 -name "*.php*" | xargs -0 grep -l "eval(gzinflate(base64_decode"
Click to expand...

Replace "/home" with your path.

Also find files that are using php command: "posix_getpwuid" as this is how they list the server's usernames!

There are other vulnerabilities with zend also!
Even if you enable Safe Mode in PHP they can still list /etc/passwd or any other system file even though Open_basedir restriction is enabled.
We are still investigating this and I will update you as soo as we have a solution.

Also we found another Perl script that came with the shell code above.
It uses the symlink() function to create symlink into vulnerable account to any other account or directory in server. this way they have access to everything.

If someone has more ideas how to secure the server againts these vulnerabilities please let us know.

I will also keep you updated.

Thanks.

PS: I can provide the shell scripts used to hack the server. They contain latest exploited vulnerabilities. Just PM me.

jols · Aug 6, 2009

Thanks for this. Quick questions:

If you include "zend" in the disable list, won't this mean that zend could no longer be used?

Same question for the above regarding using cURL and disabling "curl_exec"?

NOTE: "ini_restore" is in your list twice, as is "popen", and "exec", and "passthru", and "proc_open", and "symlink" and probably a few others. (popen was in there at least three times).

Also, I have heard that the following should also be in the disable list:

show_source, phpinfo, allow_url_fopen

Here's this finished list, with a few things taken out (that may possibly disrupted legit scripts), and the few things added in from the list just above:

disable_functions = phpinfo, allow_url_fopen, exec, popen, pclose, ini_set, php_eval, safe_dir, g lob, root, ftok, posix_access, egy_perl, symlink, set_time_limit, ini_restore, shell_exec, passthru, ini_alter, dl, openlog, syslog, readlink, link, leak, escapeshellcmd, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, pcntl_exec, wscript, curl_exec, apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_get_all, inject_code, mysql_pconnect, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_terminate, system, xmlrpc_entity_decode

REMOVED:
zend
eval
error_log
curl_exec
realpath
chdir
and most, or all of the duplicates.

ADDED:
show_source
phpinfo
allow_url_fopen

Comments? Please correct me if I'm wrong with any of this. Thanks.

hidonet · Aug 7, 2009

There is a solution

http://forums.cpanel.net/f7/solution-gumblar-iframe-javascript-hacks-stolen-ftp-passwords-127013.html#post552453

Specks · Aug 7, 2009

Don't forget to disable root logins from SSH. That could be a big help.

mambovince · Aug 9, 2009

Does disabling shell_exec, and installing Suhusin help to stop shell scripts from running?
Thanks,

- Vince

mambovince · Aug 11, 2009

Anyone?
Thanks,

- Vince

userenabled · Aug 12, 2009

ramzex said: ↑

I do not think root was compromised.
I checked the cpanel login logs and there were thousands of failed logins.
Seems that they actually used brute-force.
But since the logins were attempted from localhost Brute-Force protection from cpanel didn;t banned them as they are automatically whitelisted or it does not even go trough it.
Click to expand...

I've also had issues with brute force attacks. But never from the inside.

hidonet · Aug 14, 2009

if you don't have CSF, Try it. Blocks all bruteforcers...

Forums

The Community Forums

Interact with an entire community of cPanel & WHM users!

Server Hacked - Found out how

ramzex Member

sparek-3 Well-Known Member

ramzex Member

sparek-3 Well-Known Member

ramzex Member

ramzex Member

Spiral BANNED

ramzex Member

jols Well-Known Member

hidonet Well-Known Member

Specks Well-Known Member

mambovince Well-Known Member

mambovince Well-Known Member

userenabled Registered

hidonet Well-Known Member

Server hacked using pam_fprintd.so?

New Thread The server quit without updating PID file

New Thread Setting up new server question

Slow Dedicated Server Booting

Saving Default Reseller Nameservers and Privileges

Share This Page

Useful Searches

The Community Forums

Interact with an entire community of cPanel & WHM users!

Server Hacked - Found out how

ramzex Member

sparek-3 Well-Known Member

ramzex Member

sparek-3 Well-Known Member

ramzex Member

ramzex Member

Spiral BANNED

ramzex Member

jols Well-Known Member

hidonet Well-Known Member

Specks Well-Known Member

mambovince Well-Known Member

mambovince Well-Known Member

userenabled Registered

hidonet Well-Known Member

Server hacked using pam_fprintd.so?

New Thread The server quit without updating PID file

New Thread Setting up new server question

Slow Dedicated Server Booting

Saving Default Reseller Nameservers and Privileges

Share This Page