Very high load - httpdse?

[randomuser]

you got hacked. /usr/sbin/lsof -p pid and investigate.

edit: haha. bugs even in the cPanel forums, cute. My post was a reply to the post below, and was (obviously) not the first post. Wonder if I should open a bugzilla report about this.. NAH, I'll let them figure it out.

 

[itrends]

Can anybody explain what this is:

bt3# ps aux | grep perl
nobody 19846 13.6 0.4 5236 4544 ?? R 4:47PM 10:21.62 httpdse (perl5.8.6)
nobody 17918 13.5 0.3 4252 3580 ?? R 4:05PM 22:03.01 httpdse (perl5.8.6)
nobody 13669 13.4 0.4 4944 4248 ?? R 2:23PM 35:41.17 httpdse (perl5.8.6)
nobody 17848 13.0 0.3 4256 3588 ?? R 4:04PM 22:15.69 httpdse (perl5.8.6)
nobody 20221 13.0 0.3 4256 3592 ?? R 4:54PM 13:03.34 httpdse (perl5.8.6)
nobody 11494 12.8 0.3 4256 3584 ?? R 1:36PM 65:12.23 httpdse (perl5.8.6)
nobody 20147 12.5 0.3 4256 3592 ?? R 4:53PM 13:15.76 httpdse (perl5.8.6)
nobody 9519 12.3 0.4 4548 3852 ?? R 12:58PM 82:43.68 httpdse (perl5.8.6)
nobody 13636 12.3 0.4 4844 4148 ?? R 2:22PM 36:10.79 httpdse (perl5.8.6)
nobody 11597 12.2 0.3 4256 3588 ?? R 1:37PM 64:35.77 httpdse (perl5.8.6)
nobody 19869 11.6 0.4 5332 4636 ?? S 4:48PM 10:25.89 httpdse (perl5.8.6)
nobody 9475 11.4 0.4 4448 3752 ?? S 12:57PM 79:13.87 httpdse (perl5.8.6)
nobody 14969 5.8 0.4 5040 4344 ?? S 2:57PM 26:21.16 httpdse (perl5.8.6)
nobody 11236 5.4 0.4 4748 4052 ?? S 1:32PM 50:32.49 httpdse (perl5.8.6)
nobody 14991 5.3 0.4 5144 4448 ?? R 2:58PM 26:38.31 httpdse (perl5.8.6)
nobody 11197 3.2 0.4 4644 3952 ?? S 1:31PM 49:15.61 httpdse (perl5.8.6)
root 918 0.0 2.8 31056 29144 ?? I 11:04AM 1:26.31 spamd child (perl)


What is httpdse and WHY is he killing my server?

 

[itrends]

bt3# /usr/sbin/lsof -p 9475
/usr/sbin/lsof: Command not found.
bt3# lsof
lsof: Command not found.
bt3#

doesn't seem to work..

 

[randomuser]

bt3# /usr/sbin/lsof -p 9475
/usr/sbin/lsof: Command not found.
bt3# lsof
lsof: Command not found.
bt3#

doesn't seem to work..

# whereis lsof
# locate lsof

If you don't have it, you have 2 options:

1. install it (google)
2. dig around in /proc/pid/*

The things in /proc you will want to pay attention to are:

/proc/pid/exe
/proc/pid/cwd
/proc/pid/environ (# strings /proc/pid/environ -- easier to read than "cat")

 

[itrends]

tried to install lsof but won't work.. machine seems to be very scrwred..

gonna hit restart and try again..

 

[randomuser]

This is one advantage of using suexec / phpsuexec. Instead of seeing the processes as "nobody", you would see them as the user they're running as. So, if user "barkerb" has a php script that got owned, then the perl processes would show as running under his username, and not nobody. Immediately you know generally where the issue is, and can act accordingly. If you get lsof going, you'll have to wait for the processes to run again so you can do lsof -p. Give it time, it will probably happen again within 24 hours. In the meantime, grep around in /tmp for anything unusual, and /var/tmp (if you don't have it symlinked to /tmp already), as well as /dev/shm if it's permissions are (unnecessarily) 777.

 

[itrends]

lsof is installed and working.

I see nothin more then apache logfiles, the usual stuff. I have no idea what's causing the trouble?

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND
4926 nobody 128 0 5524K 4840K RUN 0 10:30 61.52% 61.52% perl5.8.6
4892 nobody 128 0 5428K 4744K RUN 0 10:13 57.13% 57.13% perl5.8.6
5039 nobody 125 0 5628K 4940K CPU1 1 10:12 54.88% 54.88% perl5.8.6


bt3# lsof -p 4892
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl5.8.6 4892 nobody cwd VDIR 4,14 512 2 /
perl5.8.6 4892 nobody rtd VDIR 4,14 512 2 /
perl5.8.6 4892 nobody txt VREG 4,18 10076 3486218 /usr/local/bin/perl
perl5.8.6 4892 nobody txt VREG 4,14 142236 19465 /libexec/ld-elf.so.1
perl5.8.6 4892 nobody txt VREG 4,18 1131524 3486749 /usr/local/lib/perl5/5.8.6/mach/CORE/libperl.so
perl5.8.6 4892 nobody txt VREG 4,14 120004 19441 /lib/libm.so.3
perl5.8.6 4892 nobody txt VREG 4,14 28644 19439 /lib/libcrypt.so.2
perl5.8.6 4892 nobody txt VREG 4,14 43100 19445 /lib/libutil.so.4
perl5.8.6 4892 nobody txt VREG 4,14 884716 19450 /lib/libc.so.5
perl5.8.6 4892 nobody txt VREG 4,18 17390 3486952 /usr/local/lib/perl5/5.8.6/mach/auto/IO/IO.so
perl5.8.6 4892 nobody txt VREG 4,18 27671 3487140 /usr/local/lib/perl5/5.8.6/mach/auto/Socket/Socket.so
perl5.8.6 4892 nobody 0r VCHR 2,2 0t0 15 /dev/null
perl5.8.6 4892 nobody 1u PIPE 0xc287bb2c 0 ->0xc287ba80
perl5.8.6 4892 nobody 2w VREG 4,18 157921844 3557499 /usr/local/apache/logs/error_log
perl5.8.6 4892 nobody 3u IPv4 0xc3be2534 0t0 TCP bt3.budgettrends.nl:59738->81-223-85-66.interxion.inode.at:socks (ESTABLISHED)
perl5.8.6 4892 nobody 15w VREG 4,18 157921844 3557499 /usr/local/apache/logs/error_log
perl5.8.6 4892 nobody 150w VREG 4,18 64754 3585158 /usr/local/apache/domlogs/ajaxfanz.nl-bytes_log
perl5.8.6 4892 nobody 152w VREG 4,18 4575716 3585058 /usr/local/apache/domlogs/bt3.budgettrends.nl-bytes_log
perl5.8.6 4892 nobody 153w VREG 4,18 133424 3559536 /usr/local/apache/logs/ssl_engine_log
perl5.8.6 4892 nobody 154w VREG 4,18 0 3558925 /usr/local/apache/logs/ssl_mutex.485
perl5.8.6 4892 nobody 292w VREG 4,18 0 3558925 /usr/local/apache/logs/ssl_mutex.485
bt3#

 

[randomuser]

Look closer:

perl5.8.6 4892 nobody 3u IPv4 0xc3be2534 0t0 TCP xxxxxxxxxxxxxxxxxxxxx:59738->81-223-85-66.interxion.inode.at:socks (ESTABLISHED)

Unless that's legit, someone is using your box to connect outbound to a proxy in that line..

edit: ircd, not a proxy.

 

[AndyReed]

I see nothin more then apache logfiles, the usual stuff. I have no idea what's causing the trouble?

You need to track down these hidden processes. The best thing you can do is secure and harden your server. There are many threads discussing server security in these forums.

 

[ramprage]

Yep, also grab yourself a copy of Nobody Check and have it scan your nobody and Apache processes daily to ensure they aren't fake or malicious hidden ones.

 

[itrends]

Darn't, i found it.. just had to reconfigure mod_security for a few words

 

[hitrss.com]

I'm having the same problems... what did you put in Mod_security to make sure this process will not start again?

Thanks

UPDATE: I tried to put many different rules in my mod_security file, but I still keep on getting the httpdse process every once in a while... and it's really annoying. Is there any rule you know works agains this crappy thing? It's abusing my server all the time!

 

[chirpy]

mod_security should really only be used to highight which scripts are being abused. You really need to track down the offending PHP scripts on the server and get rid of them, otherwise you'll continually be chasing after something from the wrong end.

 

[hitrss.com]

It's really hard to track down the script when all you see is "httpdse" and nothing more and you are hosting many web sites of many customers on one account and all of the scripts are PHP based.

Anyway, this thing is always writing something to my /tmp folder when it comes back. There is really no way of not allowing this thing to copy anything there? There must be a way to prevent this somehow...

Please help, anyone had experience with this thing?

 

[chirpy]

It's really hard to track down the script when all you see is "httpdse" and nothing more and you are hosting many web sites of many customers on one account and all of the scripts are PHP based.

If you don't know how, then you need to hire someone who does, otherwise you are never going to resolve the issue.

Please help, anyone had experience with this thing?

Yup, I've dealt with many servers where hackers have used that particular hacking script, and you have to track down the vulnerable PHP script that's on your server. If you do prevent this one script from being loaded, they'll simpl,y use the exact same exloit to upload one of the many different hacking scripts available to them - hence the need to fix the cause and not sticka flimsy bandage over it

 

[hitrss.com]

OK, you're right, I should find the script.

I guess I will have to ask somebody to help me here.

Until now, the moment I saw the thing eating up my CPU I killed it. Now I'm waiting for it to appear again....

As far as I understand this correctly, the moment it shows up, there should be some kind of track of it showing up in apache domain logs, right? I mean, it can't just show up without any trace, right? So that I can browse the domain logs, checking logs at the time it showed up in order to find the script it attacked, right?

Or do you know any keywords I should look for in the logs?

I'm a PHP coder, not administrator, but if I spot the script, I guess I could cure it right away.

Any help or any tricks/guidelines would help me a great deal and would be appreciated,

Thanks

- Michael

 

[chirpy]

OK, the best tool to use (though linux isn't very friendly in this) is lsof. When you see the process, get its PID and don't kill it just yet, instead use:

lsof -p PID

Look in particular at the first page of file descriptors. These usually hodl any information that you're going to get from the OS about the location of the file.

Unfortunately, it's simple for a script to obfuscate that information by changing its working directory. If that isn't helpful, then you're going to have to kill off the process and laboriously trawl through the domain logs in /usr/local/apache/domlogs/ and look for PHP activity in the last few minutes that could relate to the explit being loaded.

 

[hitrss.com]

Thanks for the advice,

I'm still waiting for the attack to commence but still nothing coming... they're afraid of me now

I will try the lsof command although I remember last time it didn't help me much....

I guess I will have to go through the dom logs then... but at least I will have the time it started.... (or maybe IP address it came from?)

If there were keywords I could search for in the logs...

Anyway, I'll get ready for the battle with the nasty PERL script! If I fail, I'll hire someone, but first I want to try myself and see if I can do it.

Should you have any more tips, please let me know - I'll be needing them.

Thanks,

- Michael