The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Possibly Exploited

Discussion in 'General Discussion' started by AlexKitch, Dec 6, 2005.

  1. AlexKitch

    AlexKitch Member

    Joined:
    Jul 9, 2004
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Hi

    OK... Yesterday I found my server *seriously* crawling, so I checked the process list in WHM and killed a load of MySQL processes, which seemed to fix the problem.

    Hours later, I got home and found my datacenter had unplugged the server, apparently due to an exploit in the binary /tmp/conn (a port scanning exploit). Also, before it was unplugged, cPanel sent me a CPU warning about high CPU load processing the log file for a particular user.

    The same thing has happened roughly 24 hours later, although this time I was able to stop the server dying completely by rebooting it.

    I've absolutely no idea what's going on, I even secured my /tmp folders. Today's crash seemed to be as a result of a log check process using tonnes of CPU.



    It's all a big problem because the server is a VPS, and the datacenter are very strict if the box starts to hammer the host machine's CPU :(

    I need ot find out if I'm infected, and if so - what to do about it. Any ideas, anybody? :)
     
    #1 AlexKitch, Dec 6, 2005
    Last edited: Dec 6, 2005
  2. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    Securing /tmp is definitely not enough. You need to secure your server properly. Did you check the log files for errors. Did you clean up, remove the downloaded and installed hacking tools? Did you suspend the Web site of the user with the culprit script?
    Just in case, security issues were discussed so many times here. Search is your best option.
     
    #2 AndyReed, Dec 6, 2005
    Last edited: Dec 8, 2005
  3. AlexKitch

    AlexKitch Member

    Joined:
    Jul 9, 2004
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    I can't find any suspect executables/binaries on the server at all. My datacenter warned me that I should investigate tmp/conn - this binary caught their scanner's attention

    The user (who I did suspend) claims he's completely innocent, and knows nothing about it. His apache log file is huge, so I guess that might explain why the 'cpanellogd' is getting bogged down to the point of crashing the VPS. Incidentally, why does WHM show this process as having a priority of '19' where everything else is '0'?

    I'm trying to setup APF on the system but it's not simple with it being a VPS. I've scoured hundreds of threads on this forum and generally tried to follow people's advice.

    I ran chkrootkit and it warned me of a possible LKM exploit.

    (Warning: Windows user at the helm. Unix knowledge very limited - if it *were* Windows the exploit would've been eliminated within 5 minutes of me searching for it)
     
    #3 AlexKitch, Dec 6, 2005
    Last edited: Dec 6, 2005
  4. Beansprout

    Beansprout Active Member

    Joined:
    Sep 12, 2005
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    You've just said you don't know what you're doing, so hire someone to fix this for you. Immediately.

    'tis the only way :)
     
  5. HostMerit

    HostMerit Well-Known Member

    Joined:
    Oct 24, 2004
    Messages:
    160
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New Jersey, USA
    cPanel Access Level:
    DataCenter Provider
    This may not be a full ROOTED issue, but just a simple PHP exploit.

    I first suggest you go into /tmp and run

    find ./ -user nobody|xargs rm -rf


    This will clear all the crap, also ls -al and look for directories / files starting with "." that usually would be hidden.


    Second thing is to run ps -u nobody. Look for processes different than httpd, like perl, sh, or psybnc.

    Take its process ID (pid) and go to /proc/(pid)/

    ls -al and look for shortcuts / symbolic links to someone /home directory, this would be where its run from, if nothing, cat environ, and look for PWD or OLDPWD that direct to someones /home directory.

    Next, install mod_security, and update with a secure ruleset. You can use mine which alot I've personally written from experience/honeypots/looking through logs.

    http://www.hostmerit.com/modsec.user.conf

    It doesnt break scripts, allows all of Frontpage / doesnt cause issues with extensions, blocks known rootkits downloading methods, and more.

    Email me at kris@hostmerit.com with your AIM info / etc if you would like me to help you with this free of charge.
     
  6. AlexKitch

    AlexKitch Member

    Joined:
    Jul 9, 2004
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Hi HostMerit

    Thanks for going to the trouble of writing all that, you're a hero! :)

    I have successfully done everything you've advised. "ps -u nobody" reported only Apache, EntropyChat and Melange. I did try to use your modsec script but Apache complained:

    Syntax error on line 992 of /usr/local/apache/conf/httpd.conf:
    Variables OUTPUT and OUTPUT_STATUS are not supported in the Apache 1.x version


    However, I did install the one suggested here: http://www.webhostgear.com/62.html . It's not quite as good as yours, but it's a start - until I can figure a way of using yours - I've heard bad stories about Apache 2.x, but maybe I just read in the wrong places ;)

    The server *seems* to be OK now, the datacenter are happy with it, there are no strange processes running, and bandwidth inbound/outbound are normal. I still wish I could find a decent firewall to run, but Virtuozzo refuses to run APF. Kiss firewall reports similar errors (same iptable errors). Shorewall seems complicated to setup, and my first attempt has failed :eek:

    Thanks again, a reply that doesn't say "use Search" or "hire someone.. you n00b" is very much appreciated (I am aware that the industry standard practice would be to reinstall the OS from a backup that isn't compromized, but I'd rather avoid that if at all possible) :cool:
     
  7. HostMerit

    HostMerit Well-Known Member

    Joined:
    Oct 24, 2004
    Messages:
    160
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New Jersey, USA
    cPanel Access Level:
    DataCenter Provider
    Hi Sorry about that.

    Please re-download it now - The problem was the line that said:

    Prevent PHP script leakage (you could find and remove by looking for OUTPUT) - I've fixed the script up there now.

    Looks like that WebHostGear article only has 1-3 things blocked, mine is the work of two years personally experiencing / finding / honeypotting these guys on Cpanel servers, it should work perfectly now, just that one OUTPUT line, was a 2.x rule that managed to slip in there.

    Also: cd /var/tmp - It should have the exact files that /tmp has.

    If it DOES NOT...

    cd /var;rm -rf tmp;ln -s /tmp

    Will fix this problem, and link /var/tmp to /tmp

    Does your VPS have a /tmp or /dev/shm partition?

    pico /etc/fstab


    Look for entries like:

    Code:
    LABEL=/tmp              /tmp                    ext3    [b]defaults[/b]        1 2
    none                    /dev/shm                tmpfs   [b]defaults[/b]         0 0
    
    If you see defaults under either of these, switch defaults to noexec,nosuid,nodev

    So the new version would look like this
    Code:
    none                    /dev/shm                tmpfs   [color=red]noexec,nosuid,nodev[/color]        0 0
    LABEL=/tmp              /tmp                    ext3    [color=red]noexec,nosuid,nodev[/color]        1 2
    
    Also run /scripts/compilers off , it never can hurt

    Let me know how this goes, and good luck!

    Email me if you need any help (or anyone else with security issues for that matter) at kris /@\ hostmerit.com
     
    #7 HostMerit, Dec 8, 2005
    Last edited: Dec 8, 2005
  8. HostMerit

    HostMerit Well-Known Member

    Joined:
    Oct 24, 2004
    Messages:
    160
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New Jersey, USA
    cPanel Access Level:
    DataCenter Provider
    Another point, alot of this, at least installing an up to date kernel or securing your /tmp and /dev/shm should all be done by the data center. While it's not their job to administrate the server, it's also not good business ethics to relase an easily exploitable server.

    Regardless of even that, without mod_security, and a good system admin, a Cpanel box is the easiest type of server to make into a Dos zombie. Also with 10mbps / 100mbps of connectivity with a big datacenter, it makes it a big threat. Becuase of all the Fantastico / popular CMS scripts being used, and their sub-par coding, its quite easy to get a file into /tmp and execute it with perl. Boom, perl bot right there.

    I think datacenters should start installing mod_security, even if it be with a small ruleset, just to block the essentials like wget, lynx, tmp, directory transversal, etc etc.

    Just my 2 cents. People shouldn't have to fork over 75+ for a standard secure server.
     
  9. blackpoint

    blackpoint Registered

    Joined:
    Nov 3, 2004
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Many Thanks from Switzerland

    Hi HostMerit

    Many thanks for your fast help on my cpanel server. You were my rescue!! :)

    You are a secure and cpanel professional.


    again, many thanks :)


    bpn
     
  10. HostMerit

    HostMerit Well-Known Member

    Joined:
    Oct 24, 2004
    Messages:
    160
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New Jersey, USA
    cPanel Access Level:
    DataCenter Provider
    Thats fine, you didn't need to pay (anyone) , someone had overwritten 777 / nobody owned files, not a total root exploit, your server was mostly clean, just needed mod_security, and a firewall, as well as some /tmp and /dev/shm flags. Glad to be of some free help, and let me know if you need anything else
     
  11. AlexKitch

    AlexKitch Member

    Joined:
    Jul 9, 2004
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Hi, thanks again for the reply

    I've found some VERY fishy files going on in both /tmp and /var/tmp

    Here is a sample from executing dir

    sess_5be0f96a7a2c8fadf2b7eb57c0637909 sess_ee082aa1986b7958b154d263ef3a702f
    sess_5f5749b66d2db2150373d47b691c7507 sess_f06b54192c39ed790c6f1ee4f3edbdf7
    sess_5f889d6d5e3fbafd0989dce3f2aee29a sess_f106e69ee2332ee5df3dc9cfeb6f3be1
    sess_60024834704cd380a213a691fb2be532 sess_f423f4887d241b57600f4ccd9285c6c2
    sess_616cb5a7928b20a2413c87c6be519639 sess_f58bf13db3102eb68578b1b569cc4e81
    sess_61cda610fb8248ad8b4e4492c7360d1e sess_f58d4688db51b46982d03b7c127badf5
    sess_63eb86e439e5bec073fb6324b4c40083 sess_f6b1109a77740a7f76abfdf2b4b696c2
    sess_653ea38caad1b1d227e1841e962417e4 sess_f7712aeae5b6b912a3619f93fed67d47
    sess_66db507388f98d3c20e6360ba64812d2 sess_f7ab150cd8935e7a5888ea43d8a4bf94
    sess_67372522ca27a3562d6b3b9358fe9ada sess_f888a247f76591c73e8d1fd014694b84
    sess_67d18f1cb47594b6d638ddbaf78d21c0 sess_f966e7136e6886f27a1f7c01f48e003d
    sess_6995ab803936d4a782f2101d0db5d561 sess_f992a184aad1456a054dd8cea68571b6
    sess_69a642d88f725cf4ae090237e3cc452b sess_fa24923c23948d26e0ea22ac6d603304
    sess_69fdb9375f87b3bb5283b4398b520590 sess_fa790409f2cd6c3e01fa3ee3b12c1f49
    sess_6b67643844ccabd4013604e352029851 sess_fbf5c3a86440cd0a49bad52f9b9d4d71
    sess_6d26fb8dc6b990faa64fc755e5397d66 sess_fd6ca40afe7843e86b405ec12b504aea
    sess_6f516886bb8c095855054f2a7c5c532e sess_fdb0fc3fddc528a70e92168598a8ac2c
    sess_705929dfcb2794c96c397920fc864089 start*
    sess_70ea84abbc6dc09d7e13dfeb58cf660b webchat.pl
    sess_7111a30183bec9e3b1cd257e43087540


    Inside webchat.pl looks "a bit dodgy", in my uneducated eyes. Here are some lines from the top of the script:

    -------------------------------------------------------------------------------------
    #!/usr/bin/perl
    #####################################################
    # udp flood.
    #
    # gr33ts: meth, etech, skrilla, datawar, fr3aky, etc.
    #
    # --/odix
    ######################################################

    use Socket;

    $ARGC=@ARGV;
    -------------------------------------------------------------------------------------

    No idea what's going on here, but that comment doesn't look good. Advise delete? :rolleyes:

    mod_security installed successfully now, thank you! :cool:


    EDIT: Additional

    fstab contained:

    none /dev/pts devpts rw 0 0
     
    #11 AlexKitch, Dec 9, 2005
    Last edited: Dec 9, 2005
  12. HostMerit

    HostMerit Well-Known Member

    Joined:
    Oct 24, 2004
    Messages:
    160
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New Jersey, USA
    cPanel Access Level:
    DataCenter Provider
    That's a perl based UPD flood DoS script. Will max out your server sending out 10-25mbps attacks (Depending on if you're 10mbps/100mbps) and of course get you disconnected / taken off your datacenters network.



    you can add another line in /usr/local/apache/conf/modsec.user.conf

    SecFilter "webchat"

    Although, you should be secure by my ruleset, never hurts add a custom rule from personal experience.


    You can rm -rf /tmp/*sess* as though are normal files, but webchat.pl and 'start' look shady.

    I would rm -rf /tmp/*chat* /tmp/start and add the ruleset above. Restart the machine to make sure no old processes are running / hanging and you should be immunized from ALOT of attacks.
     
  13. AlexKitch

    AlexKitch Member

    Joined:
    Jul 9, 2004
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    I did wonder why we tore through 85GB of bandwidth in 4 hours : :eek: :eek:

    Files disposed of as recommended. I'm still working my way through your last batch of suggestions, but it's nice to know I've found one/THE likely culprit file at last.
     
  14. AlexKitch

    AlexKitch Member

    Joined:
    Jul 9, 2004
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Maybe just spotted another in /var/tmp

    .webmin.tmp

    ------------------------------------------------------- Sample ---
    #############################################################
    # Based at Anti-Santy-Worm #
    # Modified by br0k3d #
    # You can blame me at: #
    # <edited out> #
    # #
    #############################################################

    use strict;
    use IO::Socket;
    use IO::Handle;


    my $process = '/usr/sbin/httpd';
    $0="$process"."\0"x16;;
    my $pid=fork;
    ------------------------------------------------------- Sample ---


    Also:

    .webmin.tmp.2

    ------------------------------------------------------- Sample ---
    #!/usr/bin/perl
    #
    # ShellBOT
    # 0ldW0lf - <edited out>
    # - <edited out>-br.cjb.net
    # - <edited out>cjb.net
    #
    #
    # isso eh meu B0tchZ reformulado
    #
    ###############################
    # killah... vai c ferrah :) #
    ###############################
    ------------------------------------------------------- Sample ---

    From what I can tell, this script connects to IRC and shares over DCC. This isn't really a risk though, since my datacenter have IRC blocked.
     
  15. HostMerit

    HostMerit Well-Known Member

    Joined:
    Oct 24, 2004
    Messages:
    160
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New Jersey, USA
    cPanel Access Level:
    DataCenter Provider
    Okay , apparently you didnt do the simlinking.

    Run this command

    cd /var;rm -rf tmp;ln -s /tmp

    This will symlink it, you should never have a /var/tmp directory that is standalone, it should always link to the /tmp partition.

    Also apparently your VPS doesn't have a /tmp partition, you should install a loop back device for /tmp.

    Reply to the email you sent me earlier and I will check / secure your server for you.

    Also, add this to your mod_security rules

    SecFilterSelective THE_REQUEST "webmin"
     
  16. simplestar

    simplestar Well-Known Member

    Joined:
    Nov 15, 2005
    Messages:
    97
    Likes Received:
    0
    Trophy Points:
    6
    Hi AlexKitch,

    I'm on VPS and run APF with BFD. They work well together and fine on CPanel VPS. APF is the firewall while BFD is for the Brute Force. Just be sure once you install it, to set the APF Dev mode to 'on' until you've configured correctly, then switch to 'off'. Also set the APF to auto restart. WHG has a pretty simple tutorial for installing. Lastly, you might also want to check out DDOs. Good Luck!

    APF Install Tutorial

    BFD Install Tutorial



    HostMerit, that's quit kind of you offering your help to Alex
     
  17. Nico

    Nico Well-Known Member

    Joined:
    Dec 5, 2001
    Messages:
    233
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Edmond, OK
    I've seen allot of this stuff from /tmp lately. What I'm finding is that it's usually a xmlrpc.php exploit with some scripts that are out there. I will not mention the exact names for security reasons, but they are crafting a url to use wget on the server to download the scripts to /tmp and then executing them with another crafted URL. I've disabled wget on all the servers that I've seen this on and it's pretty much came to a stop so far. Be sure to uprgrade any insecure scripts you come across as well.

    When you find a script in /tmp or other suspicious file in the /tmp do a grep for that filename on /usr/local/apache/*_log and you can see how it got there.

    As for the exploited servers I use the lsof -i command to find all the connections and programs that are running on the server and then tracking the down from there. Allot of the scripts are running under the guise of "https" as I've seen lately.
     
  18. isputra

    isputra Well-Known Member

    Joined:
    May 3, 2003
    Messages:
    576
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Mbelitar
    I'm using this ruleset and catch more action than my current ruleset. So i decided to use your ruleset, HostMerit :D

    Thanks for sharing yours.
     
  19. HostMerit

    HostMerit Well-Known Member

    Joined:
    Oct 24, 2004
    Messages:
    160
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New Jersey, USA
    cPanel Access Level:
    DataCenter Provider
    That ruleset will block almost everything...
    It will crush all xmlrpc exploits as stated, and more - It also has fixes for Frontpage, and common applications. It also will block all types of web proxies almost, including CGI Proxy, etc etc. I've personally written it so my server is as secure as it can be, without leaving much of a memory footprint

    It's meant for high traffic servers with many users running poorly coded applications. It happens, you must be ready :)

    Glad my ruleset is helping people!



    PS: "guise of "https" as I've seen lately."

    I believe you mean httpd, as to fool administrators to think it's apache. Since it runs as /usr/sbin/httpd, and ours runs from /usr/local/apache etc, we know it's rogue.

    Also, these can be easily crushed with simple filters such as

    SecFilter "xmlrpc\.php"
    SecFilter "xmlrpc"

    It may be a little hasty, but it blocks anything possibly referring to xmlrpc. It may break a blog cross-site script or two, but if it's one broken blog against a machine being used as a zombie, I'll take a broken blog script over a hacked box any day.
     
  20. simplestar

    simplestar Well-Known Member

    Joined:
    Nov 15, 2005
    Messages:
    97
    Likes Received:
    0
    Trophy Points:
    6
    Hi Kris,

    Thank you very much for sharing, I've installed your ruleset :D as well as I've been seeing a ton of those things in the logs too.
     
Loading...

Share This Page