Our server was compromised

[simonlee]

It's a disaster and i find those things in /tmp:
./ cpanel.TMP.z9c5pT91s2n4QAsV
../ horde.log
.../ k*
1* kmod*
2* lols*
3* m00-HL-portbind*
4* m00-HL-portbind.c
b* mysql.sock@
.bash_history pt*
bd.tar.gz sess_2099f6eeebf3eaa4ccce0d46126c7a06
chkrootkit-0.42b/ sess_2be29889a4f5ba349697f90f5e7599ec
chkrootkit.tar.gz sess_34d188f0617ea56fb4ed9c276c787c8a
cpanel.TMP.10LJLeJ64VN2M9qs sess_36391f4b3f693c393b5a5d090acbbd66
cpanel.TMP.4047xDkDN99ZkxYv sess_430254d05c32333afa009e0d102c00c0
cpanel.TMP.5rdGsop9kUejcXRO sess_4325d41a231b4491fe79c7360095c2f0
cpanel.TMP.Bm4ZhmbG9pl17s5Y sess_44fbc6138676fb1cb927984a1f9b72ad
cpanel.TMP.C1NfrEQBM01e2bBt sess_4db17e33bf08d28757e54473981608ba
cpanel.TMP.Ej92uDbpJrVi1zmV sess_4e2dda3805b889ae1434cd1763388fa8
cpanel.TMP.HQDqFRBTJfICuPL6 sess_4efb58f69136ffc07c23893707f52ee6
cpanel.TMP.IRK9I4M_pkwOYBs2 sess_5c5c70a5558146a64b9d710e6a18b62e
cpanel.TMP.IxDj7iGACdsqPsXw sess_86496c3bd7ad9a9de450b9d26ba6f7b3
cpanel.TMP.jC6lYZ8lmkU4dhs2 sess_a9b1e716b9ad45359c06e79afe069d75
cpanel.TMP.KhaUNucHUJuh9z6b sess_b0e2658eb5ec3e0cc7f5ea39f2e724f2
cpanel.TMP.m3SQWek3_Wz0GmWq sess_b433818195e38d1241333bbb6fbc144c
cpanel.TMP.meA8M12S78_VYOmW sess_b7245d7d52708432662d0483465bf896
cpanel.TMP.OO72lrm3GDcc_WNA sess_cba473816011daced62258df61ad053a
cpanel.TMP.ouAP9_3VGsY2C5Ku sess_e77b33e8c51ee7e8fa2d4afc27305896
cpanel.TMP.TyWDKCRop7Kndr2u sess_fc3f4e6bc463d737c7606d22042245d7
cpanel.TMP.U_DCuV13druFok8R telnetd*
cpanel.TMP.u_iecqGHPsKPktg1 wget-log
cpanel.TMP.UZ6cjYkl1yCzoglR wget-log.1
cpanel.TMP.vTywGyHlsmUzgd7E x0x*
cpanel.TMP.yZw5SNSqpm3yjAPL

following in /var/tmp:
./ ../ httpd* mysql.sock@ s2* s2.c

We spent whole night to restore the server and it is back to online now.
Can somebody here to tell how to prevent such things to happen again?

[markie]

Originally posted by simonlee
It's a disaster and i find those things in /tmp:
./ cpanel.TMP.z9c5pT91s2n4QAsV
../ horde.log
.../ k*
1* kmod*
2* lols*
3* m00-HL-portbind*
4* m00-HL-portbind.c
b* mysql.sock@
.bash_history pt*
bd.tar.gz sess_2099f6eeebf3eaa4ccce0d46126c7a06
chkrootkit-0.42b/ sess_2be29889a4f5ba349697f90f5e7599ec
chkrootkit.tar.gz sess_34d188f0617ea56fb4ed9c276c787c8a
cpanel.TMP.10LJLeJ64VN2M9qs sess_36391f4b3f693c393b5a5d090acbbd66
cpanel.TMP.4047xDkDN99ZkxYv sess_430254d05c32333afa009e0d102c00c0
cpanel.TMP.5rdGsop9kUejcXRO sess_4325d41a231b4491fe79c7360095c2f0
cpanel.TMP.Bm4ZhmbG9pl17s5Y sess_44fbc6138676fb1cb927984a1f9b72ad
cpanel.TMP.C1NfrEQBM01e2bBt sess_4db17e33bf08d28757e54473981608ba
cpanel.TMP.Ej92uDbpJrVi1zmV sess_4e2dda3805b889ae1434cd1763388fa8
cpanel.TMP.HQDqFRBTJfICuPL6 sess_4efb58f69136ffc07c23893707f52ee6
cpanel.TMP.IRK9I4M_pkwOYBs2 sess_5c5c70a5558146a64b9d710e6a18b62e
cpanel.TMP.IxDj7iGACdsqPsXw sess_86496c3bd7ad9a9de450b9d26ba6f7b3
cpanel.TMP.jC6lYZ8lmkU4dhs2 sess_a9b1e716b9ad45359c06e79afe069d75
cpanel.TMP.KhaUNucHUJuh9z6b sess_b0e2658eb5ec3e0cc7f5ea39f2e724f2
cpanel.TMP.m3SQWek3_Wz0GmWq sess_b433818195e38d1241333bbb6fbc144c
cpanel.TMP.meA8M12S78_VYOmW sess_b7245d7d52708432662d0483465bf896
cpanel.TMP.OO72lrm3GDcc_WNA sess_cba473816011daced62258df61ad053a
cpanel.TMP.ouAP9_3VGsY2C5Ku sess_e77b33e8c51ee7e8fa2d4afc27305896
cpanel.TMP.TyWDKCRop7Kndr2u sess_fc3f4e6bc463d737c7606d22042245d7
cpanel.TMP.U_DCuV13druFok8R telnetd*
cpanel.TMP.u_iecqGHPsKPktg1 wget-log
cpanel.TMP.UZ6cjYkl1yCzoglR wget-log.1
cpanel.TMP.vTywGyHlsmUzgd7E x0x*
cpanel.TMP.yZw5SNSqpm3yjAPL

following in /var/tmp:
./ ../ httpd* mysql.sock@ s2* s2.c

We spent whole night to restore the server and it is back to online now.
Can somebody here to tell how to prevent such things to happen again?

Install a firewall then remove the ability for people to recompile .c source on your box. Everyone should be taking these precautions;

chmod 000 /usr/bin/*cc*

Then when you need to recompile or run updates;

chmod 700 /usr/bin/*cc*

Protect yourself from people compiling things on your box. Sure people can drop a compiled object on your box if they cant recompile but thats much harder especially if you have a decent firewall to protect you from them gaining access.

Another thing you should be doing is finding out how they dropped this into temp. Probably though some vulnarable version of phpbb, Oscommerce etc. Why dont you ask them how they did it if you can.

[simonlee]

Thanks markie.

Another question:
We moved entire mysql database from /var/lib/mysql on old drive to the new drive, but the databases and db users are not shown in cPanel->manage mysql.

Can some body tell what file we missed out?

[jimbo762]

Simon,
I've had to do this once before. The databases will not show up untill all your cpanel/whm accounts have been recreated.
Good luck as this was not a fun ordeal.
Jim

[simonlee]

Thanks Jim.
Exactly, it's a tough job.

Yes, the databases are shown up after we'v created all accounts in WHM.