Need advice about malware on server

[RoboticPuppies]

Hello!

I need some advices. Previously I found strange processes that run under some of my client. The processes are look like this :

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                         
2801299 $USER  20   0  143780   5284   1168 S   1,6  0,0   2:51.33 sendmail                                                                       
3300210 $USER  20   0  143792   5324   1196 S   1,6  0,0   1:12.80 postfix                                                                         
3769754 $USER  20   0   45488   8060    676 S   1,6  0,0   0:00.89 sendmail                                                                       
2305990 $USER  20   0  143792   5316   1196 S   0,8  0,0   4:33.10 sendmail                                                                       
2305991 $USER  20   0  143780   5284   1168 S   0,8  0,0   4:33.17 sendmail                                                                       
2815191 $USER  20   0  143780   5284   1168 S   0,8  0,0   2:48.08 postfix                                                                         
3276506 $USER  20   0  143780   5284   1168 S   0,8  0,0   1:16.48 postfix                                                                         
3276507 $USER  20   0  143792   5316   1196 S   0,8  0,0   1:16.47 postfix                                                                         
3300213 $USER  20   0  143792   5324   1196 S   0,8  0,0   1:12.76 postfix                                                                         
2801300 $USER  20   0  143780   5288   1168 S   0,0  0,0   2:51.34 sendmail                                                                       
2815192 $USER  20   0  143780   5284   1168 S   0,0  0,0   2:48.15 postfix

I ran lsof -p PID to find out more about this process. But it executes /usr/bin/perl like this :

COMMAND     PID     USER   FD   TYPE    DEVICE  SIZE/OFF    NODE NAME
postfix 2815192 $USER  cwd    DIR     253,0      4096 3295856 /
postfix 2815192 $USER  rtd    DIR     253,0      4096 3295856 /
postfix 2815192 $USER  txt    REG     253,0     11408 3312043 /usr/bin/perl
postfix 2815192 $USER  mem    REG     253,0     86888 4194377 /usr/lib64/perl5/auto/POSIX/POSIX.so
postfix 2815192 $USER  mem    REG     253,0     19520 4194342 /usr/lib64/perl5/auto/Fcntl/Fcntl.so
postfix 2815192 $USER  mem    REG     253,0           4489097 /usr/local/lib64/perl5/auto/Socket/Socket.so (path inode=26488125)
postfix 2815192 $USER  mem    REG     253,0     19808 4194359 /usr/lib64/perl5/auto/IO/IO.so
postfix 2815192 $USER  mem    REG     253,0 106070960 3553880 /usr/lib/locale/locale-archive
postfix 2815192 $USER  mem    REG     253,0     11384 3303184 /lib64/libfreebl3.so
postfix 2815192 $USER  mem    REG     253,0   2127336 3279802 /lib64/libc-2.17.so
postfix 2815192 $USER  mem    REG     253,0    144792 3277399 /lib64/libpthread-2.17.so
postfix 2815192 $USER  mem    REG     253,0     14872 3287679 /lib64/libutil-2.17.so
postfix 2815192 $USER  mem    REG     253,0     41080 3287651 /lib64/libcrypt-2.17.so
postfix 2815192 $USER  mem    REG     253,0   1139680 3287655 /lib64/libm-2.17.so
postfix 2815192 $USER  mem    REG     253,0     19776 3287653 /lib64/libdl-2.17.so
postfix 2815192 $USER  mem    REG     253,0    113600 3287657 /lib64/libnsl-2.17.so
postfix 2815192 $USER  mem    REG     253,0    111096 3287673 /lib64/libresolv-2.17.so
postfix 2815192 $USER  mem    REG     253,0   1647272 4065914 /usr/lib64/perl5/CORE/libperl.so
postfix 2815192 $USER  mem    REG     253,0    164608 3287602 /lib64/ld-2.17.so
postfix 2815192 $USER    0r   CHR       1,3       0t0 3312383 /dev/null
postfix 2815192 $USER    1w   CHR       1,3       0t0 3312383 /dev/null
postfix 2815192 $USER    2w   CHR       1,3       0t0 3312383 /dev/null
postfix 2815192 $USER    3u  IPv4 723790372       0t0     TCP my.server.hostname.com:37160->77.72.83.84:tacnews (SYN_SENT)

I've tried to scan using Clamav and quarantine all malware that I found. But the process is coming back again, then I look into cron of the user. I found this cron :

*/10 * * * * perl /var/tmp/RICPHngaO >/dev/null 2>&1

I deleted the cron, but it came back again.
Some people gave me advices to track down the mailicious file using the start time of the process and search it in domlogs. It worked, but not for a long time.
So my questions are does anyone experience this?
How to stop this from happening? Please give me advice to prevent this.

Thank you for the help!

 

[ch3g3v4r4]

Hello, i have noticed the same issue for a couple of our users today, i came across this article by googleing the IP address
77.72.83.84 :D.

First i would like to say that i found this very useful as well github.com/bediger4000/php-malware-analysis/tree/master/chat.pl

It seems that somebody was trying to add our servers to a botnet!!!

I have managed to sort this out on our servers by>

1. Removing the script cron is calling.

2. Removing the cron itself.

3. Killing all of the processes manually using the command kill -9 PID PID PID PID

After these steps the issue was resolved for us and the IP 77.72.83.84 did not show up as a blocked outgoing connection on our firewall.

Also the crons are not appearing again and the processes as well.

I hope this helps you solve your issue mate! Cheers! :D

 

[sparek-3]

That user's account probably has either:

An outdated or abandoned script or plugin.

A compromised password, either for their cPanel account or for any CMS (WordPress, Joomla!) system they might be using. The password may have been extremely weak and easy to guess.

The client themselves is a malicious user, only out to abuse your server.

You really would need to identify which one the account fits under and then resolve that underlying issue. Otherwise, this is just going to keep happening.

 

[cPanelMichael]

Hello,

It's difficult to pinpoint the specific vulnerability or exploit used by an attacker to hack your server or websites. One could speculate on common methods (e.g. symlink attack), but it really requires a qualified system administrator to investigate the logs on your server and determine the source of the attack. There is a thread here where a similar question is asked:

Log Files To Check After Account Hacked

Thank you.

 

[RoboticPuppies]

Hello, i have noticed the same issue for a couple of our users today, i came across this article by googleing the IP address
77.72.83.84 :D.

First i would like to say that i found this very useful as well github.com/bediger4000/php-malware-analysis/tree/master/chat.pl

It seems that somebody was trying to add our servers to a botnet!!!

I have managed to sort this out on our servers by>

1. Removing the script cron is calling.

2. Removing the cron itself.

3. Killing all of the processes manually using the command kill -9 PID PID PID PID

After these steps the issue was resolved for us and the IP 77.72.83.84 did not show up as a blocked outgoing connection on our firewall.

Also the crons are not appearing again and the processes as well.

I hope this helps you solve your issue mate! Cheers! :D

Hello there!
Thanks for the information. Unfortunately, I've done all of steps that you informed for some days before I created this thread, but the issue is still coming back. For example the cron came back after I deleted it, and the process that I've killed still re-appeared after some hours. But, thank you for the info :D I'm glad you have solved your issue. And I think CSF will block any connection that created by this process. Because I configure CSF to block any unnecessary port, so it only shows SYN_SENT when I check it using lsof.

Hello,

It's difficult to pinpoint the specific vulnerability or exploit used by an attacker to hack your server or websites. One could speculate on common methods (e.g. symlink attack), but it really requires a qualified system administrator to investigate the logs on your server and determine the source of the attack. There is a thread here where a similar question is asked:

Log Files To Check After Account Hacked

Thank you.

Thank you for the info. I'll check the link and will try it out. Maybe I'll update this thread if I found something or the issue is solved :D

 

[ch3g3v4r4]

Hello,

It's difficult to pinpoint the specific vulnerability or exploit used by an attacker to hack your server or websites. One could speculate on common methods (e.g. symlink attack), but it really requires a qualified system administrator to investigate the logs on your server and determine the source of the attack. There is a thread here where a similar question is asked:

Log Files To Check After Account Hacked

Thank you.

In my case the attacker breached the CPanel account using a brute force attack from many different proxy IP's, he only tried the password 3 times from each IP so i think youd can understand why it could be hard to detect a attack like this.

But this was just the way a attacker managed to hack our CPanel account and this does not mean that the attacker was using the same method when attacking your servers.

I hope that one of this posts has helped you with resolving your issue mate!

Cheers!

 

[ch3g3v4r4]

Hello there!
Thanks for the information. Unfortunately, I've done all of steps that you informed for some days before I created this thread, but the issue is still coming back. For example the cron came back after I deleted it, and the process that I've killed still re-appeared after some hours. But, thank you for the info :D I'm glad you have solved your issue. And I think CSF will block any connection that created by this process. Because I configure CSF to block any unnecessary port, so it only shows SYN_SENT when I check it using lsof.


Thank you for the info. I'll check the link and will try it out. Maybe I'll update this thread if I found something or the issue is solved :D

Did you delete the script from /var/tmp also when removing the cron?

I trust you did, but did you change the CPanel account user pass and CMS user password?

If the process is appearing again its quite possible that your attacker still has access to this account.

Also i highly recommend resolving this issue permanently because if you dont and disable CSF for a min to check something or whatever the reason you are risking your server becoming a part of a botnet (github.com/bediger4000/php-malware-analysis/tree/master/chat.pl).

Currently the only thing saving your server from this scenario is the firewall, so keep this in mind.

Cheers mate!

 

[cPanelMichael]

In my case the attacker breached the CPanel account using a brute force attack from many different proxy IP's, he only tried the password 3 times from each IP so i think youd can understand why it could be hard to detect a attack like this.

Hello @ch3g3v4r4,

This is slightly off-topic, but I did want to mention cPHulk Brute Force Detection in-case you were not already aware of this feature:

cPHulk Brute Force Protection - Version 70 Documentation - cPanel Documentation

It allows you to block login attempts on a per-account basis, which is helpful in the event the login attempts are coming from different IP addresses.

Thank you.

 

[RoboticPuppies]

Did you delete the script from /var/tmp also when removing the cron?

I trust you did, but did you change the CPanel account user pass and CMS user password?

Hi,

Unfortunately the file in /var/tmp from the cron is not exist when I want to delete it. And when I delete the cron, it will re-appear again another hour. Oh yes, for the password, I haven't tried to change them. I'll think about it, thank you :D

If the process is appearing again its quite possible that your attacker still has access to this account.

I think you're right, the attacker may still have access to this account, or there are some vulnerabilites in the website. Or maybe the attacker has planted a backdoor to control the account.
Thanks a lot for the advice!