Log Checking

[nyoman]

One of my user on Dedicated server/WHM/Cpanel look like compromised.

There is always a strange file in /tmp (it is a phpshell) and i can not find, how the attacker upload this files.

I got a reporting log from maldet

malware detect scan report for elite.myserver.com:
SCAN ID: 051115-0408.2127
TIME: May 11 04:08:05 -0500
PATH: /var/www/html
RANGE: 2 days
TOTAL FILES: 145
TOTAL HITS: 1
TOTAL CLEANED: 0

FILE HIT LIST:
{HEX}php.cmdshell.unclassed.357 : /tmp/php3Wgm29 => /usr/local/maldetect/quarantine/php3Wgm29.17334

malware detect scan report for elite.myserver.com:
SCAN ID: 051115-0408.2188
TIME: May 11 04:08:05 -0500
PATH: /usr/local/apache/htdocs
RANGE: 2 days
TOTAL FILES: 145
TOTAL HITS: 1
TOTAL CLEANED: 0

FILE HIT LIST:
{HEX}php.cmdshell.unclassed.357 : /tmp/php3Wgm29 => /usr/local/maldetect/quarantine/php3Wgm29.9558

How to know, from where this file uploaded ?

Regards.

 

[cPanelMichael]

Hello,

What PHP handler are you using? Are there any PHP scripts installed on this account (e.g. Joomla)?

Thank you.

 

[quizknows]

I strongly discourage use of the auto quarantine or auto clean features of maldet. They can easily remove the timestamps and other information needed to properly diagnose an infection.

Michaels question of PHP handler is important; if you are using SuPHP or fcgi you should be able to gain a username from the files ownership to determine the potentially infected account. If you are using DSO then the file would almost certainly be owned by "nobody" which makes it much harder to properly investigate.

I would advise DISABLING the auto quarantine feature, and then running a maldet scan on all users public_html directories, such as:

# maldet -a /home?/?/public_html/

 

[nyoman]

@cPanelMichael : the server use modSecurity and suPHP
Domain use OLD WordPress (version 3.9.6), many time send email to the owner to update the WordPress, but they ignore my email.

@quizknows: i know the user/file owner, but i can not find the POST log from the domain access-log.
all POST log look legit.

[email protected]:[~] # cat /usr/local/maldetect/quarantine/php3Wgm29.17334.info
xxxxxspa xxxxxspa 600 /tmp/php3Wgm29

quar_hits=1
quar_clean=1
quar_susp=0

Clean already ENABLE.
Suspend DISABLE.

Yesterday i try change mysql password, if this still happen will try change wp admin password and cpanel password.

Regards.

 

[cPanelMichael]

Domain use OLD WordPress (version 3.9.6), many time send email to the owner to update the WordPress, but they ignore my email.

You may want to upgrade it to the latest version or suspending the account until the customer upgrades it to ensure the old version is not the culprit.

Thank you.

 

[nyoman]

Change mysql password.
Update WP
Change wp-admin password.

And they still able upload phpshell ?

-rw-------  1 xxxxxspa xxxxxspa  33259 May 15 14:37 phpi6xb4Z


[email protected]:[~] # more /tmp/phpi6xb4Z
<?php
$auth_pass = "866fd58d77526c1bda8771b5b21d5b11";
$color = "#df5";
$default_action = 'FilesMan';
$default_use_ajax = true;
$default_charset = 'Windows-1251';
@ini_set('error_log',NULL);
@ini_set('log_errors',0);
@ini_set('max_execution_time',0);
@set_time_limit(0);
@set_magic_quotes_runtime(0);
@define('WSO_VERSION', '?');
if(get_magic_quotes_gpc()) {
  function WSOstripslashes($array) {
  return is_array($array) ? array_map('WSOstripslashes', $array) : stripslashes($array);
  }
  $_POST = WSOstripslashes($_POST);
  $_COOKIE = WSOstripslashes($_COOKIE);
}
function wsoLogin() {
  die("<pre align=center><form method=post>Password: <input type=password name=pass><input type=submit value='>>'></form></pre>");

Files found on /tmp, uploaded about 7 hours ago.

[email protected]:[~/bin] # ./vdetect --user=xxxxxspa
Directories scanned: 00000269
====================================
--=== VERSION DETECTION REPORT ===--
====================================

Up-To-Date Applications:
========================================
Wordpress  ::  4.2.2 :: /home/xxxxxspa/public_html

 

[quizknows]

Any outdated plugins? More often than not, those are the culprit. Also, there could be code injections hiding in some files, best to force re-install the WP files just to be sure.

What you really need to look at though is what happened at 14:37 in that users apache access log. Check the domlogs for what happened at that time; ideally 'stat' the file so you have both modify and change times to the second.

Also, they can't do anything with that in /tmp anyway, but if it made it to public_html obviously that's a problem.

Like I said before I strongly discourage use of auto quarantine with maldet, at least until you have your investigation done and are happy with the results. Otherwise it can change key times and things you need to properly investigate.

 

[24x7server]

I will suggest you please configure LMD with the mod_Sec, So that maldetect will scan your files while uploading on your server.

You can add following code /usr/local/apache/conf/modsec2.user.conf (or similar mod_security2 rules file):

SecRequestBodyAccess On
SecRule FILES_TMPNAMES "@inspectFile /usr/local/maldetect/modsec.sh" \
  "log,auditlog,deny,severity:2,phase:2,t:none"

Also you can install ConfigServer eXploit Scanner (http://configserver.com/cp/cxs.html) on your server

 

[nyoman]

Any outdated plugins? More often than not, those are the culprit. Also, there could be code injections hiding in some files, best to force re-install the WP files just to be sure.

What you really need to look at though is what happened at 14:37 in that users apache access log. Check the domlogs for what happened at that time; ideally 'stat' the file so you have both modify and change times to the second.

Also, they can't do anything with that in /tmp anyway, but if it made it to public_html obviously that's a problem.

Like I said before I strongly discourage use of auto quarantine with maldet, at least until you have your investigation done and are happy with the results. Otherwise it can change key times and things you need to properly investigate.

I was check domlogs files, but i can not fins any suspect log

[email protected]:[~] # stat /tmp/phpi6xb4Z
  File: `/tmp/phpi6xb4Z'
  Size: 33259  Blocks: 72  IO Block: 4096  regular file
Device: 700h/1792d  Inode: 49282  Links: 1
Access: (0600/-rw-------)  Uid: (  500/xxxxxspa)  Gid: (  500/xxxxxspa)
Access: 2015-05-17 20:50:09.000000000 -0500
Modify: 2015-05-15 14:37:36.000000000 -0500
Change: 2015-05-15 14:37:36.000000000 -0500

What i got from the log with those time only one row

[email protected]:[~] # zgrep "14:37:36" /home/xxxxxspa/logs/xxxxx-spas.com-May-2015.gz
14.33.247.130 - - [15/May/2015:14:37:36 -0500] "POST /wp-content/plugins/wp-symposium/server/php/index.php HTTP/1.1" 404 3325 "-" "Mozilla/5.0 (Windows NT 6.1; rv:36.0) Gecko/20100101 Firefox/36.0"

Which is code 404.

There is one outdated plugin (Really Simple Share) and today, i try update this plugin.

I will suggest you please configure LMD with the mod_Sec, So that maldetect will scan your files while uploading on your server.

You can add following code /usr/local/apache/conf/modsec2.user.conf (or similar mod_security2 rules file):

SecRequestBodyAccess On
SecRule FILES_TMPNAMES "@inspectFile /usr/local/maldetect/modsec.sh" \
  "log,auditlog,deny,severity:2,phase:2,t:none"

Also you can install ConfigServer eXploit Scanner (http://configserver.com/cp/cxs.html) on your server

Thanks, will try later.

 

[quizknows]

I tested /usr/local/maldetect/modsec.sh and it's pretty broken. You have to turn on global scanning (to let all users use maldet) and it still scans /tmp on every scan. Not impressive at all. You end up with very slow uploads (that get slower as /tmp gains files), and it dumps a bunch of stuff into the apache error log every time a file is uploaded whether it's good or not.

Sadly, the maldet script, and the maldet binary itself, would need some work for that to be OK in production.

However I was able to use modsecurity and clamdscan to quickly and effectively scan files as they were being uploaded, I'll document that at a later time.

 

[nyoman]

I tested /usr/local/maldetect/modsec.sh and it's pretty broken. You have to turn on global scanning (to let all users use maldet) and it still scans /tmp on every scan. Not impressive at all. You end up with very slow uploads (that get slower as /tmp gains files), and it dumps a bunch of stuff into the apache error log every time a file is uploaded whether it's good or not.

Sadly, the maldet script, and the maldet binary itself, would need some work for that to be OK in production.

However I was able to use modsecurity and clamdscan to quickly and effectively scan files as they were being uploaded, I'll document that at a later time.

Thanks quizknows for your time to make a test on modsec.sh.

And i will waiting your document about modsecurity and clamdscan

Regards,

Nyoman

 

[quizknows]

To use clamscan to scan files as they are being uploaded (this is similar to the CXS setup):

Add this rule to your modsecurity rules (probably modsec2.user.conf unless your provider gives you a separate file like custom.conf):

#uncomment these if they are not already set by your rule set.  It doesn't break things if they're set twice.
#SecUploadDir /var/tmp
#SecRequestBodyAccess On
SecRule FILES_TMPNAMES "@inspectFile /usr/local/apache/conf/modsec2/modsec-clamscan.pl" \
"log,auditlog,deny,severity:2,id:'93847'"

I have updated modsec-clamscan.pl from it's open source version, albeit very minor updates, to use the cPanel install of clamdscan (note the "d") which is extremely fast compared with clamscan, and to correct a deprecated option. Create the file /usr/local/apache/conf/modsec2/modsec-clamscan.pl as root owned with 755 permissions, and this content:

#!/usr/bin/perl

#Use clamdscan, it's much faster than clamscan! I Don't recommend using this with 'normal' clamscan, due to super slow uploads.
$CLAMSCAN = "/usr/local/cpanel/3rdparty/bin/clamdscan";

if (@ARGV != 1) {
  print "Usage: modsec-clamscan.pl FILENAME\n";
  exit;
}

my ($FILE) = @ARGV;

$cmd = "$CLAMSCAN --stdout --no-summary $FILE";
$input = `$cmd`;
$input =~ m/^(.+)/;
$error_message = $1;

# Default behavior is to reject uploads if clamd is down.
# If you want uploads to work if clamd is down/not running, then change this "0" to a "1", however nothing will be scanned!
$output = "0 Unable to parse clamscan output or clamd is not running.";

if ($error_message =~ m/: Empty file\.$/) {
  $output = "1 empty file";
}
elsif ($error_message =~ m/: (.+) ERROR$/) {
  $output = "0 clamscan: $1";
}
elsif ($error_message =~ m/: (.+) FOUND$/) {
  $output = "0 clamscan: $1";
}
elsif ($error_message =~ m/: OK$/) {
  $output = "1 clamscan: OK";
}

print "$output\n";

This will allow modsecurity to use clamdscan to scan files while they are being uploaded through a website.

 

[stratogod]

Just wanted to say THANK YOU quizknows for sharing this modsec/clamd integration. I have been banging my head on a wall the last few days trying to get maldet's integration to work - to no avail. I was about to call it quits and stumbled on to this thread...

Everything is working great now. Tested uploading files using multiple domains with different setups (WordPress, vBulletin, Magento), and sure enough files are now being scanned by clamd before they are accepted.

Thanks!

 

[quizknows]

Just wanted to say THANK YOU quizknows for sharing this modsec/clamd integration. I have been banging my head on a wall the last few days trying to get maldet's integration to work - to no avail. I was about to call it quits and stumbled on to this thread...

Everything is working great now. Tested uploading files using multiple domains with different setups (WordPress, vBulletin, Magento), and sure enough files are now being scanned by clamd before they are accepted.

Thanks!

No problem, thanks for taking the time to let me know it works for you It's a good free option if you don't have ConfigServer exploit scanner (CXS.)

If you ever purchase CXS (which I do recommend if you host a lot of sites), their file upload scanning script for ModSecurity is also very good and it does have a bit better detection rate than clamav does. I think I've documented in another thread how to get that one going. Regardless, using clamdscan is free and fast so if nothing else it's an easy extra layer of protection.

 

[xlightwaverx]

Why not just use Maldet kernel level real time scanning and not rely on other programs to trigger the monitor of a new file?

 

[quizknows]

Why not just use Maldet kernel level real time scanning and not rely on other programs to trigger the monitor of a new file?

Are you talking about the inotify feature? I'd rather have the file blocked before it's even in the directory, but layered security is always good

Also with modsecurity and CSF, it will block the offending IP address if they keep trying to upload a bad file or otherwise trip modsecurity rules.

 

[jeffschips]

To use clamscan to scan files as they are being uploaded (this is similar to the CXS setup):

Add this rule to your modsecurity rules (probably modsec2.user.conf unless your provider gives you a separate file like custom.conf):

#uncomment these if they are not already set by your rule set.  It doesn't break things if they're set twice.
#SecUploadDir /var/tmp
#SecRequestBodyAccess On
SecRule FILES_TMPNAMES "@inspectFile /usr/local/apache/conf/modsec2/modsec-clamscan.pl" \
"log,auditlog,deny,severity:2,id:'93847'"

I have updated modsec-clamscan.pl from it's open source version, albeit very minor updates, to use the cPanel install of clamdscan (note the "d") which is extremely fast compared with clamscan, and to correct a deprecated option. Create the file /usr/local/apache/conf/modsec2/modsec-clamscan.pl as root owned with 755 permissions, and this content:

#!/usr/bin/perl

#Use clamdscan, it's much faster than clamscan! I Don't recommend using this with 'normal' clamscan, due to super slow uploads.
$CLAMSCAN = "/usr/local/cpanel/3rdparty/bin/clamdscan";

if (@ARGV != 1) {
  print "Usage: modsec-clamscan.pl FILENAME\n";
  exit;
}

my ($FILE) = @ARGV;

$cmd = "$CLAMSCAN --stdout --no-summary $FILE";
$input = `$cmd`;
$input =~ m/^(.+)/;
$error_message = $1;

# Default behavior is to reject uploads if clamd is down.
# If you want uploads to work if clamd is down/not running, then change this "0" to a "1", however nothing will be scanned!
$output = "0 Unable to parse clamscan output or clamd is not running.";

if ($error_message =~ m/: Empty file\.$/) {
  $output = "1 empty file";
}
elsif ($error_message =~ m/: (.+) ERROR$/) {
  $output = "0 clamscan: $1";
}
elsif ($error_message =~ m/: (.+) FOUND$/) {
  $output = "0 clamscan: $1";
}
elsif ($error_message =~ m/: OK$/) {
  $output = "1 clamscan: OK";
}

print "$output\n";

This will allow modsecurity to use clamdscan to scan files while they are being uploaded through a website.

With WHM 56.0 (build 24) and CPanel 56.0.24 I recieve the following error when attempting to troubleshoot quizknows script when running command line ./clamdscan:

ERROR: Can't parse clamd configuration file /etc/clamd.conf

In a typical WHM/CPanel install, clamdscan has no configuration file -- or at least one that I can find - so errors abound when attempting to institute this script.

 

[quizknows]

Does /etc/clamd.conf exist at all? I may suggest you open a cpanel ticket on that one, since I've never had that issue.

Is the clamscan binary (as well as clamdscan binary) present in this directory?:

/usr/local/cpanel/3rdparty/bin/

If not you may need to (re)install the clamd stuff via WHM.

 

[jeffschips]

Does /etc/clamd.conf exist at all? I may suggest you open a cpanel ticket on that one, since I've never had that issue.

Is the clamscan binary (as well as clamdscan binary) present in this directory?:

/usr/local/cpanel/3rdparty/bin/

If not you may need to (re)install the clamd stuff via WHM.

Opened a ticket and CPanel support says they don't support 3rd. party features. And yes, there is clamscan and clamdscan present in /urs/local/cpanel/3rdparty/bin/

The technician from CPanel asked that I "ln -s /usr/local/cpanel/3rdparty/etc/clamd.conf /etc/clamd.conf" which I did do and now my error when I run /usr/local/cpanel/3rdparty/bin/clamdscan /home/domain/public_html/? is:

Servname not supported for ai_socktype

Must the clamav service be runing for clamdscan to operate?

I'm simply trying to get the script by quizknows to work.

 

[quizknows]

Yes, clamav needs to be running.

The whole point of using clamdscan is it can socket to the running clamav process and run nearly instantly. The old modsec hook called clamscan (not clamdscan) which resulted in a 2-5 second delay for file uploads while clamscan starts up (and shuts down).

Try these commands:
/usr/local/cpanel/3rdparty/bin/clamscan -ir /home/domain/public_html/
/usr/local/cpanel/3rdparty/bin/clamdscan -i /home/domain/public_html/

Keep in mind the 'i' flag means infected so there should only be output returned if a malicious file is found under that path. Clamdscan doesn't support recursion but it's not necessary for single file scans during uploads (obviously).