Need to catch the Spamming Source

[furquan]

Hello everyone,

I am having a tough time finding the spamming source from my server, although i know that user is spamming but i am unable to locate the source.

I ran this command and found the user sending the maximum mails

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

I got these 3 users as suspect :-

1944 /home/twinciti
1945 /home/hcetawbu
3888 /home/anasdbz

Now when i am trying to find the source in one of the users above, using this command :-

ls -lahtr /home/anasdbz

I get this revert :-

-rw-r--r-- 1 anasdbz anasdbz 658 Apr 1 20:18 .zshrc
-rw-r--r-- 1 anasdbz anasdbz 124 Apr 1 20:18 .bashrc
-rw-r--r-- 1 anasdbz anasdbz 176 Apr 1 20:18 .bash_profile
-rw-r--r-- 1 anasdbz anasdbz 18 Apr 1 20:18 .bash_logout
lrwxrwxrwx 1 anasdbz anasdbz 11 Apr 1 20:18 www -> public_html
drwxr-x--- 3 anasdbz anasdbz 4.0K Apr 1 20:18 public_ftp
drwxr-x--- 2 anasdbz nobody 4.0K Apr 1 20:18 .htpasswds
-rw-r----- 1 anasdbz anasdbz 27 Apr 1 20:18 .contactemail
lrwxrwxrwx 1 anasdbz anasdbz 33 Apr 1 20:22 access-logs -> /etc/apache2/logs/domlogs/anasdbz
drwxrwx--x 6 anasdbz anasdbz 4.0K Apr 2 07:11 .cagefs
drwx------ 2 anasdbz anasdbz 4.0K Apr 2 07:17 .trash
drwxr----- 3 anasdbz anasdbz 4.0K Apr 2 07:19 .pki
drwx------ 2 anasdbz anasdbz 4.0K Apr 2 09:04 .ssh
drwxr-x--- 3 anasdbz mail 4.0K Apr 2 09:24 etc
drwxr-xr-- 19 anasdbz nobody 4.0K Apr 2 10:24 client.example.com
drwx--x--x 5 anasdbz anasdbz 4.0K Apr 2 16:00 .softaculous
drwx------ 3 anasdbz anasdbz 4.0K Apr 4 01:39 .jbm
drwx--x--x 20 anasdbz anasdbz 4.0K Apr 4 01:39 .
drwx------ 4 anasdbz anasdbz 4.0K Apr 15 14:56 .cphorde
-rw------- 1 anasdbz anasdbz 295 Apr 15 17:36 .lastlogin
drwxr-xr-x 8 anasdbz anasdbz 4.0K Apr 15 18:08 tmp
drwxr-x--- 4 anasdbz nobody 4.0K Jun 2 05:05 public_html
drwxr-xr-x 5 anasdbz anasdbz 4.0K Jun 2 05:06 ssl
drwx------ 2 anasdbz anasdbz 4.0K Jun 2 13:21 logs
-rw------- 1 anasdbz anasdbz 17 Jun 3 05:15 .ftpquota
drwxr-xr-x 2 anasdbz anasdbz 4.0K Jun 15 04:50 .cl.selector
drwx------ 5 anasdbz anasdbz 4.0K Jun 27 05:15 .cpanel
drwx--x--x. 1818 root root 68K Jul 3 10:53 ..
drwxr-x--x 10 anasdbz anasdbz 4.0K Jul 3 12:19 mail

Can anybody guide me further, what do i do next ?

Appreciate all assistance.

Thank you

 

[Jcats]

Honestly, those are probably all legit and are emails from the cronjob(s). Just look in the mail logs, example do:

grep -A1 /home/twinciti /var/log/exim_mainlog

Should give you something like:

2017-07-03 07:48:40 cwd=/home/user 3 args: /usr/sbin/sendmail -t -i
2017-07-03 07:48:40 1dS0mO-002sZB-0P <= [email protected] U=user P=local S=1499 [email protected] T="[SUBJECT OF EMAIL]\"" for [email protected]

You should be able to tell based on the subject of the email. Just add the following to the top of each users cronjob:

MAILTO=""

You can use SSH:

crontab -u USER -e

That will stop emails from being sent.

So anyway.. The command you are using to find the spam source is good but only for spam coming from PHP scripts, so if you don't see anything malicious in that aspect, then its most likely a compromised mailbox, use this:

head -1 /var/log/exim_mainlog | awk '{print $1}' ; egrep -o 'dovecot_login[^ ]+|dovecot_plain[^ ]+' /var/log/exim_mainlog | cut -f2 -d":" | sort|uniq -c|sort -nk 1 ; tail -1 /var/log/exim_mainlog | awk '{print From $1}'

If you find a user with a large amount of connections then that's most likely your guy.

*edit* Just grep for that specific user like so:

grep "dovecot_plain:[email protected]" /var/log/exim_mainlog |grep for

You will be able to easily tell by the Subject if its spam or not. If yes:
you can quickly generate a new random password for that user using:

help.bigscoots.com/cpanel/ssh/randomize-an-email-password-via-ssh-on-a-cpanel-server-requires-root

Also refer to this thread, it may be helpful:

cPanel Monitoring & Management

 

[furquan]

Thank you so very much for a such a detailed response, I shall go ahead and try to address this issue with the steps you have mentioned above.

Thank you

 

[furquan]

Hello Jcats,

I tried that command and the output that i got was :-

[email protected] [~]# head -1 /var/log/exim_mainlog | awk '{print $1}' ; egrep -o 'dovecot_login[^ ]+|dovecot_plain[^ ]+' /var/log/exim_mainlog | cut -f2 -d":" | sort|uniq -c|sort -nk 1 ; tail -1 /var/log/exim_mainlog | awk '{print From $1}'

2017-07-02
2017-07-03

Just 2 files named like dates on the server, what can it be and how do i proceed, Please guide.

Thank you

 

[Jcats]

That would indicate no one has authenticated to send out any emails between those dates.

First question, how do you know spam is coming from your server? Did you get an abuse complaint? If so, what info did it provide so we can use that to track down the abuser in the logs

Also, did you have any other results for

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

besides the ones you provided?

What if you do:

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n | grep public_html

This isn't fool proof since there can be paths inside your home directory but well pretend not.

 

[furquan]

No, I did not get any email from SPAMCOP or abuse, Our servers are set to inform us, if our IP address gets listed in any of the RBL's .

In this case i have been getting email like this for the past one week :-

S-RBL CheckerThe following ip addresses have been listed in RBLs. Please take neccessary actions.111.111.111.111
(the ip address is false here )

I'll run the commands you have mentioned above and will get back to you with an update.

Appreciate your assistance.

Thank you

 

[furquan]

Yes for the First command i got a whole list of domains who were sending mails along with the count,

With the Second command I got similar results as with the first command, But with a difference

here i do not see any of the domains preceeding with counts above 100...the maximum numer i can see is 98
eg :-
51 /home/slpfr83/public_html
56 /home/isnhosti/public_html
98 /home/sfdonl13/public_html

What is this ?, What do i do here ?

Thank you

 

[Jcats]

Yeah I mean those are all probably fine, but you can use the same command I provided previously to check the subjects of the emails being sent from those locations to determine if its spam or not, example:

grep -A1 /home/sfdonl13/public_html /var/log/exim_mainlog

S-RBL CheckerThe following ip addresses have been listed in RBLs. Please take neccessary actions.111.111.111.111

It didn't say WHICH rbl? It could be a false positive with some crappy RBL.

 

[furquan]

Thank you so very much for all your assistance Jcats

Nice to find a helping hand in times of need....Keep up the good work.

Thank you

 

[Jcats]

No problem, I try to help out when I have the time because of all the assistance cPanel has provided us over the years

 

[cPanelMichael]

Hello @Furqan,

I'm glad to see the information was helpful. Let us know if you have any additional questions.

Thank you.

 

[furquan]

Once again, My apologies


Hi Jcats & cPanelMichael,

These particular accounts have relayed mails in thousands, as per this command that i have run on the server :-

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n


4672 /home/twinciti
4675 /home/hcetawbu
9344 /home/anasdbz
10379 /root


But when i check my WHM panel for mail RELAYERS i do not see any of these account listed or coming up with any such numbers in the last 24hrs

Why this mismatch ?

Any suggestions, Please

Thank you

 

[Jcats]

Technically they are not relaying mail through the server since its originating from the server via some kind of script

but you have to dig more, grep will be your best friend here.

The /root one is most likely from CSF firewall, its probably sending out thousands of emails because of the other thousands of emails being sent out from the other accounts but we'll never know without the ability to see your exim log.

Run this:

grep "<= root\@" /var/log/exim_mainlog | awk -F"T=\"" '/<=/ {print $2}' | sort | uniq -c | sort -n

This will list out the subject of all emails originating from the root of your server and the receiving email, it will also greatly reduce the output since it will only show identical subjects once and then tally them up for you letting you know how many of that unique subject was found, example:

     16 lfd on server.example.com: SSH login alert for user CPUSER from 123.123.123.123 (US/United" for [email protected]
     19 lfd on server.example.com: Suspicious process running under user CPUSER" for [email protected]
     23 lfd on server.example.com: Suspicious process running under user CPUSER" for [email protected]
    184 lfd on server.example.com: Suspicious process running under user CPUSER" for [email protected]
    728 lfd on server.example.com: Suspicious process running under user CPUSER" for [email protected]

 

[furquan]

Thank you for your revert,

I shall have the commands executed on my server and will revert back to at the earliest.

Thanks a bunch

 

[furquan]

NOTE :- i have changed the user names and domain names for privacy sake, rest is as is.

Please let me know, what do i make of this ?

Thank you so very much

1002 Cron <[email protected]> curl http://some.domain.info/cron1.php" for Sender1
1002 Cron <[email protected]> curl http://some.domain.info/cron.php" for Sender1
1002 Cron <[email protected]> curl http://some.domain.info/refresh.php" for Sender1
1002 Cron <[email protected]> Get http://some.domain.info/cron1.php" for Sender1
1002 Cron <[email protected]> Get http://some.domain.info/cron.php" for Sender1
1002 Cron <[email protected]> php /home/prehoste/public_html/billing/bb-cron.php" for prehoste
1002 Cron <[email protected]> php -q /home/webhosti/public_html/admin/cron.php" for Sender1
1002 Cron <[email protected]> /usr/bin/php /home/webiptv/public_html/test/blesta/index.php cron" for Sender1
1045 Cron <[email protected]> php /home/isnhosti/public_html/customer/bb-cron.php" for isnhosti
5010 Cron <[email protected]> /usr/local/bin/php -q /home/anasdbz/public_html/cron.php" for Sender3
5010 Cron <[email protected]> /usr/local/bin/php -q /home/anasdbz/public_html/whmcs/cron.php" for Sender3
5010 Cron <[email protected]> /usr/local/bin/php -q /home/twinciti/public_html/clientexec/cron.php" for Sender4

 

[cPanelMichael]

Hello,

I recommend reviewing the cron jobs for the account usernames referenced in the output you provided. You can directly access the crontab files for users on your system in the following directory:

/var/spool/cron/

Look to see how often those cron jobs are configured to run, and review the individual scripts to see what types of emails they are sending.

Thank you.

 

[furquan]

Thank you so very much Michael,

I shall revert back once i have them examined .

Thank you

 

[furquan]

Thank you Michael,

This thread helped me locate the spammers and their sources :- Locate spam activity

Thank you

 

[furquan]

Hey jcats,

Apologies to bother you, In continuation to my issue mentioned above, I have noticed CRON entries for my user as mentioned below :-

30 10 * * * /usr/local/cpanel/3rdparty/bin/php /usr/local/cpanel/whostmgr/docroot/cgi/softaculous/cli.php --backup --auto=1 --insid=26_10044

00 3 * * * /usr/local/cpanel/3rdparty/bin/php /usr/local/cpanel/whostmgr/docroot/cgi/softaculous/cli.php --backup --auto=1 --insid=123_63532

SHELL="/usr/local/cpanel/bin/jailshell"


* * * * * /usr/local/bin/php -q /home/twinciti/public_html/clientexec/cron.php

Can you please let me know what is wrong in the above cron and what can i do to disable any further activity like this from any user or this user

Thank you for your time.

 

[Jcats]

There is nothing wrong with the way those are setup, as far as them possibly throwing errors when they are ran which would then result in an email being sent out, that you would need to look at the email itself as it will contain the error that is being thrown when its ran. In most cases, you can simply run the command via SSH and see if it errors, if there is an empty response, look in the folder you are currently active in for a 'error_log' file, as it may of logged a PHP error when you executed the script.

Otherwise, what you can do is either send any output from an individual cron to null which will stop the email, example:

* * * * * /usr/local/bin/php -q /home/twinciti/public_html/clientexec/cron.php > /dev/null 2>&1

Notice the > /dev/null 2>&1 added to the end, OR you can silence ALL cronjobs for that user by adding:

MAILTO=""

to the top. This can also be done if you click the 'cronjobs' icon in cPanel, then leave the email field empty and hit enter.