421 errors in Exim caused by crazy brute force attacks, need help!

[LiNUxG0d]

Hi all,

Man, I'm facing some challenges as of late... and god, this is
becoming quite rediculous... I just don't know how to handle
this issue.

Synopsis:

I have a hosting server that's being SLAMMMMMED by so many
IP's, all for the same domain, obviously dictionary attacks as they
are cycling through usernames@somedomain.com
when sending mail.

I've done as follows:

- Installed APF.
- Installed BFD.
- Installed RBL/SBL checks.
- Installed Chirpy's dictionary attack preventions. (exim.pl)

I'm totally locked down security wise and all this stuff is doing a great
job. I just don't know why hundreds - if not thousands - of IPs are spamming
so much or brute forcing... I mean, I know WHY but why just THIS domain?

How can I just tell Exim to stop delivering mail to them? Or how can I stop this
alltogether? Should I change the MX from the DNS standpoint to point to 127.0.0.1
instead of my box? Sure it'll disable all mail access but what other options do I have?

I tried some packet sniffing to see if they're spoofed IPs but man, I'm running dot1q
over Cisco Catalyst switches and I'm just getting a MAC from my Router, which, is
useless. The packets are rewritten so...

Anyway, users are now getting 421's because of this and can't send legit email from
my server.

Just while writing this thread (~25 minutes; I'm multi tasking), there have been
4200 RCPT fails for this domain alone.

A few samples:

2006-07-19 15:52:10 H=(jnp-sbs1.jnpad.juniper.co.uk) [212.57.239.59] F=<> rejected RCPT <cqwcnbxo@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:10 H=(vio-mail.vio-dgn.com) [81.89.160.145] F=<> rejected RCPT <uebvpadj@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:11 H=pih-relay06.plus.net [212.159.14.133] F=<> rejected RCPT <izivwsll@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:11 H=mx05.t-net.net.ve [200.35.64.88] F=<> rejected RCPT <woyhjubh@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:11 H=msvu.ca (serf.msvu.ca) [140.230.5.76] F=<> rejected RCPT <dmpxodt@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:12 H=isis.tpiol.com [194.224.199.218] F=<> rejected RCPT <kxnjjugbx@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:12 H=smtp1a.net-cube.net [217.113.205.233] F=<> rejected RCPT <tebdxsg@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:12 H=pih-relay06.plus.net [212.159.14.133] F=<> rejected RCPT <wnyhirx@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:13 H=smtp1a.net-cube.net [217.113.205.233] F=<> rejected RCPT <kpmxdpb@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:54 H=jessica.csd.sc.edu [129.252.59.232] F=<> rejected RCPT <tcikqjqrpyh@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:54 H=bgl1mx1-a-fixed.sancharnet.in [61.1.128.46] F=<> rejected RCPT <kmakeve@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:55 H=malik.acsalaska.net [209.112.173.227] F=<> rejected RCPT <tvtobauoq@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:55 H=(mailgate.idsc.net.eg) [163.121.2.155] F=<> rejected RCPT <hpmigxzhs@somedomain.com>: Sorry, no such address.
2006-07-19 15:52:56 H=ns.mmc.co.jp (ns2.mmc.co.jp) [202.33.132.198] F=<> rejected RCPT <ratbrizccl@somedomain.com>: Sorry, no such address.
2006-07-19 15:53:09 H=fallback-peafowl.pas.sa.earthlink.net [207.217.120.254] F=<> rejected RCPT <pukgwjhn@somedomain.com>: Sorry, no such address.
2006-07-19 15:53:10 H=octgproc-gw.abz0.ifb.net (octg.co.uk) [194.105.187.193] F=<> rejected RCPT <oulwvvu@somedomain.com>: Sorry, no such address.
2006-07-19 15:53:10 H=fallback-peafowl.pas.sa.earthlink.net [207.217.120.254] F=<> rejected RCPT <kosxfmvprc@somedomain.com>: Sorry, no such address.
2006-07-19 15:53:11 H=fallback-peafowl.pas.sa.earthlink.net [207.217.120.254] F=<> rejected RCPT <mfopgax@somedomain.com>: Sorry, no such address. 

Please, someone?

Jamie

[chirpy]

How can I just tell Exim to stop delivering mail to them? Or how can I stop this
alltogether? Should I change the MX from the DNS standpoint to point to 127.0.0.1
instead of my box? Sure it'll disable all mail access but what other options do I have?

That's the only realistic option I can suggest to you.

Once a domain gets so deep in the mire, it's often impossible to get it cleaned up. The only other thought would be to have ti put through a third-party email filtering service - not anywhere near ideal, but might get it cleaned up.

[LiNUxG0d]

Hey Jonathan,

Absolutely what I thought. You know, it's just such a pain to do it otherwise. Realisitically, I don't care. Obviously, spammers don't just pick domains at random. He must have had a few addresses crawled and probably wrote his addresses on open forums and stuff.

You know, I was telling my colleague Dan, "Only one person will answer and that's Chirpy." and I was just waiting very patiently.

Thanks for confirming this for me Jonathan, it's really appreciated.

Take care of yourself,

Jamie

[chirpy]

My pleasure

I often find it's users who reply to spammers asking them to stop it - fatal mistake.

[LiNUxG0d]

Yeah for sure,

"Hey, can you please stop sending me this SPAM?"

Spammer, "Ah, ok, so that's a valid reply-to on your domain. Also, I have propogated it to everyone and their sister."

lol!

SPAM is just getting so rediculous nowadays. Spamcop.net even list that if someone sends you an email and you have an auto-responder setup, that you're a spammer. If - for example - I spoofed chirpy@yourdomain.com and sent to an auto-responder, you could grab my auto-response and say, "You spammed me."

How do you protect against that? Disable auto-responders. So what's the point of telling someone "I'm on vacation from x to y." if some spammer could implicate your organization in SPAM issues?

Bah, so crazy nowadays. I just don't tolerate any of it, period.

Hehehehe, a much better AUP IMHO.

I have good relations with Spamcop.net and Spamhaus.org now so that's great, though SORBS are really extorting money for delisting and larger organizations such as GoDaddy see that as fair game to clean up a server.

"You need to delist at SORBS sir to get unblocked by us."
"Yes, but they want 50$ donations; they're extorting money."
"Well, you know sir... blah blah."

Purely unaaceptable IMO. It's a click to remove an entry from a DB, so why do I need to pay?

I guess that issue is for another thread anyway.

Jamie

[lloyd_tennison]

I often find it's users who reply to spammers asking them to stop it - fatal mistake.

Chirpy, I am surprised at you. That philosophy on spamming is so outdated. If the email did not bounce - they assume it gets to you. Unless the spam is blatantly so, most unsubscribes actually do work.

There are studies out there on that very fact, and most agree to TRY and unsubscribe. They already have your email address and most properly configured mail servers will not even take the email if the address is invalid (save for those poor souls that use catchall ) So, you have nothing to lose and all to gain.

[webignition]

If the email did not bounce - they assume it gets to you.

The only thing that can be assumed if an email is not bounced or rejected is that the mailbox exists.

However, if someone replies or reponds to a spam, the spammer knows not only does the mailbox exist, but that a person checks it - which clearly makes it a better spam target than a mailbox that does exist but which you don't know if a person checks it.

[chirpy]

Yup, as webignition has pointed out.

I wasn't talking about the normal background scatter of spam. I'm talkin about specific dirty domains that are receiving so much spam that it starts affecting server resources, i.e. those that Jamie is talking about.

IMX, those that suffer such extreme loads of spam are those where the user has actively exposed themselves by responding to spam/clicking on spam links.

From what I've seen, experienced and read, spammers usually ignore bounces. Not using a catchall is cetainly essential in reducing the normal levels spam that you receive. However, to get to the levels we're talking about here usually requires end-user participation.

[deftech]

dont give up on your domain. we set up a mailtrap server to handle the spam, and relay to our 2nd mail server legit mail.

the only way to deal with denial of service is to expand your infrastructure to accomadate the extra traffic.

works great for us.

[alwaysweb]

A few domains we host have had issues like this in the past, one was receiving 50,000+ dictionary attack style mail attempts. Solution? Setup a postini.com filtering account for this, change customer's MX records to flow to postini. Then, go into postini and enter the "valid" email boxes, and your server IP and they will do the filtering and then hand off whats valid to your server. Works great for us, about $2.35/email box via postini-wholesaler.com (just recently sold to another postini reseller I believe). In my opinion, the $2 you spend (or $4 or $6, etc.) is worth the reduction in server CPU, IO, and overall effect the high load can periodically cause from a problem like this!

Still stuck? Need more help? Feel free to contact me.

[Xdred]

We have seen this type of dictionary attack in the past. You could certainly use postini or another filtering service. We generally prefer to mitigate it using our own methods. Like Chirpy said this is really a resource issue. We have seen instances where the server hardware and MTA were handling things fine however the router was failing due to full buffers. From looking at the IPs your posted many are coming from the UK. Assuming you are not in the UK you could use a RBL country block uk.countries.nerd.dk. I would do reverses on the IPs and do RBL country blocks on the high offenders. You also want to load exim with other common RBLs. When this is still too much for your server it's time to change the mx to another server to handle the attack. I have seen attacks continue for quite some time even after the mx change. Bottom line this is certainly a bot army and as such only has X number of bots. Of course he has some friends that could be helping with their bots. So to mitigate you must have the bandwidth, firewall and server hardware. It then becomes an issue of obtaining the IPs of the machines involved so they can be blocked up stream of your server. In most cases this number is less than 10k. An easy way to do this is with vispan if other more expensive DDOS hardware is not available. Do a goggle search. On the MTA use a catchall address. Spamassassin will tag it as spam and vispan can be configured to gather the ip list. This list must be automatically synced to the hardware firewall block list. When the list is larger than 10k or so is is better blocked at the data center. With larger attacks the block would be placed upstream of the data center. There are many ways to combat this sort of thing and is really not that difficult when compared with more sophisticated DDOS.

[lloyd_tennison]

we set up a mailtrap server to handle the spam, and relay to our 2nd mail server legit mail.

Curious on exactly how you did that.