The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Exim Dictionary flood

Discussion in 'General Discussion' started by mike25, Aug 12, 2006.

  1. mike25

    mike25 Well-Known Member

    Joined:
    Aug 29, 2003
    Messages:
    83
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Raleigh NC, USA
    One of my servers is under a constant inbound SMTP dictionary attack. The attack is so widespread that I am unable to discover a way to prevent it. There is a constant stream of inbound attempts and then every few hours a massive spike that will push the server load into the 50 range. All attempts are against the same domain with unrouted mail set as :fail:. I have chripys very nice exim ACL list running as well. In the last few days 8k IPs have been blocked by it. All geographically widespread. I have been blacklisting IPs in the firewall as well for the last week and can not manage to stop this thing. When the major spike hits is it each attacking IP will only open 20-40 connections to the server making connection flood detection difficult. I am at my wits end and am just considering dumping the client. Has anyone experienced anything like this before?
     
  2. lloyd_tennison

    lloyd_tennison Well-Known Member

    Joined:
    Mar 12, 2004
    Messages:
    698
    Likes Received:
    1
    Trophy Points:
    18
    #2 lloyd_tennison, Aug 13, 2006
    Last edited: Aug 13, 2006
  3. Damian

    Damian Well-Known Member

    Joined:
    Oct 1, 2001
    Messages:
    95
    Likes Received:
    1
    Trophy Points:
    8
    Lloyd, mike25 mentions that he's already using Chirpy's exim deny (dictionary attack) ACL. He's looking for some other alternatives/additions.

    I too am interested as we're seeing a similar spike in load (up to around 30 - when other services start to fail) when a domain is being bombed. We are also using the exim deny (dictionary attack) ACL as well.
     
  4. RickG

    RickG Well-Known Member

    Joined:
    Feb 28, 2005
    Messages:
    238
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    North Carolina
    Mike:

    If the attacks are having that much of an impact on the server resources, it may be time talk to the client. Unfortunately there is no way to absolutely stop the attacks (short of the client changing their domain name).

    As you know, loading the iptables with thousands of IP's in itself is going to eat up server resources and slow things down. What other blocks (other than ConfigServers dictionary attack ACL) do you have in place? Are you using any HELO/EHLO tests in your exim.conf file? What about RBL's? Sometimes the order in which you have these can help reduce the load.

    You may want to also look into Exim's smtp_accept_max_per_host setting which restricts the number of simultaneous IP connections from a single host.
     
  5. mike25

    mike25 Well-Known Member

    Joined:
    Aug 29, 2003
    Messages:
    83
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Raleigh NC, USA
    Thanks for all your suggestions, the smtp_accept_max_per_host should help some, I will read up on some of the other exim tweaks that you have suggested. All of the inbound connections are being made to one domain which is only a few weeks old. I am unsure why these people are so persistant, they are really only wasting their time. I have managed to keep the load within acceptable levels now, but the dictionary attack continues. If it does not cease soon I may just have to drop the domain and save myself the headache.
     
  6. lloyd_tennison

    lloyd_tennison Well-Known Member

    Joined:
    Mar 12, 2004
    Messages:
    698
    Likes Received:
    1
    Trophy Points:
    18
    Sorry, I must have missed that in the original post.

    Try Chirpy's firewall as it blocks using DShield Block List and Spamhaus DROP List URL. These are blocked at the firewall level - so you only get one attempted connection. It does not even have to wait for the full attack.
     
  7. mike25

    mike25 Well-Known Member

    Joined:
    Aug 29, 2003
    Messages:
    83
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Raleigh NC, USA
    I am running chirpys firewall, which is also a very nice tool. I have dshield and spamhaus blocking enabled. The biggest problem with this attack is how widespread it is. 19k IPs have now been caught by the ACL. many of these are from the same netblock, but many are not, but all are from third world nations from every part of the globe. all are attacking the same domain. I am mainly just surpised at how far this attacker will go to make a domain unusable. It seems that a good deal of time has been spent on their part to launch the attack. I am still open to any suggestions, but the more I read about this it seems that all I can do is to just drop the domain.
     
  8. elenlace

    elenlace Well-Known Member

    Joined:
    Sep 10, 2002
    Messages:
    101
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    US
    Customers getting blocked

    Hello Chirpy,

    Great tool I have to say, but unfortunately some of my customers are getting blocked by this. I tried to hack the exim_deny.pl file but couldn't understand how to increase the 4 and out level to 10 and out and only block extreme dictionary attacks and not light ones.

    Customers do send erroneous addresses from time to time and that doesn't mean they are attempting a dictionary exploit.

    Looking forward to your response.
     
  9. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    You need to change the ACL to increase the number of fails before it blocks. Change this part from:

    Code:
           condition = ${if > {${eval:$rcpt_fail_count}}{3}{yes}{no}}
    to:

    Code:
           condition = ${if > {${eval:$rcpt_fail_count}}{9}{yes}{no}}
    You can also whitlist IP addresses by adding them to /etc/exim_deny_whitelist
     
  10. dandanfireman

    dandanfireman Well-Known Member
    PartnerNOC

    Joined:
    May 31, 2002
    Messages:
    117
    Likes Received:
    0
    Trophy Points:
    16
    Heavy use of RBLs can also help with a situation like this.
     
  11. randomuser

    randomuser Well-Known Member

    Joined:
    Jun 25, 2005
    Messages:
    147
    Likes Received:
    0
    Trophy Points:
    16
    That is a LOT of connections for 1 IP address. Going off that info, it shouldn't be hard detecting that really.

    If you're still reading your thread, I'm curious to know:
    About what percentage of the time are you seeing a single IP address attempt to send to more than 3 invalid users? By example, what I mean is this:

    ONE
    2006-09-03 02:51:59 H=lns-bzn-21-82-248-157-212.adsl.proxad.net (7vgur.uwau.optonline.net) [82.248.157.212] F=<annex.compute@rozierteam.com> rejected RCPT <info@victim.com>: No Such User Here

    TWO
    2006-09-03 02:51:59 H=lns-bzn-21-82-248-157-212.adsl.proxad.net (7vgur.uwau.optonline.net) [82.248.157.212] F=<annex.compute@rozierteam.com> rejected RCPT <home@victim.com>: No Such User Here

    THREE
    2006-09-03 02:51:59 H=lns-bzn-21-82-248-157-212.adsl.proxad.net (7vgur.uwau.optonline.net) [82.248.157.212] F=<annex.compute@rozierteam.com> rejected RCPT <help@victim.com>: No Such User Here


    Notice they are all from the same IP address. Some spammers will only try to send 1, 2, or maybe 3 emails, then disconnect, and reconnect with a new IP address. My question is, about how often would you say you are seeing more than 3 delivery attempts to invalid recipients (edit: from the SAME IP address I mean)?
     
    #11 randomuser, Sep 3, 2006
    Last edited: Sep 3, 2006
  12. mike25

    mike25 Well-Known Member

    Joined:
    Aug 29, 2003
    Messages:
    83
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Raleigh NC, USA
    This attack was occuring constantly. At one point my exim_deny had over 20K IPs in it. The attack goes on at a low level for several hours and then multiple IPs suddenly become involved overloading the server. I tweaked several exim settings and lowered the max connections down to 5 which created other problems on the server but seemed to lower the odds of being ddos'd. but still thet attack would come back with 30 or so new IPs each connecting 5 times about once per day. I really just had to keep constantly reading the exim logs and blacklisting the main attackers while ignoring the slower dictionary attackers. The attack has stopped now, I guess I made it harder for them to have fun, but I was never really able to completely prevent it.
     
  13. randomuser

    randomuser Well-Known Member

    Joined:
    Jun 25, 2005
    Messages:
    147
    Likes Received:
    0
    Trophy Points:
    16
    I understand that, but my question was basically: on average, were you seeing 3 or more invalid recipients per connection per IP address at the time of the attacks? I guess you must have with 20 - 40 connections per IP address.

    I don't agree with storing blocked IP addresses for more than a short time myself, based on my own personal experience with battling spammers. The reason is, they seem to use different hosts to push spam constantly, and (mostly) only send to a few invalid recipients per IP address. IF they're mostly only sending to a few invalid recipients, then disconnecting, then reconnecing from a new host, I see no point in blocking 20,000 IP addresses. That could be a waste of resources, especially if Exim is comparing the source IP address of each new connection with its block list.

    For example:

    2006-09-03 04:02:43 # first line in exim_mainlog
    2006-09-04 01:38:59 # last line in exim_mainlog

    # wc -l exim_mainlog
    381538 exim_mainlog


    So, over a period of about 21.5 hours, there are over 380,000 logs in /var/log/exim_mainlog.


    # grep -c "No Such User Here" /var/log/exim_mainlog
    3953

    3,953 of those logs contain the string "No Such User Here".


    What I just checked for was to see how many unique IP addresses caused "No Such User Here" logs (due to :fail: No Such User Here being set in /etc/valiases for numerous domains on the server), and how many No Such User Here logs each of those IP addresses generated.

    # for x in `grep "No Such User Here" exim_mainlog | sed 's/.*\\[//g' | sed 's/\\].*//g' | grep -v ^2006 | grep ^[0-9] | sort -un ; do printf "$x: " ; grep -c $x exim_mainlog ; done

    The output looked like this (partial output):

    Code:
    210.x.x.x.: 4
    221.x.x.x: 2
    222.x.x.x: 6
    
    There were 1,283 unique IP addresses found sending to invalid recipients. The average amount of invalid recipients is 7 per IP address.

    The reason I asked the question and mention all of this, is because I have developed what I believe may be a superior solution to blocking spammers, where a spammer is defined as someone who has generated 3 or more No Such User Here logs. Again this relies on specific /etc/valiases settings.

    The intent isn't to prevent spam, although it will for those who use catch all email addresses (dumb), but to maintain a stable server environment throughout the duration of a spam attack. I understand that a trigger of only 3 invalid recipients is pretty low (I think the average dictionary attack detection utility trips on 4, or at least going by one that I read about here on the forums), and as such wouldn't recommend using it constantly, just during attacks. Or perhaps increasing it to 4 invalid recipients and running it 24x7 would work just fine as well. Even then, the duration of the block would be short enough that anyone who was legitimately trying to get email through would be able to again just fine in less than 1 minute after being blocked.

    I was going to test this on a domain that we hosted that was getting spammed so hard that resource issues were being caused, but that email account was dealt with before I had the chance to do so. It has been tested locally in a lab environment, and all I can say is that it works very well, and should never cause any resource issues itself, because we're not blocking that many IP addresses at any given time anyway, certainly not 20,000, more like 20 max.

    Spammers don't tend to reuse IP addresses for sending spam to a particular domain in my findings. The ones that did so just blasted out as much spam as they could as soon as they connected, and were only seen in that 1 particular attack during that brief timeframe and not in any other logs going back several weeks. So, why block so many spammers when they're not likely to return anyway is my point. To me, and this is just my opinion, not saying it's right, it's not about the quantity of blocking, it's how quickly you can block and mitigate the issue, which in this case would be immediately since it's 100% automated.

    edit: re reading my post, I just want to make it clear that I wasn't criticizing how you dealt with the issue, it sounds like you did a terrific job at mitigating the issue, and I know what a huge pain in the butt it can be to deal with spammers, and being a sysadmin in general. I was more curious than anything about getting more information from you about the attacks just so I could theorize how much effect, if any, the utility I wrote would have had.
     
    #13 randomuser, Sep 4, 2006
    Last edited: Sep 4, 2006
Loading...

Share This Page