The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Denial of Service attack

Discussion in 'E-mail Discussions' started by brianc, Dec 6, 2007.

  1. brianc

    brianc Well-Known Member

    Joined:
    May 16, 2003
    Messages:
    141
    Likes Received:
    0
    Trophy Points:
    16
    Hi Everyone:

    My server has crashed 3 times in a 24 hour period from a DOS attack via port 25. Thousands upon thousands of incoming messages were being sent to non-existing e-mail addresses of one particular domain name. I had the server configured to fail all mail sent to non-existing addresses on this account but it did not prevent the attack from crashing the server.

    I can't block the IP addresses because the attacks are coming from multiple IP addresses, both from the US and from other countries. The only thing I can think of is to inform this client to leave or to remove their MX record and have them use another domain name for their mail. Is there anything else that can be done to prevent something like this from adversely affecting the server?

    Any input would be helpful.

    Thanks!
     
  2. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Since you're named after myself, in an uncannily spooky way, I'll respond :)

    What you're experiencing is called a dictionary attack. The latest version of cpanel now provides facilities to block this, but if you haven't got CURRENT or EDGE installed you may need to go to www.configserver.com and check out their dictionary attack code. It implements a temporary block in exim which expires after an hour; this greatly reduces load on the server. You should also check out their firewall (CSF) as that's also a big help with security.

    - Brian

    Oh - ps - also look at "nolisting" (google for more information) - which would probably cause these attacks to go away silently with no machine load. Nolisting involves setting up three MXs with the lowest and highest denying connects, and the middle being the one that works. Legitimate mail programs retry immediately but DOS and Spam sources give up on failure...
     
  3. brianc

    brianc Well-Known Member

    Joined:
    May 16, 2003
    Messages:
    141
    Likes Received:
    0
    Trophy Points:
    16
    Thanks for the response Brian.

    I do have Configserver's dictionary attack acl installed and their CSF product. As for cPanel's anti-dictionary attack function, are you talking about the rate limit function? I have that enabled. If not then please let me know what function you are referring to. I am running the Release version of cPanel.

    The one particular domain name I was having problems with signed up with Mailprotector.net so the dictionary attacks on this domain has been resolved. However there was another attack that brought my server down 5am EST that was focused on 2 other domains.

    I checked out nolisting.org and they have good instructions on implementing the nolisting technique. I went ahead and put it to practice on the 2 problem domains and I have not seen a dictionary attack come in yet after I made the changes. The instructions on this web site has you using only 2 MX records, not 3.

    Liquidweb's tech support installed swapwatch on my server so if it goes down again, they will receive a report on what processes is eating up all of my memory including all the swap memory right before the server crashes.

    BTW - do you or anyone else know how I can configure exim_mainlog to record the number of connections an IP address has made to the exim server? For example:

    2006-01-21 09:22:38 SMTP connection from [201.40.9.66]:50229 I=[70.86.232.42]:25 (TCP/IP connection count = 33)

    My log entries do not include the TCP/IP connection count. The above log entry was taken from another post.

    I do think this type of attack will only increase in frequency. LiquidWeb informed me they have seen an increase in these types of DDoS attacks.
     
  4. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    No problem Brian,

    Cpanel has a new dictionary attack module which will replace configserver's module when it makes it through to release (it's brand new, from what little I know).

    The best way to do nolisting is with three MXs as I said, I know the article only mentions two, but the spammers will try either your default or high MX record, so if you have one in the middle you can cut out nearly all traffic. (Their rationale behind trying the high MX first is that it is often a secondary mail server with less effective spam protection.)

    I don't know about the connection count itself, but iptables has some rate limiting stuff built in and you could probably apply it to port 25 incoming connections. I'm not sure how much CPU goes into the rate-managing iptables stuff but that's always something to consider.

    Cheers,

    BrianC (the real one!) :cool:
     
  5. brianc

    brianc Well-Known Member

    Joined:
    May 16, 2003
    Messages:
    141
    Likes Received:
    0
    Trophy Points:
    16
    I upgraded to cPanel current to implement this new dictionary attack module. I haven't had a spike since but I am still closely monitoring this server.

    I had a spike late afternoon yesterday where the server jumped over to 150 in its load average and generated over 600 processes in about 30 seconds. They were mostly apache, exim_incoming, and getdiskused processes. I was monitoring the exim_rejectlog and exim_mainlog at that time and I did not see any signs of a massive dictionary attack coming in like the previous spikes so again I am at a loss of what is causing this server to freak out.

    As for the nolisting technique, would it be fine to have the third mx record use the same IP address is the primary/bogus mx record? I have seen a great reduction in dictionary attacks coming in since implementing it on several troublesome domains.

    BTW - I am the real BrianC. I have a driver's license and birth certificate to prove it. However I am willing to share my great name with you. You can be the European version of BrianC and I will be the (real) American version. :D
     
  6. kkargel

    kkargel Active Member

    Joined:
    Nov 28, 2007
    Messages:
    34
    Likes Received:
    0
    Trophy Points:
    6
    Don't forget good old fashioned greylisting. I use greylisting extensively and it dramatically reduces the server workload. Well over half of the email that comes in to my domains gets successfully filtered that way. I believe greylisting is supported by ASSP.
     
  7. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Bottom line with greylisting is that it does lose some email. I know some implementations are better than others but I'm not familiar with ASSP's. Would appreciate some comments, is it rock solid, have you been using it for a while, have you had problems with receiving email from some people?

    One well-known instance where it will cause problems is where an email message is coming from a group of IPs rather than being retried by the same IP, eg where the message comes from a large provider. Many greylisting implementations won't recognize the retry and will see it as a new attempt, hence the message never gets through.
     
  8. kkargel

    kkargel Active Member

    Joined:
    Nov 28, 2007
    Messages:
    34
    Likes Received:
    0
    Trophy Points:
    6
    The greylisting implementation I use works great. It does have some good options, like the ability to use the entire source netblock or MX list as the source criteria instead of just the source host address, and an ACL to list "misbehaving mail servers" so that I can whitelist servers that do not conform to the RFC.

    I am quite pleased with this solution, but I don't want to use this forum for an advertising list. If you want specific info about the manufacturer please contact me offlist. I have no financial ties with the company, but I still don't want to spam here.

    I just looked at the stats for yesterday and out of 200K messages 180K were not retried and rejected with greylisting. At the same time I had zero complaints from customers.

    IMHO greylisting takes care of 96% of spam right off the top and greatly reduces the load on your antispam and mail servers.

    I do my greylisting on a separate box and not on the cPanel server. To accomplish this I have all MX's for the domains pointing to the greylisting box and the cPanel box is access listed at the network edge so that it only accepts port 25 and 587 traffic from customer IP's. All non-local mail traffic is forced through the greylisting box. If you leave any 'sendmail' ports open to the world spammers will find them and try to use them.
     
  9. brianc

    brianc Well-Known Member

    Joined:
    May 16, 2003
    Messages:
    141
    Likes Received:
    0
    Trophy Points:
    16
    I actually went ahead and used a third party anti-spam service (http://www.easyantispam.com) and it has stopped the DOS attacks. It also has allowed me to get rid of Configservers' mailscanner package which was a huge resource/memory log. The service filters about 80% of the spam before it even touches my server and then cPanel's exim/spamassassin's service stops the rest. I have had clients tell me their spam has been reduced to practically zero and I am now able to put more clients on this server due to the savings in resources. That alone will pay off the extra $30 per month I have to pay for the service.
     
  10. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Although their service is cheap it isn't perfect - you get what you pay for.

    http://www.webhostingtalk.com/showthread.php?t=625200

    The problems aren't enormous, to be fair, and the service may be fine for many people, but thee is some interesting discussion on the link above.
     
  11. brianc

    brianc Well-Known Member

    Joined:
    May 16, 2003
    Messages:
    141
    Likes Received:
    0
    Trophy Points:
    16
    Thanks for your input Brian. BTW - have you used them personally? Because if you have not then you really should not criticize them based upon a couple of reviewers.

    I read the WHT's thread that you referred to before signing up with them. I was satisfied with George's responses. The bottom line is that before I used them for this one server, the server load's 1 minute average was consistently over 2, now its around 0.30. Mailscanner is just not a wise choice for busy mailservers. It uses way too much resources. I was also not very happy with the support I was receiving from Config server when I was running into issues.

    Another major issue is the dictionary attack acl that comes with configserver's package. I was experiencing a major dictionary attack on this server and it was finally determined that it was the dictionary attack ACL that was bringing the server down. That is nobody's fault however we realize that using it posed a serious threat to the integrity of the server.

    I think with the enormous problem with spam, it no longer makes sense to run everything on one server. I love the idea in having an anti-spam system in place that will filter out the bulk of the spam BEFORE it touches your server. In our case, we use cPanel's exim/spamassassin's setup to filter the rest that makes it through. So far we have had no reported problems with false positives.
     
  12. hostmedic

    hostmedic Well-Known Member

    Joined:
    Apr 30, 2003
    Messages:
    559
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Washington Court House, Ohio, United States
    cPanel Access Level:
    DataCenter Provider
    swapwatch

    any clue where to find swapwatch ...

    I have tried a few searches - google - even on a few linux websites

    thanks
     
  13. brianc

    brianc Well-Known Member

    Joined:
    May 16, 2003
    Messages:
    141
    Likes Received:
    0
    Trophy Points:
    16
    The swapwatch script was provided to me by Liquidweb's tech support. I believe it belongs to them.
     
  14. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    I note you have had a good experience, so perhaps you should comment in the thread I quoted. I was simply making sure the coverage was balanced and noting that they appear to be a cheaper service. For many people, they may be ideal.

    It's interesting that you had problems with the configserver dictionary attack ACL - for many, many people it's actually been the solution to dictionary attacks. Without details I'm guessing your server was very low spec (VPS?) and found it difficult to run the perl scripts required to do the message bouncing in that ACL. Either that or you had it misconfigured somehow. Configserver's support can be slower on occasion for the simple reason they're so popular they get overwhelmed. I've always been very happy with it myself, having used them for three years now.

    It's great that this solved the problem so nicely for you! But despite your comments I need to make the point that many people run everything on one server just fine. There are complexities in introducing other servers that can make it easier to delay or lose email, unless one is careful - that's true of any introduced complexity. I'm just wanting to balance your comments; your experience differs from many others around here.

    If your server doesn't cope with a heavy spam load, another solution is ASSP. Because of the way it works (proxy architecture, intervening before the spam gets to Exim) it places a much smaller load on a server, may be worth looking at if you ever need to provide your own spam service (which you may not!).
     
  15. brianc

    brianc Well-Known Member

    Joined:
    May 16, 2003
    Messages:
    141
    Likes Received:
    0
    Trophy Points:
    16
    As I said previously in this thread, this was no normal dictionary attack. It was a denial of service of attack using a dictionary attack method. When you have thousands of different IP addresses sending mail to non-existing e-mail addresses on the server every 2-3 seconds, then it will be the dictionary attack acl that will bring your server down. That is because its logging the IP address to the exim.deny file and at the same time it is referencing the file to deny IP addresses and all the read/write activity that is going on to keep up with several thousand IP addresses every 2-3 seconds brings the server down.

    BTW - This was no low end server. It was a quad-core with 4 gig of ram.

    As for configserver's support, everyone's mileage will vary. For me the wait times (sometimes 48 hours) to get a response is unacceptable especially when I have purchase plenty of services from them in the past. I don't care about popularity I care about customer service. If a company can't provide it then I need to go somewhere else.

    There are also many complaints on these forums of the server load that mailscanner causes when it runs on a server. So my experience is not so different just my solution. The main point to keep in mind is using configserver's dictionary attack acl, imho, poses a serious security threat in the case of a particular denial of service attack. I am not talking about the thousands of spam that is sent daily to non-existing e-mail addresses that a dictionary attack ACL covers, but a DDOS attack in which thousands of IP addresses are hitting fake e-mail addresses every 2-3 seconds.
     
Loading...

Share This Page