sustained exim attack - syntax errors - mitigation measures

[BigLebowski]

Hi there

One one of our servers, the exim_mainlog is littered with these messages which I suspect is an attempt to buffer-overflow exim. There are too many source IPs to practically block. Any ideas how to filter these out?

2013-04-15 14:14:36 [22631] SMTP syntax error in "\373\315\243\302\fq<\003\2618C\215\274\231\307\225]L\307\017\271\227\310\321ugELO|\311\204\200G\312\001\207g?\363\272\200A7\323\373G\212\266]\313\377L\226A`J\301\314\001?\363\315\372=r\316\265\b$\317mLHF\334s\231\317\203;\264E\217n\005\321\251:\214F\021kWGSZ\206\322\265\314\025H\341\313\231\323r\300\313\324\234%d\325\034\276\311\325\341g\007MT%\300\340K\365`\326Y\365?N\035\215\345N\254\256\223\330\222\354\333\330\320\257\301\331\021\[email protected]\332\266F*\333lA#\334\267K\020S%\216\250S\017H\016Tz\305#Z\361\264\024Q\242r\020\337\252=\352\336\337\325\242T:~\316\337\224\224\261\341\326\256\352X\215\356\230\342\364#\345\342\354\214\255\343\2461\233ZB\222\370\343\203\220\273I'\376ZJU\374\336S\341"\263\346\274\022\335\346dF\250\347\301\255\357\350q\036\020`\032\345\206\351u\257|O\031\035\034PG\033\240Y\271uA\354\331h\241\354\240\332\200\354\377t\273\354\232\215\003jNf\227\356\207\374{\357\021b\024\360I\2458gq\314" H=[95.77.252.179]:51242 I=[195.238.172.81]:25 NULL character(s) present (shown as '?')

Best
Dude

 

[Sash]

I've got the same problem? Were you able to resolve it?

2013-08-07 21:23:20 SMTP call from [X.X.X.X]:58157 dropped: too many syntax or protocol errors (last command was "?o.Šò¹å:þ…Ê¿hµb5#a’µT™ÊµÅiš/þa:¶§")
013-08-07 21:23:21 SMTP call from [X.X.X.X]:37594 dropped: too many syntax or protocol errors (last command was "ºIãŠiˆ÷%
Ôª$Ÿ8Hˆ|ÛpƒÔj\}.%šéÏu?^è5"eeˆ²€&g"†Ÿ‰ÇKëk%W¾.³]…GK9Eäž]Š”ö:ôH¬úzÚ¶0àg ž@ÕŸªË%EŠýL^Góì›R£6¤›(öËŒ´™Ø‡Z9lŸúµ|****EoÉ1êÎ_+°qgr”1±y\5ïв¿go:¢z±†RP6****ô6q΃ÔÓú~6RŠæ$æÿ")

 

[cPanelMichael]

Hello

I recommend contacting your datacenter/NOC to see you can implement a firewall/DDoS device that can mitigate the DDoS.

Thank you.

 

[quizknows]

This would definitely be best mitigated by the data center. However, that can be extremely expensive. I found a server under this attack and this was my solution.

I was able to make a rule for csf's regex.custom.pm file. Use at your own risk and only if you're currently under this attack. This will permanantly block any IP's that cause more than two "too many syntax or protocol errors" entries in the log.

in /etc/csf/regex.custom.pm add in the middle of the file:

      if (($lgfile eq $config{CUSTOM1_LOG}) and ($line =~ /^\d+\D+\d+\D+\d+ \d+\D+\d+\D+\d+ SMTP call from \[(\d+\.\d+\.\d+\.\d+)\]:\d+ dropped: too many syntax or protocol errors/)) {
               return ("Failed exim syntax from","$1","eximdosmatch","2","25,143","1");
       }

Then in /etc/csf/csf.conf, set CUSTOM1_LOG to /var/log/exim_mainlog

As root, restart both CSF and LFD.

 

[quizknows]

Edit button seems MIA, but anyway, if you use the above solution to try to block this, make sure you disable both LF_EMAIL_ALERT and LF_PERMBLOCK_ALERT in csf.conf.

Also, updated regex to catch more hosts:

      if (($lgfile eq $config{CUSTOM1_LOG}) and ($line =~ /^\d+\D+\d+\D+\d+ \d+\D+\d+\D+\d+ SMTP call from.*\[(\d+\.\d+\.\d+\.\d+)\]:\d+ dropped: too many syntax or protocol errors/)) {
               return ("Failed exim syntax from","$1","eximdosmatch","2","25,143","1");
       }

 

[quizknows]

One update, there are two different regular expressions you might need. Be sure to read the above 2 posts too.

If your logs look like this:
2013-08-10 10:40:39 SMTP call from [SOMEIPADDRESS]:29048 dropped: too many syntax or protocol errors

Use the one I posted above. If they have more info like this:
2013-08-22 15:02:39 [13788] SMTP call from [SOMEIPADDRESS]:25910 I=[SOMEOTHERIP]:25 dropped: too many syntax or protocol errors

Then use this one:

if (($lgfile eq $config{CUSTOM1_LOG}) and ($line =~ /^\d+\D+\d+\D+\d+ \d+\D+\d+\D+\d+ \[\d+\] SMTP call from.*\[(\d+\.\d+\.\d+\.\d+)\]:\d+ I=\[(\d+\.\d+\.\d+\.\d+)\]:\d+ dropped: too many syntax or protocol errors/)) {
               return ("Failed exim syntax from","$1","eximdosmatch","2","25,143","1");
       }

It is also worth noting that you should disable both LF_EMAIL_ALERT and LF_PERMBLOCK_ALERT so that CSF does not clog your mail queue, as it will be blocking a LOT of IP addresses. You may want to raise the IP deny limit too.

 

[nyoman]

Thanks for the trick,

And how to see the scripts already work or not ?
I mean where is the log of those ip attacker keep/save ? Sorry, i am newbie on CSF

 

[quizknows]

If you tail the log /var/log/lfd.log, i.e. tail -f /var/log/lfd.log

Make sure to properly restart both services:

/etc/init.d/lfd restart ; csf -r

 

[ethical]

any idea how to adapt this patch if using ASSP and assp deluxe? since its an smtp proxy everythign goes through it first?
in my assp logs all i see are repeated entries like this:

Sep-04-13 20:29:12 82.152.113.229 got 'non printable hex data' from the client before the '220 ...' server greeting was sent - rejecting connection;
Sep-04-13 20:29:12 82.152.113.229 All connections from IP 82.152.113.229 will be rejected by assp for the next 15-30 minutes.;
thanks

John

 

[quizknows]

John,

Honestly it looks like ASSP is pretty well mitigating this for you by blocking the IPs temporarily.

If you did want to permanantly block any IP's that trip the 'non printable hex data' (i.e. that are part of this attack), it wouldn't be too hard to adapt the regex (assuming you have csf). This is what I came up with which should work and will do a perm block (add to the middle of /etc/csf/regex.custom.pm ):

if (($lgfile eq $config{CUSTOM1_LOG}) and ($line =~ /^\w+\D\d+\D\d+ \d+\D+\d+\D+\d+ (\d+\.\d+\.\d+\.\d+) got .non printable hex data. from the client before the .220 \.\.\.. server greeting was sent/)) {
               return ("non printable hex data from","$1","smtpdosmatch","1","25,143","1");
       }

In the CSF conf, set CUSTOM1_LOG to the full path of the log file you pulled the above info from.

restart csf / lfd

/etc/init.d/lfd restart ; csf -r

You should be able to tail -f /var/log/lfd.log and watch as it starts blocking IP addresses.

 

[ethical]

Hey Quiz

thanks for that! well even though it says its blocking it for 15-30 minutes, its not actually doing that from what i can tell... I gave it a try but doesn't seem to trigger it yet.

i tried several other small tweaks including this just in case i had to escape the '

if (($lgfile eq $config{CUSTOM1_LOG}) and ($line =~ /^\w+\D\d+\D\d+ \d+\D+\d+\D+\d+ (\d+\.\d+\.\d+\.\d+) got \'non printable hex data\' from the client before the .220 \.\.\.. server greeting was sent/)) {
               return ("non printable hex data","$1","smtpdosmatch","1","25,143","1");
       }

any other thoughts?

Thanks!

John,

Honestly it looks like ASSP is pretty well mitigating this for you by blocking the IPs temporarily.

If you did want to permanantly block any IP's that trip the 'non printable hex data' (i.e. that are part of this attack), it wouldn't be too hard to adapt the regex (assuming you have csf). This is what I came up with which should work and will do a perm block (add to the middle of /etc/csf/regex.custom.pm ):

if (($lgfile eq $config{CUSTOM1_LOG}) and ($line =~ /^\w+\D\d+\D\d+ \d+\D+\d+\D+\d+ (\d+\.\d+\.\d+\.\d+) got .non printable hex data. from the client before the .220 \.\.\.. server greeting was sent/)) {
               return ("non printable hex data from","$1","smtpdosmatch","1","25,143","1");
       }

In the CSF conf, set CUSTOM1_LOG to the full path of the log file you pulled the above info from.

restart csf / lfd

/etc/init.d/lfd restart ; csf -r

You should be able to tail -f /var/log/lfd.log and watch as it starts blocking IP addresses.

 

[quizknows]

I use pcregrep to check my regular expressions. I used a . instead of \' because my shell didn't like the single quotes, even when escaped. A period can take the place of any single character. I took the log data you gave me and used pcregrep with the expression I posted, and it was matching/returning the lines just fine.

You definitely set CUSTOM1_LOG correctly and ensured that LFD was restarted too? It should start banning immediately if so.

 

[ethical]

yes that was right but in a new day i see the problem.... your code is correct but we were missing the [EarlyTalker] part of the line it works now with this or of course with the . instead of the \[

if (($lgfile eq $config{CUSTOM1_LOG}) and ($line =~ /^\w+\D\d+\D\d+ \d+\D+\d+\D+\d+ (\d+\.\d+\.\d+\.\d+) \[EarlyTalker\] got .non printable hex data. from the client before the .220 \.\.\.. server greeting was sent/)) {
        return ("non printable hex data from","$1","smtpdosmatch","1","25,143","1");
}

thanks for your help!!

I use pcregrep to check my regular expressions. I used a . instead of \' because my shell didn't like the single quotes, even when escaped. A period can take the place of any single character. I took the log data you gave me and used pcregrep with the expression I posted, and it was matching/returning the lines just fine.

You definitely set CUSTOM1_LOG correctly and ensured that LFD was restarted too? It should start banning immediately if so.

 

[eurorocco]

Sample line in /var/log/lfd.log

Sep 25 17:24:29 SERVERNAME lfd[10237]: (eximdosmatch) Failed exim syntax from SOURCEIP (SOURCEIPNAME): 2 in the last 300 secs - *Blocked in csf* [LF_CUSTOMTRIGGER]

Excellent. Thanks!

 

[fcbinfo]

Some help, sorry about asking for this, but I have a firewall behind my server, and that can be better.

2013-10-24 16:22:36 SMTP call from [178.223.153.182]:2253 dropped: too many syntax or protocol errors (last command was "¾®‘Ä“]Þs—W'****˜¸›Ýõ****ŠÄ°ë¶ï© ÆN¿¢×-Ò$õr§F
A©È“ȬBœs-÷·


Anyway to do a iptables rule where that not contain utf8 or ascii characters to port 25 get droped?

Something like this:

iptables -A mychainrule -m string --string "(DONT HAVE UTF8 / OR HAVE INCORRECT CHARS)" --algo bm -j DROP

Its boring me!

=/

Thanks for help

- - - Updated - - -

Well... now I'm thinking here. If i do this, I'll drop attachments. True?

 

[quizknows]

You might be able to do something like that, but it depends on the attachment encoding. A lot of times attachments are base64 encoded for transport, so they're technically ASCII characters. I wouldn't risk it personally, but if you figure it out please share

 

[fcbinfo]

I'm getting about 30 e-mails per 5 min...

The problem, i have set 1000 perm ban, and it's full.

I have so many udp connections on port 53 too. Is that because this problem with exim?

I also tried to do what you say on that post, but I believe its not working for me.

=/

Scared with this... thinking in abandon web-hosting.

Lot of problems for little money!

 

[quizknows]

The "good" thing about this attack is I haven't seen it take down any servers, only slow them down a little bit.

If you are on a dedicated server, you could probably get away with more like 2500 for deny limit. If you are on a VPS, the limit of 1,000 is probably reasonable.

You should be able to tail -f the /var/log/lfd.log and see this denying IPs if it is working. You can try both versions of the regex I posted, and make sure that the relevant LFD alert settings are turned off.

The connections on port 53 shouldn't be related, though your server may be doing reverse DNS lookups for the IPs that are connecting.

The main problems I've seen caused by this attack are when the exim logs are on a small partition (i.e. /var/ is it's own partition and is not very big) it can result in filling the partition space causing other issues. You can check your logrotate.d settings to make them rotate logs more frequently until this attack goes away.

 

[fcbinfo]

Thanks for help.

Yes. The server still online and very fast. But this so many connections, hinet.net, baiduspider, and a lot of other connections from china, taiwan, viatenan and others from asia, and so many connections from EU too.

The host, has 192gb of ram, 16x hdd raid-10, 2x E2670.

The guest, with cPanel installed, now has 96gb of ram, 16 cores. Very fast server.

I'm under this little attack for more than 20 or 30 days. It's never stop. Anyway, its boring me when I see these connections to my server.

Then someone says that maybe my server has some virus or malware. Tried all too. Include buy the anti-malware from csf.

No way, no malwares, no virus... but... this still boring me.

I also tried all of other tools too. =/

 

[quizknows]

I doubt there is malware on your server, I've seen many hosts under this attack and it's inbound, it has nothing to do with anything found on the server itself (or at least I've been unable to discover that)

The regex in post 4 should match your log entries. be sure to restart lfd in addition to csf. The easiest way is:

csf -x ; csf -e

which will completely disable/re-enable CSF and LFD.

Seeing as you have a very nice server, as long as disk space for the logs is not an issue, this shouldn't cause you any "real" problems other than being annoying.