Intermittent exim gmail smtp connection timeout

Hi all,

I'm hoping someone here might have some thoughts as I'm really getting confused about an issue I'm seeing.

Over the weekend, I migrated two small clients to a new, clean, WHM installation (Centos 7) and today started getting complaints about mail that was taking hours to arrive. In looking at the Mail Queue Manager I saw there was a large amount of unsent mail that had built up over the day because it was getting timeout errors when trying to deliver to gmail servers (all the mail in the queue seemed to be going to gmail - but both companies almost exclusively use forwarders to send mail addressed to their domains to free gmail accounts - so the vast majority of the traffic on their accounts is just receiving e-mail and forwarding it through to gmail).

So /var/log/exim_mainlog is full of:

Code:

2016-02-02 00:24:33 1aQOji-0000oI-95 H=gmail-smtp-in.l.google.com [173.194.204.27] Connection timed out
2016-02-02 00:26:40 1aQOji-0000oI-95 H=alt2.gmail-smtp-in.l.google.com [74.125.133.27] Connection timed out
2016-02-02 00:28:48 1aQOji-0000oI-95 H=alt3.gmail-smtp-in.l.google.com [173.194.65.27] Connection timed out
2016-02-02 00:30:55 1aQOji-0000oI-95 H=alt4.gmail-smtp-in.l.google.com [173.194.222.27] Connection timed out

Now what I can't seem to wrap my head around is that it seems completely intermittent. Some e-mail goes through immediately without issue. Some sits in the queue for hours and then eventually goes through hours later on it's own.

If I try to force Delivery attempts from the Mail Queue Manager, sometimes it will immediately send without issue, but mostly it will eventually timeout. If I try to Deliver a message from the queue after another has *just* been accepted, it will *always* time out. I've never been able to force deliver two messages successfully in a row.

It doesn't appear to be a firewall issue - I've tried temporarily disabling csf and it makes no difference, and I can telnet from the server to any of the gmail smtp machines (both the main MX entry, and the alts) with no issues (when the firewall is both enabled and disabled).

Could it possibly be outgoing / incoming rate limiting controls somewhere upstream? The system is an AWS EC2 instance, and I know they have some sort of SMTP limit - but combined the machine probably has less than 10 accounts on it at the moment - so it's hardly a huge amount of mail.

Would greatly appreciate any new ideas on where else I should be looking (or if I've just completely overlooked something completely obvious) as I feel like I'm kind of hitting a wall here.

Best,

- Brad

 

Just an update for anyone with a similar issue - with some fresh eyes and some more googling it sounds like this may be caused by some SMTP rate limitations built into the AWS EC2 network as Spam prevention.

They have a form to register to remove outgoing smtp connection limitations here:
Amazon Web Services Sign In

I've submitted and will update if this resolves the issues I was seeing.

 

Amazon SMTP traffic management indeed seems to have been the cause. Within a couple of hours of filling out the above form, I got an email confirmation from AWS that "traffic restrictions had been removed" and normal function resumed immediately.

Confusing the matters is that this SMTP traffic management is not documented well (and sometimes with contradicting information). It does not appear to be a hard cap limit, nor does it trigger any notification when it's applied - it actually appears to be a *throttle* on common SMTP ports, triggered by a very small number of connections, beyond which it allows a certain number of connections per/hour - which would absolutely create the kind of "intermittent" connectivity issues I saw (and the odd delivery order of mail in the queue depending on when a retry "won the lottery" to negotiate a connection).

Anyway - I hope that info is of some use to others in the future!