exim: Connection lost while reading message data (header)

[Erik Knepfler]

I'm getting these messages in the exim_mainlog while trying to set up an email account in either Blue Mail or on an iPhone. However, they do not appear when using Outlook or Thunderbird!

SMTP connection from (localhost) [my ip address here]:1493 lost while reading message data (header)

I've found little online about this, the only suggestion was to change the ethernet MTU from 1500 to 9000 via ifconfig, which I tried, and restarted the networking service and verified is set.

It would *seem* that Outlook/Thunderbird are doing something very different when testing outgoing server settings in some way that works, while Blue Mail and iPhones suddenly are having a problem with the commands being exchanged back and forth during the SMTP authentication session. I hadn't tried Blue Mail in a while, and this was prompted when a customer was setting up her iPhone after updating iOS.

I think it is possibly not a coincidence that A.) I haven't tried BlueMail in a while and it's a more modern app, 2.) the iPhone was just updated, 3.) Outlook and Thunderbird are "older".

It certainly seems like whatever Outlook and Thunderbird are expecting from the server is being met, while whatever Blue Mail and iPhones are expecting is not being met, and the clients are terminating the connections.

Suggestions?

 

[keat63]

is :1493 the port number ???
If so, it seems very odd.

If not, ignore everything I just said.

 

[cPanelLauren]

I agree, 1493 is an odd port to be using and it could be the issue, see: lost while reading message data (header) where they were using 25 which was closed by their provider.

Also, there are a few other reasons this could happen. One of them is related to attachments, or antivirus, though it could also be network related. The following threads might be helpful:

SMTP Connection - lost while reading message data

I've got a user that is unable to send out emails with attachments. Sending out a simple email with no attachment works just fine. The exim_mainlog on the server is showing: SMTP connection from XXXXXXXXX I=[XX.XX.XX.XX.XX]:PPP lost while reading message data (header) I am not able to...

forums.cpanel.net

SOLVED - SMTP connection from - lost while reading message data

Hello, I use a VPS Server with centOS 7 and Cpanel 74.0 (Build 10). One of my customers are trying to send me some emails but I'm unable to receive them: [[email protected] ~]# grep /var/log/exim_mainlog | tail 2018-11-16 09:59:12.171 [13336] SMTP connection from mail. (Hostname) [IPServer]:34366...

forums.cpanel.net

lost while reading message data

I am 15 days with a difficult problem to solve. A lot of my clients are unable to send emails with attachments. When you send email without attachment, send normally. In emails with attachment generates timeout for the client. After a while the some email clients can send e-mail I thought it...

forums.cpanel.net

SMTP lost connection

For some reason, one of the servers can't connect to my mail server. Whenever a user tries to send email from that server to my server, the message won't go through and I see the following in the logs (var/log/exim/mainlog): 2007-02-13 23:56:06 SMTP connection from (***.ca) [***.***.***.***]...

forums.cpanel.net

 

[Erik Knepfler]

1493 is an odd port to be using

No no, I think that's a misunderstanding. Exim is running on the usual ports 25, 587, 465. Look at these additional log entries. The IP addresses are the only thing I've redacted, the rest is verbatim, note how the port is different each time:

2020-01-30 18:52:01 SMTP connection from (localhost) [my ip address]:1381 lost while reading message data (header)
2020-01-30 18:52:06 SMTP connection from (localhost) [my ip address]:1383 lost while reading message data (header)
2020-01-30 18:53:16 SMTP connection from (localhost) [my ip address]:1397 lost while reading message data (header)
2020-01-30 18:53:23 SMTP connection from (localhost) [my ip address]:1400 lost while reading message data (header)
2020-01-30 18:58:41 SMTP connection from (localhost) [my ip address]:1442 lost while reading message data (header)
2020-01-30 18:58:53 SMTP connection from (localhost) [my ip address]:1444 lost while reading message data (header)
2020-01-30 19:02:09 SMTP connection from (localhost) [my ip address]:1596 lost while reading message data (header)

2020-01-30 19:16:37 SMTP connection from my-ip-address.client.mchsi.com (localhost) [my ip address]:1938 lost while reading message data (header)
2020-01-30 19:16:57 SMTP connection from my-ip-address.client.mchsi.com (localhost) [my ip address]:1941 lost while reading message data (header)
2020-01-30 19:17:04 SMTP connection from my-ip-address.client.mchsi.com (localhost) [my ip address]:1944 lost while reading message data (header)
2020-01-30 19:17:08 SMTP connection from my-ip-address.client.mchsi.com (localhost) [my ip address]:1946 lost while reading message data (header)

So, I think it's normal for the port to vary, as I think that represents some kind of internal allocation port after the connection happens, but these are being directly triggered by attempted SMTP authentications on ports 25, 587, or 465, with or without TLS/SSL in any combination. I know this because I set a tail -f exim_mainlog | grep myipaddress and watch it in real time as I attempt to connect. And these happen with iPhone or Blue Mail, and not with Outlook or Thunderbird, which breeze through and work just fine.

At first I always suspect I have some setting wrong like the port on the client, or the SMTP authentication wrong in some way which is the most typical answer to this kind of failure to authenticate on the client. So I check all that as carefully as I can, and keep in mind I host hundreds of sites with probably a thousand email accounts over hundreds of clients and spend the better parts of my life walking people through how to configure this stuff in Outlook or their phones or whatever program they're using. Yet I can only make Outlook and Thunderbird work, not Blue Mail or an iPhone. It happens the moment that the programs attempt to validate the outgoing SMTP settings with an authenticated login.

I suspected antivirus before posting here due to the connection being suddenly closed, and most likely on the client, which is a pretty typical thing for an antivirus to do. I have ClamAV on the server, and disabled it via the service manager and that didn't help. On my client PC, I use only Windows Defender Antivirus, Defender Firewall, and Malwarebytes Anti-Malware, and I tried disabling all three, but no change. My customer was on an iPhone. It's hard to comprehend why both a PC with no antivirus and an iPhone would fail to authenticate with the connection being dropped, and only with specific apps and not others.

I use APF firewall with BFD and some custom scripts to feed it, but I've whitelisted my IP in both APF and cpHulk and even in Exim as a trusted mail host. Anywhere I could whitelist my IP, I have done so, but that doesn't help either.

All RBLs are disabled (besides, I whitelisted my IP against RBL checks in the exim config. I've tried everything! Well, except whatever is the real cause.) The server is not on any known email blacklists. I've even tried adding my IP to "Sender verification bypass IP addresses" in Exim. I'm running out of ideas.

It cannot be that the ports are closed by any provider in between because Outlook and Thunderbird work, and I can telnet to those ports just fine.

I changed the server MTU from 1500 to 9000 on the ethernet interfaces directly. I tried changing it on my Windows client but ran into a problem, I think I have to adjust that on the router first too, so I'll be trying that soon. Another task will be to install WireShark and inspect the SMTP conversation directly...

 

[cPanelLauren]

So, I think it's normal for the port to vary, as I think that represents some kind of internal allocation port after the connection happens,

You're right, looking at some of the other instances, it's prevalent throughout the rest of them

I suspected antivirus before posting here due to the connection being suddenly closed, and most likely on the client, which is a pretty typical thing for an antivirus to do. I have ClamAV on the server, and disabled it via the service manager and that didn't help. On my client PC, I use only Windows Defender Antivirus, Defender Firewall, and Malwarebytes Anti-Malware, and I tried disabling all three, but no change. My customer was on an iPhone. It's hard to comprehend why both a PC with no antivirus and an iPhone would fail to authenticate with the connection being dropped, and only with specific apps and not others.

I haven't used Blue mail but I do use an iPhone with mac mail to get my own cPanel mail for multiple accounts and I don't have this issue, so I am leaning toward a server configuration issue or networking. Is it Blue Mail on both iPhone and Android devices or just Android?

Really you shouldn't *have* to change the MTU for normal mail delivery. You're referencing the client PC but you noted the issue is only occurring on android and iPhone devices so, it's possible the behavior on the PC is irrelevant. Is there antivirus on the device?

I changed the server MTU from 1500 to 9000 on the ethernet interfaces directly. I tried changing it on my Windows client but ran into a problem, I think I have to adjust that on the router first too, so I'll be trying that soon. Another task will be to install WireShark and inspect the SMTP conversation directly...

TBH I'd go the WireShark route first and see what's happening before changing anything.

 

[sparek-3]

I'm assuming the 1493 being referenced is the port number that the server is connecting to back to the user.

Can you please clarify is [my ip address here] your server's IP address or YOUR IP address - i.e. what you'd see at http://www.whatismyip.com?

My guess is that you are trying to make a secure connection on a non-secure port or a non-secure connection on a secure port. But I have no idea how iPhones, iCloud, iApple does all of this.

The other thing that grinds my gears is that there is no set way for all of these email clients to distinguish between securing over a connection that expects to see TLS (commonly this is port 465 for SMTP, 993 for IMAP, and 995 for POP3... although these ports numbers technically aren't standard) or upgrading a connection to TLS with STARTTLS (SMTP and IMAP) and STLS (POP3) which can happen on regular standard ports (ports 25 and 587 for SMTP, port 143 for IMAP, and port 110 for POP3).

If you are telling your email client to use a STARTTLS or STLS connection and you are also telling it to use a secure port (like 465) then the connection coming into the server on port 465 is expecting a TLS handshake. But that may never happen if the client isn't configured to do that.

How do you know if you need to use a secure port or if a session-based TLS upgrade is being done? Who knows! Especially with any Apple products.

 

[rackaid]

These errors can be difficult to debug.

Note that the ports in these logs are not the server's ports but the ports of the client. So this will vary each time as the client will use different ports to connect to the STMP port. The server port would always be one of the listening SMTP ports (at least for inbound email).

This error:

SMTP connection from example.com [IP-ADDRES] lost while reading message data (header)

Means the connection dropped during the transmission of the email.

Unfortunately, the drop can occur at any point between the HELO and QUIT commands. If an exim process hangs during this phase, you can also get this error as the SMTP connection will timeout. Exim can hang for numerous reasons.

• RBLs being blocked, but you said those were disabled.
• Sender verification callouts can be a problem as well. Though you whitelisted your IP, I would try just turning them off.
• Firewalls may block remote DNS/RBL lookups. You may want to shutdown the server's firewall and run a test. Whitelisting your IP will not help if exim is trying to call out and getting blocked by your firewall or the calllout address is not responding.

Run a Manual SMTP Test
I recommend you run a manual mail test using telnet. Telnet/Openssl into the server and send a message.

DNS problems can happen if domain validation is happening and you have slow DNS resolution. Also, I've seen some servers fail after the RCPT TO command as the user lookup is slow for some reason.

You can enable debug logging by adding:

log_selector=+all

In the exim configuration using the WHM editor.

TLS issues should not be ruled out but I would expect those to happen at the connection time and not generate the error you see.

 

[cPanelLauren]

SSL/TLS issues I believe are logged differently as well, it will output that there was an SSL/TLS protocol or connection error. For example:

020-01-28 18:24:32.858 [40451] TLS error on connection from [<RemoteIP>]:38526 I=[<ServerIP>]:465 (SSL_accept): error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
2020-01-29 18:25:31.401 [38826] TLS error on connection from [<RemoteIP>]:49952 I=[<ServerIP>]:465 (SSL_accept): error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

All of those are related to connection attempts using the SSL protocol where my test server only accepts TLS connections.

How do you know if you need to use a secure port or if a session-based TLS upgrade is being done? Who knows! Especially with any Apple products.

Can you clarify what you mean? Apple's mail setup when using iOS and a desktop are pretty standard - these are from an account I just set up:




It does use TLS though: (on the phone they say SSL but this really is either SSL or TLS)

[[email protected] ~]# exigrep 1ixdJv-000A2V-Oy /var/log/exim_mainlog
2020-01-31 14:55:20.414 [38597] cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1ixdJv-000A2V-Oy

2020-01-31 14:55:20.401 [38595] 1ixdJv-000A2V-Oy H=([192.168.21.110]) [50.233.137.10]:52512 I=[<ServerIP>]:587 Warning: "SpamAssassin as cpaneleximscanner detected OUTGOING smtp message as NOT spam (-1.0/20)"
2020-01-31 14:55:20.404 [38595] 1ixdJv-000A2V-Oy <= [email protected] H=([192.168.21.110]) [50.233.137.10]:52512 I=[<ServerIP>]:587 P=esmtpsa L- X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=no SNI="mail.mydomain.tld" A=dovecot_plain:[email protected] S=685 M8S=0 RT=0.011s [email protected] T="Test" from <[email protected]> for [email protected]
2020-01-31 14:55:20.472 [38597] 1ixdJv-000A2V-Oy Sender identification U=mydomain D=mydomain.tld [email protected]
2020-01-31 14:55:20.476 [38597] 1ixdJv-000A2V-Oy SMTP connection outbound 1580504120 1ixdJv-000A2V-Oy mydomain.net [email protected]
2020-01-31 14:55:20.919 [38599] 1ixdJv-000A2V-Oy "[email protected]" from from: rewritten as "[email protected]" by rule 1
2020-01-31 14:55:21.230 [38597] 1ixdJv-000A2V-Oy => [email protected] F=<[email protected]> P=<[email protected]> R=dkim_lookuphost T=dkim_remote_smtp S=2101 H=gmail-smtp-in.l.google.com [142.250.8.26]:25 I=[<ServerIP>]:42360 X=TLS1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=yes DN="/C=US/ST=California/L=Mountain View/O=Google LLC/CN=mx.google.com" L C="250 2.0.0 OK  1580504121 w9si5249830otk.91 - gsmtp" QT=1.456s DT=0.663s
2020-01-31 14:55:21.230 [38597] 1ixdJv-000A2V-Oy Completed QT=1.456s

I recommend you run a manual mail test using telnet. Telnet/Openssl into the server and send a message.

This advice is great, in theory but if you're having an issue connecting from a mobile device it's really not quite so simple. This is more useful if you're not able to send/receive mail at all.

 

[sparek-3]

Can we agree that there are fundamental differences between using a secure port (i.e. port 465) throughout the connection and using a non-secure port to initially connect and then upgrade that connection with STARTTLS? If that is not in agreement - then that kind of shoots the whole thing in the water.

Now if the above is a given... then what does Use SSL mean?

Does it mean that the connection is secure throughout the entire connection (i.e. you should probably be using port 465 in that screenshot)?

Or does it mean that the connection should request an upgrade to TLS with STARTTLS?

Because if it means - "This connection should be secure the whole way through" then when you connect on port 587, then your client is going to be sending a TLS handshake that the server is not expecting to see.

But if it means - "We'll always upgrade the connection using STARTTLS" then if you use port 465 then the server is going to be expecting a TLS handshake that the client never sends.

I guess because I understand these terms, I'd prefer that this was two options labeled as:

"Connect to port securely"

and

"Always upgrade the connection with STARTTLS"

And it's not just Apple stuff that does this, Thunderbird has a different meaning for this, and I believe Outlook does. They all do. Instead of trying to dumb it down... explain it out so that people that know what this stuff is actually know what these options are saying.



Now for what it's worth, none of these ports are actually standard. I'm assuming you don't have port 587 listed as a tls_on_connect_ports in your exim.conf. I'm assuming you do have port 465 listed as a tls_on_connect_ports.

Further from this I read an article... don't know if there is any truth to it, and I don't remember where I read it. It said that technically port 465 was not actually standard to SMTP... it's just that the masses chose this port for TLS on connect for SMTP and it's mostly stuck. The same is true of port 993 for TLS on connect IMAP and port 995 for TLS on connect for POP3. TECHNICALLY speaking, all of these clients SHOULD be using SMTP on the standard port 25 (although port 587 for message submission is a standard port characterization I believe) with STARTTLS. They should be using IMAP on port 143 with STARTTLS. And they should be using POP3 on port 110 with STLS. But years and years of ingrained port 465 is secure SMTP, port 993 is secure IMAP, and port 995 is secure POP3... it's just really hard to change people's minds on this.

But if this is true... then really we as server administrators should block off ports 465, 993, and 995 and force the masses into using the standard ports with STARTTLS and STLS and that would remove all of this confusion. But yea... can you imagine the %&*! storm if that happened?

 

[cPanelLauren]

465 was never intended to be used for SMTP over SSL, though it did become official with RFC 8314 - (also 993 and 995 were updated as well to be used with POP3S and IMAPS). Port 465 is no more or less secure than using 587 w/STARTTLS. Encryption for 587 is started by STARTTLS and doesn't require tls_on_connect to be set in the exim configuration, this is also discussed in RFC 8314 Section-3.3 and the requirements for MUA's are laid out in Section-5

However, to maximize the use of encryption for
submission, it is desirable to support both mechanisms for Message
Submission over TLS for a transition period of several years. As a
result, clients and servers SHOULD implement both STARTTLS on
port 587 and Implicit TLS on port 465 for this transition period.
Note that there is no significant difference between the security
properties of STARTTLS on port 587 and Implicit TLS on port 465 if
the implementations are correct and if both the client and the server
are configured to require successful negotiation of TLS prior to
Message Submission.

This notes clients and servers, and that's primarily the point I'm trying to make. The apple iOS mail client handles STARTTLS....it just doesn't tell you it's doing it. This is a result of a long-running trend apple has of just doing it for you assuming you don't know the difference, where the average person receiving their email on their phone truly doesn't.
Admittedly, this is confusing if you're looking for something specific like setting up STARTTLS specifically.

I'm not sure where the idea that tls_on_connect needed to be set for 587 - tls_on_connect would (I believe) imply it is implicit and STARTTLS is explicit SSL mode - both methods handle which protocol is used during the negotiation. There's a pretty good discussion on this here: Why is STARTTLS preferred over tls_on_connect_ports?

STARTTLS is advertised in the EHLO for the server on connect, and no user will be able to actually authenticate to the server with plain-text due to the fact I also have "Require clients to connect with SSL or issue the STARTTLS command before they are allowed to authenticate with the server." so no connection to the server can even be made without a secure connection. This is the default on all cPanel servers as well.

I can confirm my connection to the server is secure by looking at the "P=" line in the exim log followed by the TLS cipher user:

P=esmtpsa L- X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256

RFC 3848 - ESMTP and LMTP Transmission Types Registration

ESMTP and LMTP Transmission Types Registration (RFC 3848)

tools.ietf.org

ESMTPSA is defined in RFC3848 as follows

o The new keyword "ESMTPA" indicates the use of ESMTP when the SMTP
AUTH [3] extension is also used and authentication is successfully
achieved.

o The new keyword "ESMTPS" indicates the use of ESMTP when STARTTLS
[1] is also successfully negotiated to provide a strong transport
encryption layer.

o The new keyword "ESMTPSA" indicates the use of ESMTP when both
STARTTLS and SMTP AUTH are successfully negotiated (the
combination of ESMTPS and ESMTPA).

Most importanly the definition of ESMTPSA.

Further from this I read an article... don't know if there is any truth to it, and I don't remember where I read it. It said that technically port 465 was not actually standard to SMTP... it's just that the masses chose this port for TLS on connect for SMTP and it's mostly stuck. The same is true of port 993 for TLS on connect IMAP and port 995 for TLS on connect for POP3. TECHNICALLY speaking, all of these clients SHOULD be using SMTP on the standard port 25 (although port 587 for message submission is a standard port characterization I believe) with STARTTLS. They should be using IMAP on port 143 with STARTTLS. And they should be using POP3 on port 110 with STLS. But years and years of ingrained port 465 is secure SMTP, port 993 is secure IMAP, and port 995 is secure POP3... it's just really hard to change people's minds on this.

This is actually 100% true. It's noted in the RFC (at least for 465) and the story behind 465's assignment changes is a bit funny:

Historically, port 465 was briefly registered as the "smtps" port.
This registration made no sense, as the SMTP transport MX
infrastructure has no way to specify a port, so port 25 is always
used. As a result, the registration was revoked and was subsequently
reassigned to a different service. In hindsight, the "smtps"
registration should have been renamed or reserved rather than
revoked. Unfortunately, some widely deployed mail software
interpreted "smtps" as "submissions" [RFC6409] and used that port for
email submission by default when an end user requested security
during account setup. If a new port is assigned for the submissions
service, either (a) email software will continue with unregistered
use of port 465 (leaving the port registry inaccurate relative to
de facto practice and wasting a well-known port) or (b) confusion
between the de facto and registered ports will cause harmful
interoperability problems that will deter the use of TLS for Message
Submission. The authors of this document believe that both of these
outcomes are less desirable than a "wart" in the registry documenting
real-world usage of a port for two purposes. Although STARTTLS on
port 587 has been deployed, it has not replaced the deployed use of
Implicit TLS submission on port 465.

But at this point, I think we may have strayed pretty far from the OP's concerns and for that, I do apologize.

 

[sparek-3]

But at this point, I think we may have strayed pretty far from the OP's concerns and for that, I do apologize.

I agree here and I'm somewhat reluctant to ask this for fear that it's venturing the topic off course. But perhaps this will tie all this up in a pretty bow for me.

On the second screenshot you have - IMG_8343.jpg - you've got Use SSL checked and using port 587. If you change the port to 465 and leave Use SSL checked, does the connection still work? If you change the port to 465 and uncheck Use SSL, does the connection still work?

I'm having trouble remembering specifics - perhaps it was more IMAP/POP3 than SMTP - but it seems we have had customers before that they were using the "implicit" TLS port for those services AND checking the Use SSL box and it wasn't connecting. Changing the port back to the service's standard (non-implicit) port seemed to resolve the issue. This was telling me that the Use SSL box (I can't be certain that the incidents I am remembering actually refer to an iPhone or Apple product... it might've been another client and "Use SSL" might've been a different term) is for STARTTLS and not for specifying implicit TLS.

If you start an openssl client on a non implicit TLS ports with starttls:

openssl s_client -connect %server%:587 -starttls smtp

this works as expected. But switch out 587 for 465 (or whatever implicit TLS port you may be using):

openssl s_client -connect %server%:465 -starttls smtp

this does nothing.

That's why it would just be beneficial to me (maybe only me) if they actually spelled this out... "Use SSL means it's going to use a STARTTLS command to upgrade the connection"

 

[Erik Knepfler]

I'm very familiar with which ports Exim is running, which security protocols work and don't work on various ports, and the key thing here is that none of them work - no combination of security settings and ports, right or wrong, none of them work.

cPanelLauren - the problem does NOT occur on Blue Mail Android, Outlook windows desktop, Thunderbird windows desktop. The problem ONLY occurs on Blue Mail Desktop and iPhone. The ones that work tell me that the server is basically OK, more or less, and that the ports/settings I'm using are correct. It's only these two applications, Mail on iPhone and Blue Mail on Desktop, which fail.

The IP address I masked is just my client IP from which I am testing these connections.

Yeah this thread went off on a bit of a tangent but my short response to some of that is that I don't see STARTTLS as any kind of solution. I've never used that before, though I did try it at one point, and it made no difference. I can't see how adding another security protocol to the mix is going to change the fundamental problem that some apps can authenticate an account over SMTP while others cannot, and have their connections suddenly dropped.

This also seems to eliminate the firewall as a possibility - because I'm using APF which is just an interface to IPTABLES in which I have whitelisted my client IP address, to no avail.

However, you got me thinking that maybe since the Firewall is loading some remote rules, even though I'm whitelisted something might be happening.
So, I shut down the firewall, and tried Blue Mail again, and this time it worked.
I reactivated the firewall, and I could still send messages. So I thought, maybe it's only during account setup, initial authentication?
So I removed the account and re-added it, and everything just worked, automatically, no ports/security manual settings required, it just worked (using autodiscover or howeever bluemail figures that out.)

So, something seems to have changed in the last few days which I cannot account for. The only thing I did differently was fully shutting down APF and starting it up again, instead of using the apf -r restart option which may work differently.

So, this COULD have been just a firewall malfunction of some kind that required a complete restart. I will update this thread if anything changes, but for now, it seems that either it resolved itself through unknown means, or completely shutting down the firewall and fully starting it back up again has somehow solved it, which would be very mysterious if true.

Update: My Blue Mail is still working ok at the moment, I removed the account from it and re-added it a few times now, but my client is saying the iPhone is still having trouble...