Weird Non-Delivery Receipt from external domain

[tmurdock]

I have encountered a very strange issue that I am stumped on. I have a user on my domain that sent an email to another user on the same domain. The email was sent successfully and I can see in the exim logs that the email was delivered using the virtual_user which I assume is for internal emails.

However, about five hours later the original sender received a non delivery report from Microsoft stating that her email was unable to be delivered. The strange part is the address listed in the NDR is on an external O365 domain that I have never seen before. I've searched the logs for any matches and am unable to find it. I've also checked both users computers and can not find any sort of forwarding rule.

Also in the Microsoft NDR it shows the message hops which starts from the sender's IP address to our mail server IP address using SMTP, and then a second hop with both the from and destination being my mail server address using LMTP which I would assume is delivering the mail to the other internal user.

The next hop shows the hop going from 127.0.1.1 to a Outlook SMTP server based in Europe. This relay time takes almost 7 hours before it ultimately fails and sends back the NDR.

I am at a loss as to how an email that is being sent and received from internal users would even be received by any external email server, let alone one I have never seen before and can find no information on. The NDR also has a copy of the original email that does not contain the email address of the Office365 user at all.

If anyone has any ideas I would greatly appreciate them!

 

[cPRex]

Hey there! Are there any forwarders or filters in place that could explain this behavior? If you search for the original mail ID in the /var/log/exim_mainlog file, I would expect to see the full transaction and it would show if the message was handled in a way that would have triggered the message to be sent elsewhere.

Is it also possible this is just some unrelated spam of some sort or is it too specific for that?

 

[tmurdock]

Thanks for the reply! I did look in the forwarders, all forwarders are pointing to local addresses so I don't think that is it. I did look in filters and I only have two in place, both set to discard messages that match that filter.

I did grep the message ID and it only shows the delivery to the local address. Normally I would chalk it up to spammers spoofing our email addresses and getting bogus NDR, but this NDR has a copy of the original email that actually made it to it's intended destination. That's what is so strange about it. The failed recipient on the Office 365 NDR is just so random and the third hop on the NDR confuses me since the delivery should have stopped at the internal recipient. Here are the hops with my info redacted:

HOP TIME (UTC) FROM TO WITH RELAY TIME 1​ 6/8/2023 5:51:41 PM 0**-0**-1**-1**.biz.spectrum.com host.********.com esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <*****@********.com>) 2 sec 2​ 6/8/2023 5:51:41 PM host.********.com host.********.com LMTP * 3​ 6/9/2023 12:39:28 AM [127.0.1.1] DB8EUR05FT028.mail.protection.outlook.com Microsoft SMTP Server 6 hr, 47 min, 47 sec 4​ 6/9/2023 12:39:29 AM DB8EUR05FT028.eop-eur05.prod.protection.outlook.com DB8PR06CA0057.outlook.office365.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) 1 sec 5​ 6/9/2023 12:39:29 AM DB8PR06CA0057.eurprd06.prod.outlook.com GV2PR08MB9951.eurprd08.prod.outlook.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) *

I am not sure on the 3rd hop if 127.0.1.1 is supposed to represent my mail server or Microsoft's, but there is no O365 mail listed anywhere in the exim_mainlog nor in the mail client on the sending or receiving end. I am at a loss!

 

[cPRex]

I don't have a good server-side explanation for that. Could there have been something on the user's side (in a mail client like Outlook or Thunderbird maybe?) that would explain it?

 

[tmurdock]

At this point that's what I am leaning to as well, I had looked briefly but didn't see anything. I am not seeing anything on the server either so I will focus on the end machines. Thank you for the help!

 

[cPRex]

That's my only other idea - if you're not seeing things in the server logs, I'd start and look to outside sources.

 

[mtindor]

Thanks for the reply! I did look in the forwarders, all forwarders are pointing to local addresses so I don't think that is it. I did look in filters and I only have two in place, both set to discard messages that match that filter.

I did grep the message ID and it only shows the delivery to the local address. Normally I would chalk it up to spammers spoofing our email addresses and getting bogus NDR, but this NDR has a copy of the original email that actually made it to it's intended destination. That's what is so strange about it. The failed recipient on the Office 365 NDR is just so random and the third hop on the NDR confuses me since the delivery should have stopped at the internal recipient. Here are the hops with my info redacted:

HOP TIME (UTC) FROM TO WITH RELAY TIME 1​ 6/8/2023 5:51:41 PM 0**-0**-1**-1**.biz.spectrum.com host.********.com esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from <*****@********.com>) 2 sec 2​ 6/8/2023 5:51:41 PM host.********.com host.********.com LMTP * 3​ 6/9/2023 12:39:28 AM [127.0.1.1] DB8EUR05FT028.mail.protection.outlook.com Microsoft SMTP Server 6 hr, 47 min, 47 sec 4​ 6/9/2023 12:39:29 AM DB8EUR05FT028.eop-eur05.prod.protection.outlook.com DB8PR06CA0057.outlook.office365.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) 1 sec 5​ 6/9/2023 12:39:29 AM DB8PR06CA0057.eurprd06.prod.outlook.com GV2PR08MB9951.eurprd08.prod.outlook.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) *

I am not sure on the 3rd hop if 127.0.1.1 is supposed to represent my mail server or Microsoft's, but there is no O365 mail listed anywhere in the exim_mainlog nor in the mail client on the sending or receiving end. I am at a loss!

If you grep DB8EUR05FT028 /var/log/exim_mainlog does it reveal anything? If it does, look for the msgid of that SMTP transaction and grep it.

It really does sound like you've got a forwarder or filter in place causing the message to be delivered locally and forwarded outbound.

 

[tmurdock]

If you grep DB8EUR05FT028 /var/log/exim_mainlog does it reveal anything? If it does, look for the msgid of that SMTP transaction and grep it.

It really does sound like you've got a forwarder or filter in place causing the message to be delivered locally and forwarded outbound.

Hi mtindor,

I just tried that grep and it returned no results. From what I can tell there is nothing on the server that would have forwarded it, especially to the email address that rejected it as the address does not exist on the destination O365 server. The only thing I haven't been able to check is the user also uses the Outlook app on her phone and in the past I have seen that app rout messages through Microsoft servers even if they aren't being sent to a Microsoft account.

I appreciate the suggestion!

 

[mtindor]

Hi mtindor,

I just tried that grep and it returned no results. From what I can tell there is nothing on the server that would have forwarded it, especially to the email address that rejected it as the address does not exist on the destination O365 server. The only thing I haven't been able to check is the user also uses the Outlook app on her phone and in the past I have seen that app rout messages through Microsoft servers even if they aren't being sent to a Microsoft account.

I appreciate the suggestion!

It could be that by the time you checked exim_mainlog, it may have rotated.

Try:

zgrep DB8EUR05FT028 /var/log/exim_mainlog*