Duplicate, repeat, looping emails.

TogaDave

Well-Known Member
Apr 13, 2003
134
0
166
I'm wondering if anyone else is experiencing this weird issue.

Starting a few weeks ago, myself and some of my customers have begun receiving multiple copies of emails randomly.

For example:

I'll receive an email at 4:00pm. Then hours later I'll receive it again, and then later again.

This does not happen with every email, and it does not happen with every account on the server, but it does happen every day.

I'm running the latest "current" release of WHM and cPanel, and I've tried repairing / restting mailboxes and databases, I've restarted exim and Apache. I've also checked to make sure there are no accidental duplicate forwarding or POP accounts.

Everything looks clean as far as I can tell.

Any ideas?
Thanks!
Dave
 

PWSowner

Well-Known Member
Nov 10, 2001
2,934
4
343
ON, Canada
One thing that would cause this, as I recently noticed, is if the mail program is downloading the message but getting disconnected before it has the chance to tell the server it has been downloaded. The server won't delete the message until it has been told to do so by the mail program.
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,448
31
473
Go on, have a guess
Indeed. POP3 will only delete email if it has received a successful QUIT command after all other processing has been done (it's doesn't actually delete anything after a DELE command, just flags it to be done at the end of the session. So, if you do have a flaky connection from either your PC or from the server, you will have problems.

Other issues include using "Leave mail on server" option in some email clients. This is a non-standard thing that can easily go wrong (and is not what the POP3 protocol was designed for), so if you use that, you could have issues there too.

One way to tell whether it's something like that, or something more ominous, is to check the Message-id header record of the duplicate emails. If they are the same, then it's most likely a POP3 problem. If they're different, then there's something wrong at one or the other end of the SMTP mail transfer.

One final thought, it might be a good idea to check the contents of the actual POP3 inbox file on disk to see if there are duplicates in there. If not, then it's almost definitely an email client/connection issue.
 

TogaDave

Well-Known Member
Apr 13, 2003
134
0
166
chirpy said:
One way to tell whether it's something like that, or something more ominous, is to check the Message-id header record of the duplicate emails. If they are the same, then it's most likely a POP3 problem.
Thanks for the tips guys!

This all makes sense now. The problem seems to only happen to those of us using "Leave mail on server for XX days" option, and the message ID's in the hidden headers are duplicate.

So apparently there's no way to make "leave mail on server" play nice with the box? :(
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,448
31
473
Go on, have a guess
The problem with the feature, is that there's no RFC specification for it, because POP3 was devised for retrieve and delete. This means that different email clients use different ways to track which messages they have retrieved (most do it by message-id, other do it in differently). It is usually reliable, but there are many ways for it to go wrong.

If you want to leave email on the server despite having already downloaded it, you might be better off looking at using IMAP instead of POP3.
 

jefferymac

Registered
Mar 19, 2005
3
0
151
Imap?

I am struggling with this same issue, but I'm using imap, not pop.

The only pattern I've found in the behavior is that the repeat emails always contain attachments or embedded images.
 

wh7702

Member
Mar 13, 2005
6
0
151
I am struggling with this same issue, but I'm using imap, not pop.

The only pattern I've found in the behavior is that the repeat emails always contain attachments or embedded images.

Did you ever figure out your problem?
 

inkhead

Registered
Dec 28, 2007
1
0
51
I'm having this very problem, but I'm on IMAP, it only started since about 3 weeks ago, around the time I think WHM upgraded itself or something, ever since then every freaking email has at least 3 copies, sometimes 10. It's something with the email server. Restarting it doesn't help either, and I'm at my wits end, any suggestions?

Why would all my incoming messages exact duplicates (even when I send to myself from another address) come in 3-10 times?

How can I fix this asap? I've done all the mail tricks, and tried everything.
 

acenetryan

Well-Known Member
PartnerNOC
Aug 21, 2005
197
1
168
We're having this issue as well. We thought we had fixed it, but it showed up again today. Here was my take on it. I was seeing:

SMTP connection lost after final dot

in the exim_mainlog. If our server is not telling the sending server it has received the message, the sending mail server may be delaying the message and retrying it at it's internally set intervals. You receive the message, but the sending server doesn't know you received it so it continues to send the message on subsequent queue runs.

At the time, I was able to send mail from some domains without a probelm. This did not appear to be an issue with all sending domains, just some (notably Yahoo and Gmail). I noticed that when sending from Yahoo servers that messages did not always arrive and there were frequent timeouts.

At the time, we reseteximtodefaults and the problem went away. It's back again today and I'm again seeing the "SMTP connection lost after final dot" within exim_mainlog.
 

acenetryan

Well-Known Member
PartnerNOC
Aug 21, 2005
197
1
168
I think I fixed it.

I noticed in the Yahoo email box for one of our test boxes, we received:

MAILER-DAEMON said:
Hi. This is the qmail-send program at yahoo.com.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

<[email protected]>:
Connected to IPADDRESS but connection died. Possible duplicate! (#4.4.2)
I'm not going to try again; this message has been in the queue too long.
I went digging 'round the net and didn't turn up much, but I did find a mention that this could be related to the filters being overworked.

I tested emails from Yahoo and Hotmail and they took various amounts of times to deliver. Some would show up immediately, some took a while.

I went into the Exim Configuration editor and disabled all of the following:

"SpamAssassinTM: Reject mail with a spam score is greater then 20.0 at SMTP time."
"Ratelimit incoming SMTP connections that violate RFCs (this is usually spammers and broken mtas.)"
"Reject messages with potentially dangerous attachments."
"SpamAssassinTM: Reject mail with a spam score is greater then 20.0 with a bounce message (not at SMTP time)."

As soon as I disabled all of these, messages from Yahoo and Hotmail went through immediately so I'm expecting that the duplicate emails will cease as well. Only time will tell. Waiting client update.
 

acenetryan

Well-Known Member
PartnerNOC
Aug 21, 2005
197
1
168
Sadly, this did not appear to have been resolved. We are still getting intermittent reports of duplicate emails being received.

What is also odd is that emails are being delayed from some domains. For example, I sent an email from Hotmail to an account on this server. Nothing arrived, no line in the exim_mainlog. I then sent a message from this server to the Hotmail account. It arrived. When I replied, the message I had originally tried to send arrived in the inbox, but the one I just sent did not show up. Again when I sent another, the previous message arrived, but the one I just sent didn't.

Very weird. Will probably have to open a ticket on this one.
 

dgbaker

Well-Known Member
PartnerNOC
Sep 20, 2002
2,557
9
343
Toronto, Ontario Canada
cPanel Access Level
DataCenter Provider
Anyone every get a good handle on how to correct this? We have this one client only, so I am suspicious of it being a server config issue as that would imply more clients would be having this issue.
 

acenetryan

Well-Known Member
PartnerNOC
Aug 21, 2005
197
1
168
We had two distinct cases of this.

We had one occurance where several clients were receiving duplicate emails. We were using an old SURBL mod in our exim and after removing this, it seemed to correct the problem. Though, we also disabled several other SA filters enabled in WHM. In the end, we only have the spamhaus mod running and the problem hasn't resurfaced since.

In another case, only a single user on the server was having this issue and it was sending multiple copies of email he had sent. In this circumstance, the user was sending through Outlook and the message was never leaving his outbox despite it being successfully sent. He was getting the error:

"Sending' reported error (0x8004210B) : 'The operation timed out waiting for a response from the sending (SMTP) server. If you continue to receive this message, contact your server administrator or Internet service provider (ISP).' "

This one we chalked up to an Outlook bug:

http://support.microsoft.com/kb/315049

After disabling SPA and any SSL settings this user had for their POP3 connection, their issue was resolved. The user only encountered these issues in Outlook, Thunderbird worked fine.