Mail stuck in outbox or sending multiple copies

[scottrichardson]

Hey all.

hoping you could assist me.

This morning I rebooted my WHM server to apply the latest update changes and since then email has been behaving strangely.

I’m receiving calls from clients advising that they are having issues connecting to the server, or that emails are being stuck in outbox. The ones stuck in outbox ARE delivering even though appearing to remain in outbox. In many instances several copies of the emails are being delivered.

Others are reporting that emails are sending very slowly.

finally, I jumped onto a client’s machine and tried to ping our server’s server address but got no response but when using the browser to visit the webmail interface it worked fine.

has there been anything in the latest update that could break email in this way?

what else could be causing these issues?

up until today the server has run flawlessly with minimal load. It’s on AWS and this server is used only for hosting cPanel accounts with email mailboxes. I have a separate server that’s used for hosting cPanel account websites (connected to RDS for database).

happy to provide further info and logs if requested. Not sure where to look just yet.

many thanks in advance.

Scott

 

[scottrichardson]

Additional information:

It appears that clients using Microsoft Outlook are the ones who seem to be sending multiple copies of an email that's stuck in the outbox. I am assuming Outlook is repeatedly trying to send the email over and over again - and it IS sending, but NOT syncing the outbox/sent status?

Further, I had a client who is having issues attempt to log into webmail and send an email. He could log in, and send me an email (my email is on the same server, too), but received a small red error in the bottom right of RoundCube webmail interface:

SMTP Error: [-1]

To confirm, when I use Terminal to ping my email server host name, it is reporting:

Request timeout for icmp_seq 0

But I am still able to visit the WHM interface no problem via a browser.

 

[scottrichardson]

Update...

Could it be too do with max. emails per hour per domain?

ALL of my clients connect to the host name for their outgoing smtp email server, which also happens to have its own cPanel account. Could it be that cumulatively, all of their emails exceeded the max. per hour limit, which caused the bizarre behaviour? Even though they have their own mailboxes on their own cPanel accounts, they all use the one outgoing mail server which is the host name.

 

[microvax]

What versions of CPanel and Outlook are you using?
I also received reports of the same behaviour yesterday early in the morning.
The problem dissapeared during the day but appeared again today.

 

[cPRex]

Hey there! I'm wondering if this is related to a problem with the server time that we found in case CPANEL-36312. Can you try restarting Exim manually to see if that gets things working well on the system?

 

[Bob Busch]

What versions of CPanel and Outlook are you using?
I also received reports of the same behaviour yesterday early in the morning.
The problem dissapeared during the day but appeared again today.

Update...

Could it be too do with max. emails per hour per domain?

ALL of my clients connect to the host name for their outgoing smtp email server, which also happens to have its own cPanel account. Could it be that cumulatively, all of their emails exceeded the max. per hour limit, which caused the bizarre behaviour? Even though they have their own mailboxes on their own cPanel accounts, they all use the one outgoing mail server which is the host name.

This sounds like the same problem as I am having. A client has started reporting issues with webmail daily. It started a few days ago and seems to happen around the same time of a cpanel update.

When the problem starts, users have trouble sending email. After trying to send a message, the webmail hangs and eventually comes back with an SMTP Error.

In order to resolve the issue, I attempted a system reboot which worked. Around the same time the next day the issue happened again, I rebooted again and it was resolved.

I noticed that the cpanel update was happening mid day in Pacific time around the same time that the error started. I changed the cron job to have the updates take place in the middle of the night. The next day, the issue was reported in the morning.

This morning I checked and saw the issue was happening again. After doing some research I tried updating the settings in the exim configuration manager which also restarts the exim config manager. After I did that the problem went away. So it seems that restarting exim makes the issue go away.

I didn't make any configuration changes in the last few days which is why I suspected it had to do with a cpanel update.

I tried to analyze the logs to see if I could see any error logging but I did not see any.

When the problem is happening I have also heard of some users getting duplicate email messages for email sent internally within the same domain.

I confirmed that by restarting exim the problem is (temporarily) resolved.

My server OS and Cpanel version: CENTOS 7.9 hyper-v [vps]. v94.0.4

 

[cPRex]

@Bob Busch - that does sound a lot like CPANEL-36312. More details on that can be found here:

Exim v1.94-1 causes duplicate emails to be delivered due to server time change

 

[microvax]

Today we received again reports of this issue after several days of no problems, and as mentioned by others resetting the EXIM service seems to solve the problem.
BTW I also noticed several error messages in EXIM log files with this text
"failed: Connection reset by peer"

 

[cPRex]

@microvax - that would be case CPANEL-36312. We're working on it and planned to have that fixed both in 94 and 96.

 

[Kenric Ashe]

This seems like a huge issue considering how many cPanel users there are worldwide. It was reported March 20th. Why is it taking so long for the Exim devs to fix it? And while we're waiting, should cPanel release an emergency update with an Exim rollback?

 

[cPRex]

@Kenric Ashe - it's important to note this isn't affecting all cPanel users. If there was a massive Exim issue affecting all cPanel machines, that would have caused incredibly widespread issues and there would have been an emergency fix in place. The issue only seems to happen when there is a time adjustment on the server by the chrony daemon, which seems to confuse Exim.

We hope to have the fixes out in 94 and 96 soon.

 

[Kenric Ashe]

systemctl disable chronyd

Might that be a better solution than periodically restarting Exim?

Or is the use of chrony recommended?

 

[Bob Busch]

@Bob Busch - that does sound a lot like CPANEL-36312. More details on that can be found here:

Exim v1.94-1 causes duplicate emails to be delivered due to server time change

Thanks for the suggestions and research.

I did look into this and tried what was suggested.. "grep -i "system clock" /var/log/messages"
There was nothing in the log.

I also noticed that the duplicate emails seem to come when messages are sent using the Windows MS Outlook Client. Other clients (except of course webmail) don't seem to have the issue.

The last time I had the issue I rebooted my server. I haven't had a complaint lately.

Current version is still: v94.0.4

 

[Bob Busch]

The issue has recurred and I have some more information. It seems like if exim is slow to respond, the problem occurs.. Not sure what is causing exim to be slow to respond.

 

[cPRex]

@Bob Busch - if you restart Exim, does the issue resolve itself for a period of time?

 

[Bob Busch]

Yes.. restarting exit fixes the issue for a period of time. Not sure what causes it to recur. I am resisting restarting exim right now so we can troubleshoot. The problem continues.

 

[cPRex]

Even though you may not see an entry for chrony in the logs, it still seems like it's related to that case. If you'd like to submit a ticket to our team we'd be happy to check things on our end.

 

[Bob Busch]

Even though you may not see an entry for chrony in the logs, it still seems like it's related to that case. If you'd like to submit a ticket to our team we'd be happy to check things on our end.

I created a new ticket: #94318257

 

[cPRex]

Thanks for that - I'm currently following along with that ticket on my end now.

 

[JFWebDesign]

Here we are, the end of April and STILL no fix? Duplicate emails, emails sitting in queues, Outlook time-outs. To expect us to reboot Exim every few hours to make sure our clients don't have email problems is NOT a fix nor is it a bandaid.

What is the update on this?

Also, there was never a response to the user who said:
Apr 16, 2021

systemctl disable chronyd

Might that be a better solution than periodically restarting Exim?

Or is the use of chrony recommended?