Server never sends the full 36,000 emails

[Mrs Bee]

I have used the same CGI script on my server for sending email to my customers for around ten years, with no problems in the first seven years (previous host). I have quite a large mailing list of 36,000 email addresses, however over the past three years I have experienced a problem whereby the script (or the server, I don't know!) just kind of gives up part-way through the task and never gets to the end of the 36,000 email addresses. I know this because A) I don't get the automated 'Congratulations your mailing was successful' email at the end and B) I have 26 test addresses in the list (a@mydomain, b@mydomain, c@mydomain, etc) and so whichever of the 26 test emails I don't receive, I know roughly how far it got through the list (as the list is in alphabetical order).

For some reason the last three or four months have been rather good in that all 36,000 emails got fired off without a problem - it took maybe 20 minutes for the server to pass them all through the Mail Queue Manager, but it did get through all 36,000 eventually and I received the automated 'Congratulations' email from the CGI script, telling me that all of the email addresses got processed. However, I tried last week and only received five test emails (up to e@mydomain) and I tried again today and only received two test emails (a@mydomain and b@mydomain).

I have spoken with my hosting company today who said that there is "no server side issue" and then he restarted the firewall and asked me to try again. It didn't work. I don't understand why they say it's not a "server side issue" because I haven't changed the script in ten years and it used to work perfectly with my previous hosting company who I was with for seven years.

My Linux semi-dedicated server has WHM 11.38.2 (build 7). I don't know if you need to know this but in Server Status > Server Information it states that the server has four processors which are all Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, Speed 2394.063 MHz, Cache 12288 KB.

Is it a CGI script timeout? Or the processor or memory on the server not powerful enough? Is there something I need to edit in Server Configuration > Tweak Settings > Mail? Some of the settings I already have in there are as follows:-

Email delivery retry time ....... 60 minutes
Track email origin via X-Source email headers ....... Off
Max hourly emails per domain ....... Unlimited
The percentage of email messages (above the account’s hourly maximum) to queue and retry for delivery. (minimum: 100; maximum: 10,000) ....... Unlimited
Count mailman deliveries towards a domain’s Max hourly emails ....... On
Maximum percentage of failed or deferred messages a domain may send per hour ....... Unlimited

As you will know, there are more settings on that page but I didn't want to list them all unnecessarily so please just ask if you need to know anything further. I have attached the CGI script to this message in TXT format.

I would appreciate any help that anybody can offer. Thank you.

 

[cPanelMichael]

Hello

You mentioned contacting your web hosting provider. Do you have root access to this system? Or, are you providing us with information that your web hosting provider sent to you? If you have root access, try checking the following log file when sending out the emails:

/var/log/exim_mainlog

It should output the status of each message as the delivery attempt is made.

Thank you.

 

[Mrs Bee]

Hi Michael, thank you for your reply. Sorry yes, I do have root access (in fact I was just checking if I could amend my profile information to 'Root Administrator' - I only put 'Website Owner' when I registered because it said 'Please select the title that describes your cPanel/Enkompass Experience Level" and I really don't feel that 'experienced' :D)

Anyhow, my setup is that I have root access to a semi-dedicated server which I normally only access via Web Host Manager in a browser, however I have used Putty a little in the past.

I have now connected to the server using SSH and I am currently looking through the logs. I can see lots of lines saying why emails have failed to be delivered (which I expected) but I was hoping to find out the reason why the CGI script stopped running. It seems like the server processes, for example, 4,000 email addresses from my text file and then refuses to do any more. So, for that reason I wouldn't get any delivery reports for the remaining 32,000 email addresses if you see what I mean, because the server had not even tried to send them. Sorry if that's garbled.

 

[Mrs Bee]

Would anybody have any further information that might help me please?

 

[cPanelMichael]

I have now connected to the server using SSH and I am currently looking through the logs. I can see lots of lines saying why emails have failed to be delivered (which I expected) but I was hoping to find out the reason why the CGI script stopped running. It seems like the server processes, for example, 4,000 email addresses from my text file and then refuses to do any more. So, for that reason I wouldn't get any delivery reports for the remaining 32,000 email addresses if you see what I mean, because the server had not even tried to send them. Sorry if that's garbled.

It's possible the script is still attempting to send the emails, and Exim is rejecting the delivery attempts. What types of error messages are you seeing in the Exim logs?

Thank you.

 

[Mrs Bee]

Thank you for your reply Michael. I'm not sure what I am looking for in the results which are displayed in Putty - I can't really paste it all in here as my customers' email addresses are in there. There are lots of delivery messages (failed and successful).

I did notice in the log that around the time that emails seem to stop sending, there are quite a few lines which seem to refer to a Google email time-out:-

2013-09-17 10:47:36 1VLrpb-0001vp-Af gmail-smtp-in.l.google.com [2a00:1450:400c:c05::1a] Connection timed out
2013-09-17 10:47:36 1VLrpb-0001w6-Ew gmail-smtp-in.l.google.com [2a00:1450:400c:c05::1a] Connection timed out
2013-09-17 10:47:36 1VLrpb-0001wC-Gn gmail-smtp-in.l.google.com [2a00:1450:400c:c05::1a] Connection timed out
2013-09-17 10:47:36 1VLrpb-0001wg-My gmail-smtp-in.l.google.com [2a00:1450:400c:c05::1a] Connection timed out

There are several subsequent sets of the above output which follow afterwards, too (mixed in with further delivery/non-delivery reports).

I also noticed the following line and I thought maybe that would help me to find the last email address to successfully get processed, so I searched backwards (from the bottom of the log) and found that the last instance of the following code was timed at 10:45:23:-

2013-09-17 10:45:23 cwd=/home/mywebsite/public_html/cgi-bin 2 args: /usr/sbin/sendmail -t

I wondered if the Google connection time-out was causing a blockage on my mail queue, so I thought about limiting the number of emails my server sends per hour (in Tweak > Mail) but after reading another thread on here I understand that if I limit my 'per hour' to, say, 5,000 then the remaining 31,000 emails on my list would get bounced back, ie the server wouldn't even attempt to send them an hour later. Is that correct or have I misunderstood it?

 

[simonas]

Hello,

I had similar problem.

Please pay attention to IP that google server resolves - 2a00:1450:400c:c05::1a
That's IPv6. It is not supported by cpanel yet, and i doubt hosting supports it.

Go to WHM, exim configuration, advanced configuration, scroll down to add your own directive.
Type:
disable_ipv6 and make it true.
Scroll down and click save.

That should solve your problem.

 

[Mrs Bee]

Hi Simonas, thank you for your reply.

I have now made the change that you suggested (although I didn't have a clue where to go in Advanced Configuration so I sought help from this thread) and it seems to have worked as I got the following confirmation message:-

Exim MTA...
Waiting for exim to restart....finished.
exim (/usr/sbin/exim -bd -q60m) running as mailnull with PID 29721 (pidfile check method)
exim started ok
...Done

I will try my mailing lists again this week and update this thread accordingly. Thank you both for your help.

 

[cPanelMichael]

It could also be an issue where Google is blocking connections from your server due to the number of emails that you are sending to it in a short time frame. They may detect this as SPAM activity and thus prevent email from your server temporarily. If disabling IPV6 does not help, you may want to consult with Google to see why they are blocking connection attempts from your server.

Thank you.

 

[simonas]

Michael,

At first i thought it too... That google was blocking me. But it was the case of Ipv6 being resolved instead proper ipv4.

I have task list of what i do to a new Cpanel servers, and disabling exim ipv6 is one of the task.

Could you maybe ask dev team why they set use ipv6 to true as default option?

Thanks!

 

[cPanelMichael]

It's not necessarily that it's enabled on purpose. Rather, it's the default setting with Exim and we do not explicitly disable it for servers that intend to use IPV6.

Thank you.

 

[Mrs Bee]

I just wanted to update this thread with my results as promised.

A day or two after my last comment I sent a mailshot to the 36,000 or so email addresses and, if I recall correctly, they all went through with no problems. A day or so after that sent a smaller one of around 3,000 email addresses and that was fine too.

However, I sent a mailshot earlier today which didn't complete. There were 36,230 email addresses on the list and it managed to process around 26,200 of those email addresses before it stopped. It took 25 minutes to process those 26,200 email addresses.

I sent a different mailing this morning to 2,885 email addresses (before the one mentioned above) and that one completed fully with no problems.

I am going to re-send the remaining 10,030 emails from the first mailshot above in a little while. Hopefully they will all go through.

Michael - I have checked with Google and I am not blacklisted, which is good.

 

[Mrs Bee]

I have just found two error messages which I have not seen before (found in WHM > Email > Mail Delivery Reports) which seem to have been generated at the same time as my most recent mailshot (this afternoon) failing part-way through.

The first error is as follows and the recipient email address is shown as the newsletter administrator:-

check_mail_permissions failed to complete or set a status

The second error is as follows and doesn't have a recipient email address listed:-

failed to expand condition "${perl{should_archive_outgoing_message}}" for archive_outgoing_email router: syntax error in Exim::expand_string argument: failed to open /etc/userdomains for linear search: No such file or directory at /etc/exim

Would anybody know what the above two error messages might mean, please and if/how they might relate to the problem that I am having with the mailshot failing part-way through?

 

[cPanelMichael]

check_mail_permissions failed to complete or set a status

I would not be too concerned with this "check_mail_permissions" message. It's primarily just going to complete tasks like determining the mail sender, and failed to do so in this instance.

failed to expand condition "${perl{should_archive_outgoing_message}}" for archive_outgoing_email router: syntax error in Exim::expand_string argument: failed to open /etc/userdomains for linear search: No such file or directory at /etc/exim

Check to ensure the /etc/userdomains file exists, has 0640 permissions, and "root:mail" ownership.

Thank you.