Fun With Yahoo Mail (How I Finally Got The Mail Out)

[esse]

PLEASE DON'T DO THIS UNTIL THE EXPERTS CHIME IN

I manage a mailinglist with about one hundred thousand members. The list is double opt-in and has been maturing for 5 years. I've read enough to know I am not alone with the recent problems with yahoo deferring their email. We have about 18,000 yahoo subsribers. I've had 10,077 of them stuck in the queue all day and have after several hours of tweaking with settings and pushing the queue I was only able to send about 40 of them.

Finally by the end I was working from the shell using:

exim -qff -v

Yahoo keeps the connection live but stops talking to my server so the thing just hangs.

So this is what I finally ended up doing. This may not be a very good solution, but keep in mind I just need to get these emails out to our subsribers.

I tried different settings for triggering the queue and those didn't help. I finally set it to run once a day until I could work something out. This might be resource intensive but it works. Which means it might lead to a real solution from one of you guys that know your stuff.

So Yahoo just sits there and hangs and then eventually runs a few seconds and then stops again. However if you break into the session and start it again it picks right up. I did that for a short time manually and that was a bear.

No matter what it's going to reach a point and hang, sometimes for minutes on end. So what I did was put exim -qff in crontab to run every minute, and the mail is moving. The mail queue on my server is down to 8000 in the queue in about 45 minutes.

At least at this point it looks like the key to defeating these deferred emails is breaking the session stall and then starting up again.

I am going to push this as long as it works to get these emails out and then I'm putting eveything back to my servers default settings. I'd sure like to know if there's a way to do this without pegging the system. I have to restart Exim once an hour while I keep this up.

Thanks for any feedback, ideas, and solutions in advance.

Update: This process moved all the yahoo mail in the queue in about 6 hours.

As best as I can tell the starting of exim -qff every minute did the job. Just to play it safe I put restarted exim on the hour thoughout the process. Normally there would be email in this queue until it timed out. I tried one of the suggestions having WHM run the queue every 10 minutes earlier but that didn't work and I believe mainly due to the volume of emails. By the time I got things going I had emails that matured 20 hours.

Also worth mentioning, outside of web hosting, this server only hosts abuse and postmaster accounts for the domains we host, with one or two general support or contact accounts. So I have the luxury of taking a few chances. Other than bogging down the CPU and maybe losing some emails going to yahoo accounts I figured it was worth trying.

I too have been trying to work with Yahoo, but they are unresponsive. In the process of cleaning up their own act and the practices used to exploit thier services they're really slamming legiitmate emails from the mom and pop organizations who have built a healthy digital relationship with their customers over the years.

It is unfortunate that people who sign up for newletters and eventually tire from them siimply do not unsubscribe and therein lies the rub. Unless you can somehow get on Yahoo's feedback loop you don't know who they are so the more you mail them it seems the more you worsen your reputation as a sender. There doesn't seem to be anything you can do.

I think this fix will get me past what I plan on doing now. Which is remove the active status of all the Yahoo accounts we mail too and send a re-opt in email to reactivate their account.

[brianc]

I came across the same idea but I was using exim -Rff yahoo.com. I will try using -qff and see if that speeds things along.

I think Yahoo.com should be sued for their delaying antics, maybe through a class action suit. I watched one batch of messages that were sent to three yahoo addresses, one address generated a 451 error (temporarily deferred), the other a 421 error (delayed due to user complaints) and a 250 message. So one batch that is sending the exact message generates 3 different responses from yahoo.com's servers.

I am also trying to get my servers whitelisted and they are just not interested in helping you out. I feel bad for their mail users. If they only knew how Yahoo is interfering with their mail delivery.

Sorry to digress. Please update this thread if you get anywhere with Yahoo or if you come up with a better way of getting mail delivered to yahoo mail boxes. I am especially interested if there is anyway to improve exim's queue runs. I too see a lot of hang ups and sometimes I see a "timeout=5 min" message given as a reason for the hangup. Please let us know if there are any settings that can speed up the queue process.

[esse]

I took all the Yahoo addresses off the mailing list and resent the original opt-in message to all subscribers. The only way Yahoo is going to listen to anyone who sends volume email to their servers is if the recipients remove them from spam status.

Quite frankly if they're assigning subscriber emails to the spam folder. Who wants them as subscribers anyway?

Yahoo continues to bury its head in the sand.

I've kept the cron running and no longer restart exim. Depending on how much mail there is in the queue I will get a number of alerts that a process of sending from the mail queue crashed. The queue crashes seem to happen when delievery flow is not impaired or at least not interrupted by the time one minute elapses and the cron job breaks the stream. Regular incoming and outgoing email does not appear to impacted and it looks like the crashes just pick up where they left off.

The server seems to be working fine so I am not concerned and no one has chimed in telling me I am out of my mind. So I am going to keep on trucking until someone jumps in that knows better.

The key to defeating the yahoo delivery problem is breaking off the sessions when they stall. It is the only thing that seems to work.
As best I can tell if Yahoo returns a set number of deferred messages within a set number say 8 out of 10 the session stalls.

one address generated a 451 error (temporarily deferred), the other a 421 error (delayed due to user complaints) and a 250 message. So one batch that is sending the exact message generates 3 different responses from yahoo.com's servers.

I think the stall might also be more severe depending on which of the errors it hits and how many in a set number as well.

This problem is not unique to exim it's across the board. I was told there might be a postfix solution.

The most aggravating part of this experience is the process of upgrading an older system (which always leads to complaints) and installing domain keys that no other system seems to use. And I would also like to add I am not going to pay some third party certification fee to reach people who have requested they be contacted. That is simply Bull...

[Kent Brockman]

Hi, I manage two small lists, one with 4500 suscribers and other with 2000+. I had the same problem with Yahoo and the only thing that worked for me was setting up DomainKeys and SPF data in my dns, and voila!
Did you try to do that?

[brianc]

I am moving my domains to DomainKeys but that has not stopped the deferred yahoo.com issues. I think the best long term solution is simply educate yahoo.com users about how Yahoo is interfering with their mail delivery and to recommend that they start using another free e-mail provider.

I tried applying for Yahoo's feedback loop but they are asking for data on each and every domain name you want to receive notification for and I simply do not have the time to meet that type of request.

[Kent Brockman]

I do use this config for SPF and have to said that simply worked ok for me:
In WHM goto "Edit DNS Zone", select the domain and add a line like this:

yourdomain.com. 14400 IN TXT "v=spf1 a ptr ~all"

[interestingloss]

I have found out after numerous talks with yahoo "spam dept" that it is going to reject message and send them to the junk box or defer them for a while because of a few reasons. If you are using a shared IP to send mail via many domains they will automatically junk the message and defer it. Unless you have countless people click "not junk" in their box. This is the same with yahoo and hotmail.

Just because you have domain keys, DKIM, SPF all set up doesn't mean they will accept your mail. you need to visit postmaster.yahoo.com and fill out a form (if you have a dedicated IP sending mail for only that domain) They will not accept shared ips to add to their system as a valid email server.

The best thing temporarily is to notify the customer upon sign up of services that if they are using an free email that they emails may be delivered to their junk box and to add you to their safe senders list.

Chris

[brianc]

I have no problems with sending mail to hotmail accounts even for those domains that don't have SPF setup.

I have filled out Yahoo's forms, called them, etc. all to no avail. They just don't want to work with ISPs, especially web hosts. AOL is wonderful to work with in comparison to Yahoo.

[bking]

My understanding is that yahoo will accept 5 messages per connection. You can connect immediately again, but only 5 messages again. This ties in with what the OP was saying.. breaking the connection and starting again..

[BraveX]

Yahoo and AOL have caused huge headaches for us. Yahoo is worst. They are not very responsive. They send form replies. Pathetic. We are encouraging our customers not to use Yahoo and to encourage their customers to do the same.

If there's ever a class action suit, count us in!

[lloyd_tennison]

As far as I understand it, you would be creating a new running thread of exim each minute -which could definitely slow down the server. I have few problems with Yahoo, and their error messages are usually pretty clear. A great deal of the retrying is not intentional, so they say, is just a server busy - as they state the do not do greylisting.

I send multiple lists, and about 400,000 on certain days, and can have a backlog of up to 2,000 messages for Yahoo. Running exim with the -Rff yahoo great for me.

Hotmail is more the problem as I added a new IP. The old ones work fine - the new ones do not have the "reputation" the want.

Yahoo has also started a feedback loop. Signup for that and see if that also helps.

[cwboaze]

PLEASE DON'T DO THIS UNTIL THE EXPERTS CHIME IN

I manage a mailinglist with about one hundred thousand members. The list is double opt-in and has been maturing for 5 years. I've read enough to know I am not alone with the recent problems with yahoo deferring their email. We have about 18,000 yahoo subsribers. I've had 10,077 of them stuck in the queue all day and have after several hours of tweaking with settings and pushing the queue I was only able to send about 40 of them.

Finally by the end I was working from the shell using:

exim -qff -v

Yahoo keeps the connection live but stops talking to my server so the thing just hangs.

So this is what I finally ended up doing. This may not be a very good solution, but keep in mind I just need to get these emails out to our subsribers.

I tried different settings for triggering the queue and those didn't help. I finally set it to run once a day until I could work something out. This might be resource intensive but it works. Which means it might lead to a real solution from one of you guys that know your stuff.

So Yahoo just sits there and hangs and then eventually runs a few seconds and then stops again. However if you break into the session and start it again it picks right up. I did that for a short time manually and that was a bear.

No matter what it's going to reach a point and hang, sometimes for minutes on end. So what I did was put exim -qff in crontab to run every minute, and the mail is moving. The mail queue on my server is down to 8000 in the queue in about 45 minutes.

At least at this point it looks like the key to defeating these deferred emails is breaking the session stall and then starting up again.

I am going to push this as long as it works to get these emails out and then I'm putting eveything back to my servers default settings. I'd sure like to know if there's a way to do this without pegging the system. I have to restart Exim once an hour while I keep this up.

Thanks for any feedback, ideas, and solutions in advance.

Update: This process moved all the yahoo mail in the queue in about 6 hours.

As best as I can tell the starting of exim -qff every minute did the job. Just to play it safe I put restarted exim on the hour thoughout the process. Normally there would be email in this queue until it timed out. I tried one of the suggestions having WHM run the queue every 10 minutes earlier but that didn't work and I believe mainly due to the volume of emails. By the time I got things going I had emails that matured 20 hours.

Also worth mentioning, outside of web hosting, this server only hosts abuse and postmaster accounts for the domains we host, with one or two general support or contact accounts. So I have the luxury of taking a few chances. Other than bogging down the CPU and maybe losing some emails going to yahoo accounts I figured it was worth trying.

I too have been trying to work with Yahoo, but they are unresponsive. In the process of cleaning up their own act and the practices used to exploit thier services they're really slamming legiitmate emails from the mom and pop organizations who have built a healthy digital relationship with their customers over the years.

It is unfortunate that people who sign up for newletters and eventually tire from them siimply do not unsubscribe and therein lies the rub. Unless you can somehow get on Yahoo's feedback loop you don't know who they are so the more you mail them it seems the more you worsen your reputation as a sender. There doesn't seem to be anything you can do.

I think this fix will get me past what I plan on doing now. Which is remove the active status of all the Yahoo accounts we mail too and send a re-opt in email to reactivate their account.

What was your cron you added, I added

*/1 * * * * exim -qff -v

but, it emails me every minute with:
/bin/sh: exim: command not found

Thanks In Advance!

[WebHostDog]

Hello,
In cron put:

/usr/sbin/exim -qff -v

not only exim


Thanks,

[brianc]

Hello,
In cron put:

/usr/sbin/exim -qff -v

not only exim


Thanks,

I would get rid of the "-v" flag since you don't need that in your cron job since you don't need to see the output of a command that is run via cron.

[cwboaze]

Hello,
In cron put:

/usr/sbin/exim -qff -v

not only exim


Thanks,

From research, I don't want to receive a email evertime it runs,
so putting it this way would stop the reports. right!??

*/1 * * * * /usr/sbin/exim -qff -v MAILTO=""

Thanks, For your help!