Mail throttling [Case 59372]

[egillette]

Hey David,

Quick question. . .what tree is 11.31 in??

I tried RELEASE and I also tried CURRENT -- is it in the EDGE tree??

Neither RELEASE nor CURRENT upgraded me to 11.31. . .

[Infopro]

EDGE Tier. DKIM is not ready for prime time just yet and EDGE should be avoided unless you're comfortable working with problems. HTH!

[egillette]

EDGE Tier. DKIM is not ready for prime time just yet and EDGE should be avoided unless you're comfortable working with problems. HTH!

Thanks!

Well I'll wait until it's one of the other tiers then, because while I am totally comfortable working with problems, at the same time, the system I'm looking to set this up on is a production system, and I'd have clients raising all kinds of hell if snags are encountered.

On the lighter side, I do have a VPS that was just deployed, so I'll run it on that one, and see what happens. =0)

[SoftDux]

EDGE Tier. DKIM is not ready for prime time just yet and EDGE should be avoided unless you're comfortable working with problems. HTH!

Do you have an ETA on when DKIM will be supported in RELEASE, or even CURRENT builds?

[Infopro]

I do not. I wish I could give you more Info on this other than to say there is a lot of work going on with the EDGE tier right now in regard to DKIM but that's all I've got.

DavidG may have more to add here at some point that would be more useful to you.

[cPanelDavidG]

I do not. I wish I could give you more Info on this other than to say there is a lot of work going on with the EDGE tier right now in regard to DKIM but that's all I've got.

DavidG may have more to add here at some point that would be more useful to you.

The best place to get information about when new versions will propagate to any update tier is the cPanel Releases list. You can subscribe to that list at: - cPanel Inc.

[cPanelDavidG]

3. Last failure messages with rules:
if you receive a 421 error message for instance it would be logged and mail delivery attempts to that domain would cease or route to another virtual mail server in the cluster that doesn't throw the error, this would PROTECT ip's that a client for example is abusing, BEFORE it becomes permanently blacklisted. This would also send a notification mail to the owner of the server

This one seems like an interesting suggestion. However, just rotating through IPs is a good way to get ALL your IPs blacklisted. What if we just stopped delivery to a server when it generated a 421, emailed the admin so they can take a look and resume delivery after they are satisfied things are working.

I'm assuming this should also be behavior that should be able to be enabled/disabled for the hands-off admin types out there.

What do you all think?

[egillette]

This one seems like an interesting suggestion. However, just rotating through IPs is a good way to get ALL your IPs blacklisted. What if we just stopped delivery to a server when it generated a 421, emailed the admin so they can take a look and resume delivery after they are satisfied things are working.

I'm assuming this should also be behavior that should be able to be enabled/disabled for the hands-off admin types out there.

What do you all think?

I like that idea -- A LOT!!

This would prevent MSP's like Yahoo from outright blocking you because you're "sending too fast" or "making too many connections" and would allow you to determine the best way for each individual IP/client how you want to send messages, to prevent blacklisting.

[cPanelDavidG]

I met with our Product Manager who let me know that our current mail delivery failure etc. functionality even as of 11.31 does not directly look at the status codes returned by servers, instead it looks for other indicators that a message failed to be delivered like FAILED or ** when we look for the status of an email.

As a result, let me get a call for comments started. Please review the below for accuracy and completeness before I file a case about it with our developers in 2 weeks.

Call for Comments:

It is common practice for a server that feels you are sending it spam to send you a status reply code of 421. While this server may continue to accept connections, your server is essentially gray-listed and persistent connections get you blacklisted.

SMTP Reply Code 421 according to specification is not explicitly for this purpose, just to indicate that the service is unavailable. However, it is widely assumed that SMTP Reply Code 421 indicates that one's server has previously sent that server spam. It is understood this assumption can lead to false positives.

As a result, server owners desire the functionality that if a single SMTP Reply Code 421 is generated, no further email is sent to that destination, ever. Furthermore, a screen is added to WHM to list all the email not delivered as a result of this so system administrators can investigate the issue and upon their investigation, perhaps choose to move those messages back into the outbound queue for delivery or discard those messages as spam.

This would be enabled via a tweak setting that would be disabled by default as not to inadvertently result in most email eventually not being delivered.

[monarobase]

My question about this is, what about servers that do grey listing ? Lots of servers do grey listing and you could end up with 100's of new e-mails everyday in the list if it's done on a per server basis and not on a per sender basis…

Also what about setting a limit over a period of time (that can be 1 so that the first one triggets it) but could also be 50 or 100 per period of time (1 hour, 2 hours, 3 hours…) that would block all e-mails sent over the next period of time (1 hour, 2 hours, 3 hours…)

[egillette]

My question about this is, what about servers that do grey listing ? Lots of servers do grey listing and you could end up with 100's of new e-mails everyday in the list if it's done on a per server basis and not on a per sender basis…

Also what about setting a limit over a period of time (that can be 1 so that the first one triggets it) but could also be 50 or 100 per period of time (1 hour, 2 hours, 3 hours…) that would block all e-mails sent over the next period of time (1 hour, 2 hours, 3 hours…)

I agree. . .however I think it should be a setting in this proposed cPanel feature where the server admin could choose one of the following:

1) Stop/Delay delivery to domain based on specific IP receiving 421 response (allows messages to be sent to the domain from other IP's on the server) -- also IP's should be displayed allowing the server-admin to check off the ones he wants to enable this policy for. Please be sure to add a check-all and an uncheck-all checkbox as well for those of us who have hundreds of IP's on a server. (Enables "A" and "B" and "C" below)

2) Stop/Delay delivery to domain based on sender (i.e. customer@<domain>) receiving 421 response (allows other senders on other domains to send to that domain irrespective of whether their domain is hosted on the same IP or a different one). (Again all domains on the server should be displayed allowing the server admin to click a check box next to each domain he wants this policy to apply to. Please be sure to add a check-all and an uncheck-all checkbox as well for those of us who have hundreds of domains on a server. (Enables "A" and "B" and "C" below)

3) Stop/Delay delivery to domain based for entire server (server-wide policy) if a 421 response is received for any message on the server (this would be for server admins like me who want to be able to scrutinize even a single 421 response to prevent blacklisting). (Enables "A" and "B" and "C" below)

4) Default would be "(*) Disabled - No 421 Response Mailing Policy" (Disables all options below)

I also agree with monarobase in that enabling any of the other 3 options, would activate a list of sub-options that would allows the server-admin to choose:

A) Limit (how many 421 responses are needed to trigger one of the above selected policies). The server admin can type in a number, default would be 1 (meaning a single 421 response would trigger the policy chosen above).

And this next option I preface by saying I think it would be the most USEFUL option:

B) Stop Delivery Completely To Domain based on policy above (this would put the subsequent messages into a queue for the server admin to review and either put messages back into the outbound queue, put messages back into the outbound queue with a specified delay [server admin would be able to set delay in hours by typing in a number, since a drop-down with pre-existing options of 1 hour, 2 hours, etc may not suffice if the admin wants to delay a message or group of messages for 72 hours for example], delete the specified messages, or bounce specified messages to sender(s) [this would return the messages to the person/people who sent them allowing that person/people to then follow up with the server admin as to why their message(s) were returned to them].) (Enables "I" and "II" Below)

C) Delay Delivery To Domain automatically based on policy above (This option would have a corresponding text box the server admin to type in a number and then select from a drop-down either minutes, hours, or days for the delay allowing the server admin to indicate how long a message will be delayed where a 421 response is received for that message). And then the server-admin would have to set a limit below as well if they choose this option. (Enables "I" and "II" Below)

I) 421 Response Limit (i.e. how many times can a specific message be delayed), and then they'd choose from a drop-down that says 1 time, 2 times, 3 times, 4 times. Anything more than that would just bottleneck their system anyway and cause mail queues to become massive depending on the volume of messages sent, and then after this, they'd then choose one of the options below:

II) Final Message Disposition (i.e. what happens when the message has reached the 421 response limit), and then they'd choose from one of 3 options: Move Message to Review Queue (for server admin to decide final message disposition -- requeue message, requeue message with delay, bounce to sender, or delete message), Message automatically deleted, Message bounced to sender.

I think structured in this way, server admins who are strict about 421 policies and server admins who are more lenient with 421 policies would both be accommodated since this setup provides the most flexibility for both types of admins.

[cPanelDavidG]

My question about this is, what about servers that do grey listing ? Lots of servers do grey listing and you could end up with 100's of new e-mails everyday in the list if it's done on a per server basis and not on a per sender basis…

Also what about setting a limit over a period of time (that can be 1 so that the first one triggets it) but could also be 50 or 100 per period of time (1 hour, 2 hours, 3 hours…) that would block all e-mails sent over the next period of time (1 hour, 2 hours, 3 hours…)

At this time we simply do not support nor accommodate graylisting. Given the wider implications of supporting graylisting, I believe that graylist support/accommodation should be its own feature request.

[cPanelDavidG]

I agree. . .however I think it should be a setting in this proposed cPanel feature where the server admin could choose one of the following:

1) Stop/Delay delivery to domain based on specific IP receiving 421 response (allows messages to be sent to the domain from other IP's on the server) -- also IP's should be displayed allowing the server-admin to check off the ones he wants to enable this policy for. Please be sure to add a check-all and an uncheck-all checkbox as well for those of us who have hundreds of IP's on a server. (Enables "A" and "B" and "C" below)

2) Stop/Delay delivery to domain based on sender (i.e. customer@<domain>) receiving 421 response (allows other senders on other domains to send to that domain irrespective of whether their domain is hosted on the same IP or a different one). (Again all domains on the server should be displayed allowing the server admin to click a check box next to each domain he wants this policy to apply to. Please be sure to add a check-all and an uncheck-all checkbox as well for those of us who have hundreds of domains on a server. (Enables "A" and "B" and "C" below)

3) Stop/Delay delivery to domain based for entire server (server-wide policy) if a 421 response is received for any message on the server (this would be for server admins like me who want to be able to scrutinize even a single 421 response to prevent blacklisting). (Enables "A" and "B" and "C" below)

4) Default would be "(*) Disabled - No 421 Response Mailing Policy" (Disables all options below)

I also agree with monarobase in that enabling any of the other 3 options, would activate a list of sub-options that would allows the server-admin to choose:

A) Limit (how many 421 responses are needed to trigger one of the above selected policies). The server admin can type in a number, default would be 1 (meaning a single 421 response would trigger the policy chosen above).

And this next option I preface by saying I think it would be the most USEFUL option:

B) Stop Delivery Completely To Domain based on policy above (this would put the subsequent messages into a queue for the server admin to review and either put messages back into the outbound queue, put messages back into the outbound queue with a specified delay [server admin would be able to set delay in hours by typing in a number, since a drop-down with pre-existing options of 1 hour, 2 hours, etc may not suffice if the admin wants to delay a message or group of messages for 72 hours for example], delete the specified messages, or bounce specified messages to sender(s) [this would return the messages to the person/people who sent them allowing that person/people to then follow up with the server admin as to why their message(s) were returned to them].) (Enables "I" and "II" Below)

C) Delay Delivery To Domain automatically based on policy above (This option would have a corresponding text box the server admin to type in a number and then select from a drop-down either minutes, hours, or days for the delay allowing the server admin to indicate how long a message will be delayed where a 421 response is received for that message). And then the server-admin would have to set a limit below as well if they choose this option. (Enables "I" and "II" Below)

I) 421 Response Limit (i.e. how many times can a specific message be delayed), and then they'd choose from a drop-down that says 1 time, 2 times, 3 times, 4 times. Anything more than that would just bottleneck their system anyway and cause mail queues to become massive depending on the volume of messages sent, and then after this, they'd then choose one of the options below:

II) Final Message Disposition (i.e. what happens when the message has reached the 421 response limit), and then they'd choose from one of 3 options: Move Message to Review Queue (for server admin to decide final message disposition -- requeue message, requeue message with delay, bounce to sender, or delete message), Message automatically deleted, Message bounced to sender.

I think structured in this way, server admins who are strict about 421 policies and server admins who are more lenient with 421 policies would both be accommodated since this setup provides the most flexibility for both types of admins.

Does anyone else desire this level of complexity? I ask because such complexity will significantly add to implementation time. Also, it presents its own issues with usability. Also, should cPanel account-level stuff be somehow meshed in with the existing (in 11.32) functionality for what happens when an account generates too many email failure messages when that triggers a policy rather than a server-wide setting? What about 421 bouncebacks being queued similar to when a cPanel account sends more email than they are permitted for a 1 hour period? Just looking for ways to mesh this into existing expectations of the software as to make things simpler for techs that will need training on this system in the future.

I don't want to cut things out to the point of it being useless and not resolving this problem, at the same time I would prefer a simple solution (not just with regards to implementation, but with regards to using this effectively).

[sehh]

Personally, I don't want to go into that much detail. I don't even want to see the emails that bounced. I would prefer if they bounced back to the sender, who can then investigate and see what the problem is.

[cPanelDavidG]

I have delivered the previously posted call for comments to our development team. This has been assigned a case number which appears in the title of this thread.