mailman/cpanel bug can lead to false unsubcriptions!

[cyberspirit]

Here is what I found out so far:

The latest mailman version 2.1.5. uses a new mechanism to test bouncing addresses. If an address has been bouncing and is marked to be unsubscribe it will be tested one last time with an email "probe". This probe is sent in VERP format, if it fails the email addresses is unsubscribed right away!
Now here lies the problem. Cpanel's exim configuration is not fully VERP aware. So this means these bounces fail right away and will lead to false unsubscriptions!!!
One of my customer has thousands of unsubcriptions in one day. We tested about 100 of these addresses and they are all ok!
I have submitted this bug to the cpanel team and we will see what they will do with it.
For the time being I suggest turning the bounce handling completely off if you do not want to scrue up lists completely.

[cyberspirit]

yep, mailman did not notify anyone about this little changed behaviour. so typical for the mailman developers, I do not like how they ignore the whole community anyway, it is not the first time they screw up things big time!
thanks to Nick to look into this right away and fix it. There is a nice VERP aware config for exim from Nigel, that should help so Nick does not need to reinvent the wheel..... ;-)

[Osaka]

hello

i am not seeing this behavior with any of my clients lists across 4 servers. does anyone know how to tell if mailmans VERP is causing users to be unsibscribed ?
any guidance in replicating this problem will be most appreciated.

very many thanks

[darren]

Hi All,

I have been looking into this as well. Looking at http://exim.mirror.anlx.net/howto/mailman21.html , it would appear the mailman_virtual_transport is currently set up to work with mailman+VERP ; according to Nigel's howto ::

"VERP settings within Mailman are done on a per-installation basis - ie they affect all the lists within the installation. To configure VERP within Mailman read the information in ~mailman/Mailman/Default.py for the options that start with VERP. In a nutshell, all you need to do to enable VERP with Exim is to add these lines to ~mailman/Mailman/mm_cfg.py:
( on cpanel boxes: /usr/local/cpanel/3rdparty/mailman/Mailman/mm_cfg.py )

VERP_PASSWORD_REMINDERS = 1
VERP_PERSONALIZED_DELIVERIES = 1
VERP_CONFIRMATIONS = 1
VERP_DELIVERY_INTERVAL = 1

(The router and ACLs above are smart enough to deal with VERP bounces.)"

For those obviously suffering from this problem, do you have a support ticket opened with us ? I would like to be able to replicate the problem so we can get it taken care of asap.

Thanks,
Darren

[cyberspirit]

Darren,
If you read up on VERP you will find out that the MTA needs to be able to route VERP-addresses properly. Right now the way exim is set up to route mailman addresses VERP does not work. I contacted Nick months ago about the VERP problem and never heard back. Now, with the change in bounce probing in mailman 2.1.5. this issue obviously became very important. Why can't you guys act more proactively? VERP has been discussed for a long time now, not only on the mailman list. If you as cpanel integrate a product like mailman then I expect that you stay up-to-date on it and its developments and not just jump into action after everything broke.
The fact alone that Nigel felt the need to post new VERP aware scripts examples for exim shows that this needs special handling. Again, something that you could have known about simply by following the mailman developers mailing list.
Now, the big questions is, are you guys staying up-to-date on the planned 3.x mailman or will you once again just read up on it once it is already out?

[Marty]

Has this patch been issued yet?

[cyberspirit]

no, there has been no patch yet. I guess the "24" hours that were promised for this patch will more turn into 24 days..........

[cyberspirit]

Well thaphantom,
It looks like your connection to Nick is not as good as you make it sound like. You mentioned that there would be a fix within 48 hours and nothing yet. In fact, the bugreport 618 has been out for a while and nothing. This is a critical issue and nothing is being done. And I have to correct you, mailman did properly document and report the change when 2.1.5 came out. And since cpanel was not really one of the early adopters of mailman 2.1.5 all the information was in the readme. Sometimes it helps to read readmes, lol.
And in response to Darren, VERP can be turned off in mailman 2.1.5. but not the VERP bounce probes that create the problem. So there is no fix other than setting the bounce counts very high for a list and wait until cpanel finally realizes the severity of the issue and patches the exim.conf

[cyberspirit]

Thaphantom, once again you seem out of touch with the facts and current developments. The bug has been verified by many and as you can see by the response of this cpanel tech it is a known issue:

Hello,

By defualt exim is not setup to use VERP and mailman was by default not using VERP.

It seems recently mailman was changed to use VERP by default.

The solution would be to turn VERP on in exim as well as mailman.

That is on the to do list here: http://bugzilla.cpanel.net/show_bug.cgi?id=618

You can subscribe yourself to the CC lust to be notified of any updates to the issue.

Thanks
--
Daniel Muey
Technical Support
cPanel Inc

[cyberspirit]

A tickets was opened for this but closed by cpanel tech support because it is a known bug and will be addressed by the people working on solving reported bug 618 which was opened by cpanel tech support.
It is actually very easy to see that current cpanel added mailman transports in exim.conf do NOT support VERP addresses. This issue only comes up when mailman tries the new default VERP bounce probes and is independent of any VERP settings in either Default.py or mm_cfg.py
(How many times do I need to repeat this until thaphantom will get it?).

[cpanelnick]

Originally posted by cyberspirit
A tickets was opened for this but closed by cpanel tech support because it is a known bug and will be addressed by the people working on solving reported bug 618 which was opened by cpanel tech support.
It is actually very easy to see that current cpanel added mailman transports in exim.conf do NOT support VERP addresses. This issue only comes up when mailman tries the new default VERP bounce probes and is independent of any VERP settings in either Default.py or mm_cfg.py
(How many times do I need to repeat this until thaphantom will get it?).

http://www.python.org/cgi-bin/faqw-m...=faq02.002.htp

exim -bt <listname>-bounces+r.barrett=openinfo.demon.co.uk@mailman.domain.com

works fine for me

# exim -bt zipper-bounces+r.barrett=openinfo.demon.co.uk@koston.org
zipper-bounces+r.barrett=openinfo.demon.co.uk@koston.org
router = mailman_virtual_router, transport = mailman_virtual_transport


If it was broken it would have returned " Unrouteable address
"

[cpanelnick]

I've also enabled VERP

VERP_PASSWORD_REMINDERS = 1
VERP_PERSONALIZED_DELIVERIES = 1
VERP_CONFIRMATIONS = 1
VERP_DELIVERY_INTERVAL = 1

(in Defaults.py)

on the list zipper@koston.org I welcome you to find a problem with it.

[cyberspirit]

Thanks for the invitation to use your list for my tests but that would not work unless you are willing to give me root access.
Reason being that changes in Defaults.py or mm_cfg.py only become effective after the maiman thread has restarted (default in a cpanel setup is several hours).

Anyway, I have done some further testing and have this excerpt here from the exim mailqueue (in WHM) as proof that VERP bounce probes are not handled correctly and end up in nowhere.
I have changed the domain name and ip address of the mailman host for privacy and security reasons. But in a direct email to you Nick I have sent you the original message.

Message 1BgxIx-0002LY-7t is no longer frozen
transport error EPIPE ignored
LOG: MAIN
** testlist_domain.net@domain.net <testlist_domain.net-bounces+f59acd63293cb9aa8a52e5ebfc11bf02f746b879@domain.net> R=mailman_virtual_router_nodns T=mailman_virtual_transport: Child process of mailman_virtual_transport transport returned 1 from command: /usr/local/cpanel/3rdparty/mailman/mail/mailman
LOG: MAIN
Frozen (delivery error message)
Return to Mail Queue



1BgxIx-0002LY-7t-H
mailnull 47 12
<>
1088910275 0
-helo_name omr-m12.mx.aol.com
-host_address 64.12.136.10.58402
-interface_address 65.65.65.65.25
-received_protocol esmtp
-body_linecount 89
-frozen 1088910814
-manual_thaw
XX
1
testlist_domain.net-bounces+f59acd63293cb9aa8a52e5ebfc11bf02f746b879@domain.net

243P Received: from [64.12.136.10] (helo=omr-m12.mx.aol.com)
by smtp.domain.net with esmtp (Exim 4.34)
id 1BgxIx-0002LY-7t
for testlist_domain.net-bounces+f59acd63293cb9aa8a52e5ebfc11bf02f746b879@domain.net; Sat, 03 Jul 2004 20:04:35 -0700
176 Received: from rly-ya02.mx.aol.com (rly-ya02.mail.aol.com [172.18.141.34]) by omr-m12.mx.aol.com (v98.19) with ESMTP id RELAYIN7-840e773a71c6; Sat, 03 Jul 2004 23:04:07 -0400
157 Received: from localhost (localhost)
by rly-ya02.mx.aol.com (8.8.8/8.8.8/AOL-5.0.0)
with internal id XAB24594;
Sat, 3 Jul 2004 23:04:07 -0400 (EDT)
043 Date: Sat, 3 Jul 2004 23:04:07 -0400 (EDT)
054F From: Mail Delivery Subsystem <MAILER-DAEMON@aol.com>
056I Message-Id: <200407040304.XAB24594@rly-ya02.mx.aol.com>
088T To: <testlist_domain.net-bounces+f59acd63293cb9aa8a52e5ebfc11bf02f746b879@domain.net>
018 MIME-Version: 1.0
113 Content-Type: multipart/report; report-type=delivery-status;
boundary="XAB24594.1088910247/rly-ya02.mx.aol.com"
037 Subject: Returned mail: User unknown
041 Auto-Submitted: auto-generated (failure)
024 X-AOL-IP: 172.18.141.34
041 X-WhitelistedRCPT-nohdrfromcallback: Yes
071 X-MailScanner-Information: Please contact the ISP for more information
033 X-MailScanner: Found to be clean
027 X-MailScanner-SpamScore: s


1BgxIx-0002LY-7t-D
This is a MIME-encapsulated message

--XAB24594.1088910247/rly-ya02.mx.aol.com

The original message was received at Sat, 3 Jul 2004 23:03:47 -0400 (EDT)
from www.domain.net [65.65.65.65]


*** ATTENTION ***

Your e-mail is being returned to you because there was a problem with its
delivery. The address which was undeliverable is listed in the section
labeled: "----- The following addresses had permanent fatal errors -----".

The reason your mail is being returned to you is listed in the section
labeled: "----- Transcript of Session Follows -----".

The line beginning with "<<<" describes the specific reason your e-mail could
not be delivered. The next line contains a second error message which is a
general translation for other e-mail servers.

Please direct further questions regarding this message to your e-mail
administrator.

--AOL Postmaster



----- The following addresses had permanent fatal errors -----
<blah94523@aol.com>

----- Transcript of session follows -----
... while talking to air-ya03.mail.aol.com.:
>>> RCPT To:<blah94523@aol.com>
<<< 550 MAILBOX NOT FOUND
550 <blah94523@aol.com>... User unknown

--XAB24594.1088910247/rly-ya02.mx.aol.com
Content-Type: message/delivery-status

Reporting-MTA: dns; rly-ya02.mx.aol.com
Arrival-Date: Sat, 3 Jul 2004 23:03:47 -0400 (EDT)

Final-Recipient: RFC822; blah94523@aol.com
Action: failed
Status: 5.1.1
Remote-MTA: DNS; air-ya03.mail.aol.com
Diagnostic-Code: SMTP; 550 MAILBOX NOT FOUND
Last-Attempt-Date: Sat, 3 Jul 2004 23:04:07 -0400 (EDT)

--XAB24594.1088910247/rly-ya02.mx.aol.com
Content-Type: text/rfc822-headers

Received: from smtp.domain.net (www.domain.net [65.65.65.65]) by rly-ya02.mx.aol.com (v100.23) with ESMTP id MAILRELAYINYA210-14a40e773923d2; Sat, 03 Jul 2004 23:03:47 -0400
Received: from localhost ([127.0.0.1] helo=smtp.domain.net)
by smtp.domain.net with esmtp (Exim 4.34)
id 1BgxI2-0002IT-Iv
for blah94523@aol.com; Sat, 03 Jul 2004 20:03:38 -0700
Subject: Testlist mailing list probe message
From: testlist_domain.net-bounces+f59acd63293cb9aa8a52e5ebfc11bf02f746b879@domain.net
To: blah94523@aol.com
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1346133609=="
Message-ID: <mailman.1.1088910216.6801.testlist_domain.net@domain.net>
Date: Sat, 03 Jul 2004 20:03:36 -0700
Precedence: bulk
X-BeenThere: Testlist@domain.net
X-Mailman-Version: 2.1.5
List-Id: testlist_domain.net.domain.net
X-List-Administrivia: yes
Sender: testlist_domain.net-bounces+f59acd63293cb9aa8a52e5ebfc11bf02f746b879@domain.net
Errors-To: testlist_domain.net-bounces+f59acd63293cb9aa8a52e5ebfc11bf02f746b879@domain.net
X-MailScanner-Information: Please contact the ISP for more information
X-MailScanner: Found to be clean
X-MailScanner-SpamScore: sss
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - smtp.domain.net
X-AntiAbuse: Original Domain - aol.com
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - domain.net
X-Source:
X-Source-Args:
X-Source-Dir:
X-AOL-IP: 65.65.65.65
X-AOL-SCOLL-SCORE: 0:XXX:XX
X-AOL-SCOLL-URL_COUNT: 0

--XAB24594.1088910247/rly-ya02.mx.aol.com--

[cyberspirit]

Just to update everyone who is waiting for news on this bug (it seems nobody from the cpanel team feels the need to update the rest of us about this):

Nick from cpanel has finally agreed that there is a problem in the combination of cpanel/exim/mailman. According to him it is a problem with the mailman integration. This problem only arrises with VERP bounce probe messages from mailman that are sent out but bounce back from the receiving smtp host. They then end up in the exim queue and for some reason are not handed off correctly to mailman.
Nick has created a patch and once I hear what the details and changes are in this patch (always good to know before testing something) I will try it and see if it solves the problem.
I just wonder now how thorough the testing of thaphantom and others was.... ;-):
Thaphantom: "4 people have looked into the issue, 3 from cpanel plus me and wnnot find the error. It is working corretly on atleasta dozen servers."

[cyberspirit]

Well, it looks like now even the version history of cpanel recognizes this bug and has posted a fix for it:

-----------
9.4.1 (build 80) Sun Jul 4 09:29:20 2004

patch mailman to make verp probes work correctly
------------

Let's see what people report here, I will test it once I hear back from cpanel what the changes are they made.