Mailman troubleshooting tips

[sneader]

Yesterday, I noticed several mailman processes that were taking up significant cpu. I tailed /var/log/exim_mainlog but didn't see anything interesting. I restarted exim via WHM, no change.

I killed one of the mailman processes with a basic 'kill pid', then the load shot up very high. I then killed off the two remaining mailman processes, and the box went back down to it's normal 0.5 load. Problem solved or so I thought.

Today, I have had a couple customers say their list posts are going into "thin air".

I have run /scripts/fixmailman in hopes of getting things going again.

HOWEVER... I realized that I am not finding much in the forums, with regards to troubleshooting Mailman problems. I've seen people reference "starting and stopping Mailman"... "checking to be sure the qrunner is running"... and "reviewing Mailman logs"... but nobody is referencing HOW to do these things.

So... anybody out there willing to share Mailman troubleshooting tips?

- Scott

[cpanelkenneth]

These are the Mailman processes:

mailman  21608  0.0  0.4 11848 4556 ?        Ss   Oct01   0:00 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/mailmanctl -s start
mailman  21620  0.0  0.5  9948 5944 ?        S    Oct01   0:00 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=ArchRunner
mailman  21621  0.0  0.5 10652 6080 ?        S    Oct01   0:01 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=BounceRunn
mailman  21622  0.0  0.5 11228 5948 ?        S    Oct01   0:00 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=CommandRun
mailman  21623  0.0  0.5 10784 6128 ?        S    Oct01   0:01 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRu
mailman  21624  0.0  0.5 11260 5984 ?        S    Oct01   0:00 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=NewsRunner
mailman  21625  0.0  0.6 11156 6344 ?        S    Oct01   0:01 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRu
mailman  21626  0.0  0.5 11368 6168 ?        S    Oct01   0:01 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=VirginRunn
mailman  21627  0.0  0.5 11132 6088 ?        S    Oct01   0:00 /usr/local/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=RetryRunne

As you can see, some of the processes are qrunner. A simple restart using /usr/local/cpanel/etc/init/startmailman will ensure all the mailman processes are running.

You can check for mailman deliveries in /var/log/exim_mainlog:

grep -i mailman /var/log/exim_mainlog


Should reveal items like:

2008-10-02 08:00:04 1KlNmm-0003d9-K5 <= george-bounces@abc.com H=localhost (mundane.cpanel.com) [127.0.0.1] P=esmtp S=1129 id=mailman.0.1222952402.13948.george_abc.com@abc.com
2008-10-02 08:00:05 1KlNmm-0003d9-K5 => george <george-owner@abc.com> R=mailman_virtual_router T=mailman_virtual_transport
2008-10-02 08:00:05 1KlNmn-0003dE-Pz <= mailman-bounces@mundane.cpanel.com H=localhost (mundane.cpanel.com) [127.0.0.1] P=esmtp S=1404 id=mailman.0.1222952402.13948.george_abc.com@abc.com

Once you verify the mailman processes are running, start looking through exim_mainlog for entries related to the mailing list(s) in question.

[sneader]

Kenneth, you are my hero! This is exactly what I was looking for!

Thanks!

- Scott

[cpanelkenneth]

Kenneth, you are my hero! This is exactly what I was looking for!

Thanks!

- Scott

Always glad to help.

[4u123]

Is there a way of logging indivdual messages sent by mailman for a particular mailing list?

When I look at the logs I just see that a message was sent to mailman_virtual_transport. It doesnt log each attempted delivery.

A customer send out two emails to her list members yesterday and she says none of them received the email. I can see the message was sent but thats about it. I've done a fixmailman so hopefully that will have resolved it but I'm not any clearer as to why the messages were not received.

She says the email went into her mailman archive, and she is receiving emails from the list as the admin.

[sneader]

They'd all be logged in exim_mainlog.

Ask her what e-mail addresses claimed not to get them, then grep your exim_mainlog for them.

grep user@domain.com /var/log/exim_mainlog | more

You will see what is happening pretty quickly.

- Scott

[craigedmonds]

so now that we can identify whats going on in the mailman log, is there a ssh command to STOP certain lists from certain bounces being generated?

The issue I am am having right now is that I have removed a mailman list from the server a few days ago, but its still generating s shed load of bounces every few minutes.

[sneader]

so now that we can identify whats going on in the mailman log, is there a ssh command to STOP certain lists from certain bounces being generated?

The issue I am am having right now is that I have removed a mailman list from the server a few days ago, but its still generating s shed load of bounces every few minutes.

Not enough information here to help. Need more info about these bounces. Examples perhaps?

- Scott

[craigedmonds]

here is an example of one of the bounces, so far I have had to clear out around 140,000 of these manually via mail manage.

Even though the mailing list was deleted from the server over 5 days ago its still generating them at the rate of several hundred a minute.

If I turn off mailman, they stop, which is not the point really as I have people using mailman in a legit fashion.

1OVmkT-0006BN-N6-D
This is a reminder, sent out once a month, about your
biphasic-mesothelioma.info mailing list memberships.  It includes your
subscription info and how to use it to change it or unsubscribe from a
list.

You can visit the URLs to change your membership status or
configuration, including unsubscribing, setting digest-style delivery
or disabling delivery altogether (e.g., for a vacation), and so on.

In addition to the URL interfaces, you can also use email to make such
changes.  For more info, send a message to the '-request' address of
the list (for example, list-request@biphasic-mesothelioma.info)
containing just the word 'help' in the message body, and an email
message will be sent to you with instructions.

If you have questions, problems, comments, etc, send them to
list-owner@biphasic-mesothelioma.info.  Thanks!

Passwords for securedone@yahoo.com:

List                                     Password // URL
----                                     --------  
list@biphasic-mesothelioma.info          uhoxufid  
http://biphasic-mesothelioma.info/mailman/options/list_biphasic-mesothelioma.info/securedone%40yahoo.com 

[Miraenda]

In regards to logs, I did also want to mention that for qrunner and error logs (not the delivery logs), you can view those at /usr/local/cpanel/3rdparty/mailman/logs location (error and qrunner are the names of the log files).

As for this list, I would bet it didn't get completely removed from mailman. If you go to /usr/local/cpanel/3rdparty/mailman area, you can check the lists folder to see if the mailing list is still showing up there as list_biphasic-mesothelioma.info folder. If it is, remove the folder.

Next, go to /usr/local/cpanel/3rdparty/mailman/archives/private and /usr/local/cpanel/3rdparty/mailman/archives/public and remove the list_biphasic-mesothelioma.info and list_biphasic-mesothelioma.info.mbox folders if any exist there (normally, public will simply have a symlink to the private one).

If you don't find any of these there and it still keeps sending, then you might want to submit a technical support ticket for assistance in checking what wasn't removed when the list was deleted in cPanel.

[m4rc3]

I wrote a little "how to" that explains how to enable debug logging on mailman.
troubleshooting mailman problems

It's one of those things that really save you a lot of headaches when dealing with problematic mailing lists.

[sneader]

Thanks, I've bookmarked this for future reference. FYI, your link is to an https page, but it is giving SSL errors.

- Scott

[craigedmonds]

I wrote a little "how to" that explains how to enable debug logging on mailman.
troubleshooting mailman problems

It's one of those things that really save you a lot of headaches when dealing with problematic mailing lists.

I now just disable mailman on my servers, its not worth the hassle. If people want mailing lists I tell them to bring them selves up to date and use something like mailchimp/aweber or constant contact.

Here is my process for disabling mailman on each new sevrer I set up now.

1. Run the following command in SSH which will stop the mailman process from running: /usr/local/cpanel/3rdparty/mailman/bin/mailmanctl stop

2. Create an .htaccess for mailman - Create a file at in the following location on your server: 
pico /usr/local/cpanel/3rdparty/mailman/cgi-bin/.htaccess

3. put the following code in this .htaccess folder, this will stop users accessing it:

<Limit GET POST>
order deny,allow
deny from all
</Limit>

<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>

4. Finally, add the .htaccess to the exclude file so cPanel won’t remove / change it. Run the following in SSH:

"/usr/local/cpanel/3rdparty/mailman/cgi-bin/.htaccess" >> /etc/cpanelsync.exclude

I hope this helps someone else from their mailman nightmares.

[m4rc3]

@Scott: It's the twitter plugin that doesn't run on https so the browser tells you some parts of the site are unencrypted., either way I'll be moving the blog to http in the next few days. Thanks for the heads up

@craigedmonds: you can disabled mailman from WHM, it's under "tweak settings"

[craigedmonds]

@craigedmonds: you can disabled mailman from WHM, it's under "tweak settings"

Yes mate you can "disable" mailman from WHM but it still allows users to access it. The only sure fire way to permanently disable it and not give users access is to follow the procedure as mentioned in my post.

I had LOTS of issues with mailman, its very easy for cunning users to send out millions of spams with it.