High Exim Load: Suggestions to improve??

[bmcpanel]

I use RH 7.3, CPanel latest edge.
I use suexec and have disallowed nobody from sending mail.
I use CLam AV with BFD.

I have a lot of clients who use email during business hours. My server load average will shoot to 7-9 and averages around 3-4 during business hours. It then levels off to 0.7 - 1.3 during the evening.

I watch the server with "top" and "pstree", but can only see that exim processes are open and cannot see who has opened them.

I guess I am seeking suggestions about how to reduce the load average on the server that is being caused by exim. What would you guys do in this instance?

[dezignguy]

tail your exim logs for a while... though it can be hard to extract any useful info from them... so use WHM's exim statistics page and see how your traffic looks.

Are you using any spam or virus filter? Those can significantly increase loads because of the cpu/disk intensive scanning.

set a limit on email size in exim... emails with huge attachments can easily bog down a mailserver... and the office drones really don't need to be sending 10 or 20 MB AVI movies to each other. Use an ftp server or a fileshare for large files.

Consider getting a more powerful server...

Consider moving half your clients onto another server.

Reject (:fail:) unwanted email in the smtp conversation (spam, viruses, and email sent to non-existent accounts). Don't use catch-all addresses.

[bmcpanel]

tail your exim logs for a while... though it can be hard to extract any useful info from them... so use WHM's exim statistics page and see how your traffic looks.

I do this daily. I don't notice anything out of the ordinary, except for a large level of incoming spam, which I suppose most of us have a problem with. Also, the load is always higher during business hours (9am to 5pm).

Are you using any spam or virus filter? Those can significantly increase loads because of the cpu/disk intensive scanning.

Yes, Clam AV with Brute Force Detection installed. When I first installed Clam AV, it reduced my server load because the mail server was not delivering all that junk mail. Now, I don't know... maybe it is the BFD that is causing the high load. Don't know.

set a limit on email size in exim... emails with huge attachments can easily bog down a mailserver... and the office drones really don't need to be sending 10 or 20 MB AVI movies to each other. Use an ftp server or a fileshare for large files.

That's a good idea. Where is the global setting for this, exim.conf?

Consider getting a more powerful server...

Probably not an issue as this problem did not exist on this server a few months ago and the number of accounts is about the same as before.

Consider moving half your clients onto another server.

See above...

Reject (:fail unwanted email in the smtp conversation (spam, viruses, and email sent to non-existent accounts). Don't use catch-all addresses.

I prefer to :blackhole: them so exim doesn't waste time sending a failure reply to the origin address.

Thanks for the suggestions.

[junglecat]

I prefer to :blackhole: them so exim doesn't waste time sending a failure reply to the origin address.

I may be wrong, but I *think* :blackhole: makes them bounce around in the server for a while before they are deleted, and :fail: means they never enter the server.

Therefore :fail: should reduce server resources.

Someone correct me if my information is faulty.

[DavidR]

From everything that I read, you are correct - :fail: is less server intensive and much preferred over :blackhole:.

David

[chirpy]

Yup, :blackhole: is a bad idea:
http://www.configserver.com/free/fail.html

[dezignguy]

heh, I got stalled looking for your link, chirpy.

bmcpanel, I limit the size of messages with this near the top of exim.conf (you can put it in the first box in the WHM config editor)

message_size_limit = 11M

That limits mails to 11M... anything coming in through SMTP, I believe... which means incoming and outgoing mail will be limited together. But 11 megs is quite a large enough email incoming or outgoing.

[chirpy]

But 11 megs is quite a large enough email incoming or outgoing

Indeed, IIRC, a default sendmail installation limits (or at least used to limit) mail to 5MB.

[eurorocco]

Someone is poking at port 25 with several threads, probably from a host connected with broadband. Then I see the mail log and it is someone poking at exim.

Also, it could be because one of your customers is sending out an email with a very long CC or BCC line, and heavy attachments, like many of the messages in the thread have already pointed out.

I have several simple Perl scripts, since I haven't found a better solution.

One is cpumonster, that goes in the crontab and checks /proc/loadavg every minute. If it is greater than 8, then service chkservd stop, service cpanel stop, service http stop, service exim stop. It does everything it can do to cool the server down. Then, a notifier in my console tells me http there is down, then I investigate. If the load is greater than 3, just shutdown http. It's an arbitrary choice I made to never never see the load skyrocket to 90. If it does 90 I cannot even ssh into the box, so I got these things in place. When the load drops below 1, service chkservd start, then service cpanel start , service http start, service exim start.

Another is whatpain, for instance whatpain 80 grabs the output of netstat -n , parses the output and tells me how many connections there are open on port 80.

Or whatpain 25, to see who is poking the SMTP port and how many threads.

Or whatpain 110 for POP3

Then another script like blockip X.Y.Z.M b to block the ip or the network.

Also, another script scanmailog IP that checks the tails of all the /var/log/exim_ and /var/log/maillog files to tell me what this IP is doing with exim or cppop.

Maybe it should be a single script doing the whole thing.

Sometimes, when I just don't feel like dealing with attacks, I just block the whole Southeast Asia region and the server, whatever problem, cools off.

Most likely, now, serious, if exim causes an overload it is either a customer of yours sending thousands of heavy emails, or some intruder attempting a brute force attack on Exim to see what it accepts. Try tail -f /var/log/exim_mainlog or tail -f /var/log/exim_rejectlog and you'll see.

Then, serious too, what we did was to get twice the number of servers and twice the power per server, and systems became less sensitive to attacks. They do spike every now and then though, but much less often.

Now, my question, is there a formal tool I can use to replace the script zoo I got?

ER

[junglecat]

I have several simple Perl scripts, since I haven't found a better solution.

One is cpumonster, that goes in the crontab and checks /proc/loadavg every minute.

Where does one find cpumonster?

[eurorocco]

This script stops http, exim and cpanel, and chkservd if load average exceeds 8.

It stops http if load exceeds 6 but is less than 8.

If it stops anything it checks every 10 seconds to see how load drops until load drops below 3, then it restarts whatever it stopped.

It is a bit drastic, but has saved us from losing control of servers when a nasty script takes over or someone from the outside is pounding on the server. Our idea was to drop services before losing server control.

Place this in /usr/local/sbin and then add a line into the crontab (ie crontab -e to edit crontab).

*/3 * * * * /usr/local/sbin/cpumonster >>/var/log/cpumonster.log 2>&1

This is what I use. Use at your own risk. Improvements are welcome.

#!/usr/local/bin/perl

$deathlimit = 8.0;
$deathstop = 0;

$hardlimit = 6.0;
$softlimit = 3.0;
$timespan = 10;

$debug = 0;
$quiet = 0;

$load = getload();

if($load >= $hardlimit){
    do{
        track("load $load too hight!\n");
        track("stopping apache!\n");
        hookup("/sbin/service httpd stop");
        hookup("/sbin/service httpd stop");
        hookup("/sbin/service httpd stop");

        if($load >= $deathlimit){
            hookup("/sbin/service exim stop");
            hookup("/sbin/service cpanel stop");
            hookup("/sbin/service ftp stop");
            hookup("/sbin/service chkservd stop");

            track("deathstop stop\n");

            $deathstop = 1;
        }

        track("sleep $timespan seconds...\n");
        sleep($timespan);

        $load = getload();
    }while($load >= $softlimit);

    track("load $load is okay!\n");
    track("starting apache!\n");
    hookup("/sbin/service httpd start");

    if($deathstop){
        track("deathstop start\n");
        hookup("/sbin/service exim start");
        hookup("/sbin/service cpanel start");
        hookup("/sbin/service ftp start");
        hookup("/sbin/service chkservd start");
    }

    exit(0);
}
else{
    track("load $load is okay!\n");
    exit(0);
}

sub getload{

    my $what = `cat /proc/loadavg`;
    chomp($what);


    my @cells = split(/\s+/,$what);
    chomp(@cells);

    return($cells[0]);
}

sub track{

    my $what = $_[0];
    my $stamp = gettimestamp(time);

    $quiet == 0 || return;
    print "$stamp $what";
}

sub hookup{

    my $cmd = $_[0];

    track("$cmd\n");

    if($debug == 0){
        system($cmd);
    }
}

sub gettimestamp{

    my ($sec,$min,$hour,$mday,
        $mon,$year,$wday,$yday,$isdst) =
        localtime($_[0]);
    return(sprintf("%04d-%02d-%02d %02d:%02d:%02d ",
         1900+$year,$mon+1,$mday,$hour,$min,$sec));

}

Hope it helps!

ER

[iconraul]

Hello,

I'm wondering. How long will it take for the e-mail to stay in the mail queue (cPanel -> Email -> Mail Queue)? Is there a way to control this?

The situation is, i have one user whose account is being bombarded by newsletters and spam. Yesterday, I got chirpy's MailScanner working and also the Exim Dictionary attach ACL (also from chirphy).. but today, I found 100+ e-mails addressed to the same person. I've cheched his e-mail stats and he's like using 100% of the allowed qouta.


Some of the e-mails were from clamav(mailscanner) saying that the attachment is not acceptable... how can I disable this e-mail informing the user of any virus or unacceptable attachments? because I think it also contributes to the Mail queue being clogged by LOTS of e-mails.


Thanks! Thanks!



More power to everyone!

jun

[eurorocco]

Responding to the previous post, in Service Configuration > Exim Configuration Editor > Advanced , at the bottom of that page, between the last textboxes you can see something like

* * F,2h,15m; G,16h,1h,1.5; F,4d,8h

This is the line that controls number of retries, and the lifespan of a message in the queue. Cpane does not let you modify it, but you can manually edit /etc/exim.conf.

Unless you are some kind of Exim guru I recommend that you don't play with these.

Consider the Anti Spam and Virus solution in rvskin.com, which is part of what we use.

I struggle with viruses and spam using Cpanel. I wish it were more robust out of the box, but it isn't. I guess it is better than other systems I have seen, but it has a long way to go.

ER

[beehave]

I find cpumonster very helpfull. The following amended code checks to make sure script is not already running. Hope it helps...

#!/usr/local/bin/perl

$deathlimit = 8.0;
$deathstop = 0;

$hardlimit = 6.0;
$softlimit = 3.0;
$timespan = 10;

$debug = 0;
$quiet = 0;

$load = getload();

# Checking to see if we are already running
$pid = `/sbin/pidof -s cpumonster`;
if($pid >= 1) {
    print "CPUMonster is already running!\n";
    exit[0];
} else {

if($load >= $hardlimit){
    do{
        track("load $load too hight!\n");
        track("stopping apache!\n");
        hookup("/sbin/service httpd stop");
        hookup("/sbin/service httpd stop");
        hookup("/sbin/service httpd stop");

        if($load >= $deathlimit){
            hookup("/sbin/service exim stop");
            hookup("/sbin/service cpanel stop");
            hookup("/sbin/service ftp stop");
            hookup("/sbin/service chkservd stop");

            track("deathstop stop\n");

            $deathstop = 1;
        }

        track("sleep $timespan seconds...\n");
        sleep($timespan);

        $load = getload();
    }while($load >= $softlimit);

    track("load $load is okay!\n");
    track("starting apache!\n");
    hookup("/sbin/service httpd start");

    if($deathstop){
        track("deathstop start\n");
        hookup("/sbin/service exim start");
        hookup("/sbin/service cpanel start");
        hookup("/sbin/service ftp start");
        hookup("/sbin/service chkservd start");
    }

    exit(0);
}
else{
    track("load $load is okay!\n");
    exit(0);
}

sub getload{

    my $what = `cat /proc/loadavg`;
    chomp($what);


    my @cells = split(/\s+/,$what);
    chomp(@cells);

    return($cells[0]);
}

sub track{

    my $what = $_[0];
    my $stamp = gettimestamp(time);

    $quiet == 0 || return;
    print "$stamp $what";
}

sub hookup{

    my $cmd = $_[0];

    track("$cmd\n");

    if($debug == 0){
        system($cmd);
    }
}

sub gettimestamp{

    my ($sec,$min,$hour,$mday,
        $mon,$year,$wday,$yday,$isdst) =
        localtime($_[0]);
    return(sprintf("%04d-%02d-%02d %02d:%02d:%02d ",
         1900+$year,$mon+1,$mday,$hour,$min,$sec));

}}