The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

My Exim greylisting solution

Discussion in 'E-mail Discussions' started by tprice42, Oct 31, 2007.

  1. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    Hi all, i thought i would share my implementation of exim greylisting for cpanel. What it essentially does is initially defer all received messages to and from any unique sender / recipient combination for 15 minutes, if the sender re-tries after that the message is accepted and any further messages are accepted immediately for a period of 36 days. Whenever any further messages are received the greylist also refreshes that 36 day timeout so regular correspondence between two parties should remain unmolested after the first 15 minute greylisting.

    Instructions are as follows and if anyone has any questions, bugs or other concerns please respond to this thread.

    • Firstly create a mysql database called 'greylist';
    • Create a mysql user and password with access to that database from the server(s) you're going to be running your exim services (i used username 'greylist' password 'greylist').
    • Create the tables in database as follows:

      Code:
      CREATE TABLE `relaytofrom` (
        `id` bigint(20) NOT NULL auto_increment,
        `relay_ip` varchar(16) default NULL,
        `mail_from` varchar(255) default NULL,
        `rcpt_to` varchar(255) default NULL,
        `block_expires` datetime NOT NULL default '0000-00-00 00:00:00',
        `record_expires` datetime NOT NULL default '0000-00-00 00:00:00',
        `blocked_count` bigint(20) NOT NULL default '0',
        `passed_count` bigint(20) NOT NULL default '0',
        `aborted_count` bigint(20) NOT NULL default '0',
        `origin_type` enum('MANUAL','AUTO') NOT NULL default 'MANUAL',
        `create_time` datetime NOT NULL default '0000-00-00 00:00:00',
        `last_update` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
        PRIMARY KEY  (`id`),
        KEY `rcpt_to` (`rcpt_to`),
        KEY `mail_from` (`mail_from`),
        KEY `relay_ip` (`relay_ip`)
      ) ENGINE=InnoDB DEFAULT CHARSET=latin1
      
      CREATE TABLE `spam_records` (
        `relay_ip` varchar(16) default NULL,
        `mail_from` varchar(255) default NULL,
        `rcpt_to` varchar(255) default NULL,
        `blocked_count` bigint(20) NOT NULL default '0',
        `passed_count` bigint(20) NOT NULL default '0',
        `last_update` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP
      ) ENGINE=MyISAM DEFAULT CHARSET=latin1
    • Copy your '/etc/exim.pl' to /etc/exim_greylist.pl
    • Edit your exim_greylist.pl file and look for this section:

      Code:
      my $hasmd5 = 0;
      eval {
              require Digest::Perl::MD5;
              $hasmd5 = 1;
      };
      This should be near to the top of the file, insert this under it changing the mysql connection variables as required:

      Code:
      use DBI;
      
      sub greylist() {
      my $lp = Exim::expand_string('$local_part');
      my $sender_addr = Exim::expand_string('$sender_address');
      my $host = Exim::expand_string('$sender_host_address');
      
      # put your mysql connect options here
      my $mydb = "greylist";
      my $myhost = "localhost";
      my $myuser = "greylist";
      my $mypasswd = "greylist";
      
      if(checkrelayhost($host)==1) {
              return "yes";
      } else {
      
      my $date = localtime;
      my $LOGFILE = "/var/log/exim_greylist";
      
      open(FILE, ">>$LOGFILE");
      
      my $isp= DBI->connect("DBI:mysql:$mydb:$myhost", "$myuser", "$mypasswd") || print FILE "$date Can't connect\n";
      $lp =~ s/\'/\\\'/g;
      $sender_addr =~ s/\'/\\\'/g;
      $host =~ s/\'/\\\'/g;
      
      my $query = "select UNIX_TIMESTAMP(block_expires)-UNIX_TIMESTAMP() from greylist.relaytofrom where rcpt_to='$lp' and mail_from='$sender_addr'";
      $sth = $isp->prepare($query)
        || print FILE "$query\n";
      $sth->execute || print FILE "$query\n";
      my @status_array = $sth->fetchrow_array;
      $sth->finish;
         
      unless (@status_array) {
              $query = "insert into greylist.relaytofrom values (NULL, '$host', '$sender_addr', '$lp', DATE_ADD(NOW(), INTERVAL 15 MINUTE), DATE_ADD(NOW(), INTERVAL 36 DAY), 1, 0, 0, 'AUTO', NOW(), NULL)";
              $sth = $isp->prepare($query)
              || print FILE "$query\n";
              $sth->execute || print FILE "$query [".$sth->errstr()."]\n";  
              close(FILE);
              $isp->disconnect;
              return 1;
      } else {
              if ($status_array[0]>0) {
                      $query = "update greylist.relaytofrom set blocked_count=blocked_count+1 where rcpt_to='$lp' and mail_from='$sender_addr' and relay_ip='$host'";
                      $sth = $isp->prepare($query)
                      || print FILE "$query\n";
                      $sth->execute || print FILE "$query\n";
                      close(FILE);
                      $isp->disconnect;
                      return 1;
              } else {
                      $query = "update greylist.relaytofrom set passed_count=passed_count+1,  record_expires=DATE_ADD(NOW(), INTERVAL 36 DAY) where rcpt_to='$lp' and mail_from='$sender_addr' and relay_ip='$host'";
                      $sth = $isp->prepare($query)
                      || print FILE "$query\n";
                      $sth->execute || print FILE "$query\n";
              }
      }
      
          close(FILE);
          $isp->disconnect;
      return "yes";
      }
      }
    • Create a file called /etc/clear_greylist and paste the following in:

      Code:
      #!/usr/bin/perl -w
      
      use strict;
      use DBI;
      
      my $date = localtime;
      my $LOGFILE = "/var/log/exim_greylist";
      my $i = 0;
      my $host = "";
      my $mail_from = "";
      my $rcpt_to = "";
      my $block = "";
      my $pass = "";
      
      # put your mysql connect options here
      my $mydb = "greylist";
      my $myhost = "localhost";
      my $myuser = "greylist";
      my $mypasswd = "greylist";
      
      
      open(FILE, ">>$LOGFILE");
      
      my $isp= DBI->connect("DBI:mysql:$mydb:$myhost", "$myuser", "$mypasswd") || print FILE "$date Can't connect\n";
      my $query = "select relay_ip, mail_from, rcpt_to, blocked_count, passed_count from greylist.relaytofrom where (passed_count=0 and (UNIX_TIMESTAMP() - UNIX_TIMESTAMP(block_expires)) > 28850) or (UNIX_TIMESTAMP() - UNIX_TIMESTAMP(record_e
      xpires) > 0) ";
      my $sth = $isp->prepare($query)
        || print FILE "$query - Can't prepare retrieving dates";
      $sth->execute || print FILE "$query - Can't execute retrieving dates";
      
      my $lines = $sth->fetchall_arrayref || &error("Can't fetch all period data");
      $sth->finish;
      
      
      foreach $i (0 .. $#{$lines}) {
      $host = $lines->[$i][0];
      $mail_from = $lines->[$i][1];
      $rcpt_to = $lines->[$i][2];
      $block = $lines->[$i][3];
      $pass = $lines->[$i][4];
      
      $host =~ s/\'/\\\'/g;
      $mail_from =~ s/\'/\\\'/g;
      $rcpt_to =~ s/\'/\\\'/g;
      $block =~ s/\'/\\\'/g;
      $pass =~ s/\'/\\\'/g;
      
      $query = "insert into greylist.spam_records values ('$host', '$mail_from', '$rcpt_to', $block, $pass, NULL)";
      $sth = $isp->prepare($query)
        || print FILE "$query - Can't prepare inserting triplet\n";
      $sth->execute || print FILE "$query - Can't execute inserting triplet [".$sth->errstr()."]\n";
      $query = "delete from greylist.relaytofrom where rcpt_to='$rcpt_to' and mail_from='$mail_from' and relay_ip='$host'";
      $sth = $isp->prepare($query)
        || print FILE "$query - Can't prepare retrieving dates";
      $sth->execute || print FILE "$query - Can't execute retrieving dates";
      }
      
      $isp->disconnect;
      close(FILE);
    • Save and exit and chmod 755 this file.
    • Enter something along these lines into your crontab to run this script at midnight every night:

      Code:
      0 0 * * * /etc/clear_greylist.pl
    • Go to WHM/Exim Configuration Editor/Advanced Editor and enter the following into the top box:

      Code:
      perl_startup = do '/etc/exim_greylist.pl'
    • Enter the following into your ACL's section under "#if it gets here it isn't mailman":

      Code:
      warn set acl_m0 = ${perl{greylist}}
      
      defer message = Greylisted for 15 minutes, please try again later
               log_message = Greylisted for 15 minutes, please try again later
               !hosts = +relay_hosts
               !authenticated = *
               condition = ${if eq{$acl_m0}{1}{1}}
    • Save and immediately check your /var/log/exim/mainlog or /var/log/exim_mainlog (system depending) and you should start to see this message pop up:

      Code:
      Greylisted for 15 minutes, please try again later
    • If anything breaks simply remove the ACL's section that you added and post a message in this thread and i'll see what i can do to help, i've been running it without problems for ~2 years now.
    • Final note is that the greylist time can be adjusted by editing the messages in the ACL's area and editing the following line in /etc/exim_greylist.pl

      Code:
              $query = "insert into greylist.relaytofrom values (NULL, '$host', '$sender_addr', '$lp', DATE_ADD(NOW(), INTERVAL 15 MINUTE), DATE_ADD(NOW(), INTERVAL 36 DAY), 1, 0, 0, 'AUTO', NOW(), NULL)";
     
    #1 tprice42, Oct 31, 2007
    Last edited: Oct 31, 2007
  2. DavidR

    DavidR Well-Known Member

    Joined:
    Feb 25, 2003
    Messages:
    177
    Likes Received:
    0
    Trophy Points:
    16
    I already have a line at the top of my exim.conf:
    Code:
    perl_startup = do '/etc/exim.pl'
    Another perl based spam solution required that. How will adding your version of this line affect that? The other perl routine comes from this solution.

    Looks interesting, I want to try it :)
     
  3. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    cPanel implements /etc/exim.pl in it's default installation then overwrites it at every possible opportunity hence why i called the script something else and pointed the exim config there instead. Your implementation should still be to copy the exim.pl file to exim_greylist.pl, insert the function where specified and change the
    Code:
    perl_startup = do '/etc/exim.pl'
    to
    Code:
    perl_startup = do '/etc/exim_greylist.pl'
    This way both systems will work fine.
     
  4. big

    big Well-Known Member

    Joined:
    Aug 12, 2001
    Messages:
    224
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Earth
    what do you mean the sender re-tries in 15 minutes?

    this means emails won't go through unless the sender resend his message within 15 minutes buffer time so it can be delivered?

    or there is an automatic message that will be sent to the sender (like boxtrapper) notifies them to resend the message?
     
  5. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    Basically it will defer the message for 15 minutes, the sender can retry as many times as they like within those 15 minutes but the message will only be accepted after those 15 minutes are up. This means that most spam messages that try to deliver once, maybe twice to a receiving MTA will give up. But genuine sending MTAs will retry on a schedule for up to 7 days.
     
  6. nickp666

    nickp666 Well-Known Member

    Joined:
    Jan 28, 2005
    Messages:
    770
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    /dev/null
    greylisting - the easy way to piss your customers off and lose legitimate e-mail!

    how many of you guys actually adopt this method out of curiousity?

    Can I also add that 15 mins seems a bit short, Especially on cPanel hosts given that the default queue retry is an hour.....
     
  7. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    Greylisting combined with fake primary MX records is currently the most effective way of not only reducing spam but reducing the load of your MTA as the spamassassin daemon doesn't have to process those messages, stats don't lie:
    [​IMG]

    The default retry times for cPanel exim are as follows:

    Code:
    *                      *           F,2h,15m; G,16h,1h,1.5; F,4d,8h
    Which in plain english means that any defered message will be retried every 15 minutes for 2 hours, then with intervals starting at one hour and increasing by a factor of 1.5 up to 16 hours, then every 8 hours up to 4 days.

    This however is besides the point as what you are really relying on is the sending MTA's retry configuration, not your receiving MTA's retry configuration. The greylisting interval could be as low as a couple of minutes because all you are really trying to do is reject the first couple of attempts made by spam bots as they usually give up after that. Every genuine SMTP sending MTA should be compliant with RFC2821 and under section 4.5.4.1 of that RFC titled 'Sending Strategy' it specifies guidelines for retry intervals ranging from 30 minutes to 5 days.

    As far as annoying customers is concerned: As an ISP we have been using this method on our receiving MTA for the last two years and have yet to receive a complaint about our imposed filtering methods. We have received inquiries, sure, but all customers have happily accepted that there is a tradeoff to having a good spam filtering service.
     
  8. nickp666

    nickp666 Well-Known Member

    Joined:
    Jan 28, 2005
    Messages:
    770
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    /dev/null
    Then you are very lucky my friend, My customers simply would not let me get away with delaying their e-mail, regardless of the benefits of doing so.

    I have had mixed experiences with greylisting, getting the intervals properly staged so that mail comes in from all over the place (so many conflicting ISP's setups) is an absolute nightmare, as such I can only see greylisting as being worthwhile on MX's that are just there to queue and forward if the primary is down.

    Your findings are based on the assumption that the servers that are sending to you are RFC compliant, now I know they all SHOULD be, but I could stream off an endless list of servers that are not.
     
    #8 nickp666, Nov 29, 2007
    Last edited: Nov 29, 2007
  9. trevHCS

    trevHCS Well-Known Member

    Joined:
    Nov 1, 2003
    Messages:
    69
    Likes Received:
    1
    Trophy Points:
    8
    I've seen it work extremely well, but you do need to keep a close eye on it initially because people like AOL use server farms to send their e-mail and you often won't get the same recipient : senderIP combo for what is a legit message.

    Also I'm told it's not very compatible with the callback system as used often with Cpanel and others.

    As for the defer times - it's very handy for getting rid of the spambots running on peoples computers which don't understand "I'm busy, please wait" codes (can't remember the exact code) so won't retry. So as mentioned setting it very low will work, although it depends totally on the remote SMTP server as to when it retries. You get legit delays anyway if a server is down for example.

    Trev
     
  10. cinusik

    cinusik Active Member

    Joined:
    Jul 31, 2003
    Messages:
    31
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Great greylisting solution tprice42, thanks.
    It runs on half of my servers but on the second half I got this error in my exim_log:

    2007-12-03 17:42:22 H=corpmailer04.prod.mesa1.secureserver.net [68.178.232.202] F=<offers@godaddy.com> temporarily rejected RCPT <mktg@adv-media.net>: failed to expand ACL string "${perl{greylist}}": install_driver(mysql) failed: Can't locate DBD/mysql.pm in @INC (@INC contains: /usr/local/cpanel /usr/lib/perl5/5.8.5/i386-linux-thread-multi /usr/lib/perl5/5.8.5 /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl/5.8.4 /usr/lib/perl5/site_perl/5.8.3 /usr/lib/perl5/site_perl/5.8.2 /usr/lib/perl5/site_perl/5.8.1 /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl/5.8.4 /usr/lib/perl5/vendor_perl/5.8.3 /usr/lib/perl5/vendor_perl/5.8.2 /usr/lib/perl5/vendor_perl/5.8.1 /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl . /etc/exim/perl /usr/lib/perl5/5.8.1 /usr/lib/perl5/5.8.0) at (eval 4) line 3.


    I have the same config on my all servers RHEL4 + cpanel, DBD and DBI perl modules installed and looks the same on both - working and not working servers.
    Any idea?
     
    #10 cinusik, Dec 3, 2007
    Last edited: Dec 3, 2007
  11. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    It looks like you've been through a lots of perl upgrades since the server was installed and it's possible that your mysql.pm file has been left behind in one of your old versions. Try locating the file and make sure it's in one of those paths for a start at least.
     
  12. CaMer0n

    CaMer0n Well-Known Member

    Joined:
    Nov 8, 2004
    Messages:
    59
    Likes Received:
    0
    Trophy Points:
    6
  13. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    Not from my initial browse of their features page, no.
     
  14. norelidd

    norelidd Well-Known Member

    Joined:
    Jan 15, 2007
    Messages:
    173
    Likes Received:
    1
    Trophy Points:
    18
    ASSP does do greylisting, and much more intelligently, to boot. One of ASSP's most important features, IMO, is the auto-whitelist capability. Anyone that you send mail out to is whitelisted, and they will never be stopped by the spam filter (aside from virus scanning) again.

    This way you get the benefits of greylisting, while also not delaying whitelisted addresses (the people you send mail out to).

    Personally, I think greylisting is bunk. There are wayyy too many servers out there that will not retry mail properly for me to even consider greylisting. It's also such a hassle to explain to people that they're not going to get that message for another 1-24hrs, if the sending mta even tries anyways.
     
  15. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    Thats cool. I've never tried ASSP but if it works then thats the main thing eh? In my experience i find that all anti-spam techniques only work for a limited amount of time. They all launch with claims of low IO load time and greater efficiency than the last. But at the end of the day all techniques used these days will eventually be their undoing as spammers adapt.

    As well as adapting to what spammers are doing these days you also have to beer in mind how cpanel updates are going to affect any customizations that you have implemented and for me at least there have been no updates that have stuffed up my mail system in this configuration.

    This technique has worked well for me for the last 24 months and all i want to do is share it with the cpanel community as there seemed to be limited well documented information about greylisting in the forums.

    There may well be better ways to do this out there but thats besides the point, this is my method and i'm sharing it with the community at large. Use it if you want to, or not. I'm only interested in questions regarding this post, not whether something else might or might not do the same thing or do it better.
     
  16. xoviboy

    xoviboy Member

    Joined:
    Jul 28, 2006
    Messages:
    13
    Likes Received:
    0
    Trophy Points:
    1
    Hi !

    Is there a way to deliver any e-mail with the rest of the ACL if mysql is down.

    I get the following error:
    "...temporarily rejected RCPT <address@domain.ext>: failed to expand ACL string "${perl{greylist}}"...

    Thanks
     
  17. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    Thats not normal, if mysql is down it should give a more verbose error message than that. I'd look somewhere else like manually running your exim.pl file through perl so ensure your syntax is correct.
     
  18. xoviboy

    xoviboy Member

    Joined:
    Jul 28, 2006
    Messages:
    13
    Likes Received:
    0
    Trophy Points:
    1
    Sorry... I meant, if mysql is down, can exim continue with the rest of the acls ?

    Right now all e-mails are rejected if you run the solution with greylisting, and mysql is down...

    Thanks.
     
  19. tprice42

    tprice42 Active Member

    Joined:
    Sep 10, 2004
    Messages:
    37
    Likes Received:
    0
    Trophy Points:
    6
    You can accomplish this by changing the line

    Code:
    my $isp= DBI->connect("DBI:mysql:$mydb:$myhost", "$myuser", "$mypasswd") || print FILE "$date Can't connect\n";
    to

    Code:
    my $isp= DBI->connect("DBI:mysql:$mydb:$myhost", "$myuser", "$mypasswd") || return "yes";
     
  20. LasseTK

    LasseTK Active Member

    Joined:
    Apr 15, 2005
    Messages:
    33
    Likes Received:
    0
    Trophy Points:
    6
    I get the following error

    Code:
    failed to expand ACL string "${perl{greylist}}": error in perl_startup code: Can't load '/usr/lib/perl5/site_perl/5.8.8/x86_64-linux//auto/Clone/Clone.so' for module Clone: /usr/lib/perl5/site_perl/5.8.8/x86_64-linux//auto/Clone/Clone.so: undefined symbol: PL_stack_sp at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230
    Please help :confused:
     
Loading...

Share This Page