Community Forums
Connect with us on LinkedIn
+ Reply to Thread
Page 1 of 3 1 2 3 LastLast
Results 1 to 15 of 42
  1. #1
    Member
    Join Date
    Sep 2004
    Posts
    37

    Default My Exim greylisting solution

    Hi all, i thought i would share my implementation of exim greylisting for cpanel. What it essentially does is initially defer all received messages to and from any unique sender / recipient combination for 15 minutes, if the sender re-tries after that the message is accepted and any further messages are accepted immediately for a period of 36 days. Whenever any further messages are received the greylist also refreshes that 36 day timeout so regular correspondence between two parties should remain unmolested after the first 15 minute greylisting.

    Instructions are as follows and if anyone has any questions, bugs or other concerns please respond to this thread.
    • Firstly create a mysql database called 'greylist';
    • Create a mysql user and password with access to that database from the server(s) you're going to be running your exim services (i used username 'greylist' password 'greylist').
    • Create the tables in database as follows:

      Code:
      CREATE TABLE `relaytofrom` (
        `id` bigint(20) NOT NULL auto_increment,
        `relay_ip` varchar(16) default NULL,
        `mail_from` varchar(255) default NULL,
        `rcpt_to` varchar(255) default NULL,
        `block_expires` datetime NOT NULL default '0000-00-00 00:00:00',
        `record_expires` datetime NOT NULL default '0000-00-00 00:00:00',
        `blocked_count` bigint(20) NOT NULL default '0',
        `passed_count` bigint(20) NOT NULL default '0',
        `aborted_count` bigint(20) NOT NULL default '0',
        `origin_type` enum('MANUAL','AUTO') NOT NULL default 'MANUAL',
        `create_time` datetime NOT NULL default '0000-00-00 00:00:00',
        `last_update` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
        PRIMARY KEY  (`id`),
        KEY `rcpt_to` (`rcpt_to`),
        KEY `mail_from` (`mail_from`),
        KEY `relay_ip` (`relay_ip`)
      ) ENGINE=InnoDB DEFAULT CHARSET=latin1
      
      CREATE TABLE `spam_records` (
        `relay_ip` varchar(16) default NULL,
        `mail_from` varchar(255) default NULL,
        `rcpt_to` varchar(255) default NULL,
        `blocked_count` bigint(20) NOT NULL default '0',
        `passed_count` bigint(20) NOT NULL default '0',
        `last_update` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP
      ) ENGINE=MyISAM DEFAULT CHARSET=latin1
    • Copy your '/etc/exim.pl' to /etc/exim_greylist.pl
    • Edit your exim_greylist.pl file and look for this section:

      Code:
      my $hasmd5 = 0;
      eval {
              require Digest::Perl::MD5;
              $hasmd5 = 1;
      };
      This should be near to the top of the file, insert this under it changing the mysql connection variables as required:

      Code:
      use DBI;
      
      sub greylist() {
      my $lp = Exim::expand_string('$local_part');
      my $sender_addr = Exim::expand_string('$sender_address');
      my $host = Exim::expand_string('$sender_host_address');
      
      # put your mysql connect options here
      my $mydb = "greylist";
      my $myhost = "localhost";
      my $myuser = "greylist";
      my $mypasswd = "greylist";
      
      if(checkrelayhost($host)==1) {
              return "yes";
      } else {
      
      my $date = localtime;
      my $LOGFILE = "/var/log/exim_greylist";
      
      open(FILE, ">>$LOGFILE");
      
      my $isp= DBI->connect("DBI:mysql:$mydb:$myhost", "$myuser", "$mypasswd") || print FILE "$date Can't connect\n";
      $lp =~ s/\'/\\\'/g;
      $sender_addr =~ s/\'/\\\'/g;
      $host =~ s/\'/\\\'/g;
      
      my $query = "select UNIX_TIMESTAMP(block_expires)-UNIX_TIMESTAMP() from greylist.relaytofrom where rcpt_to='$lp' and mail_from='$sender_addr'";
      $sth = $isp->prepare($query)
        || print FILE "$query\n";
      $sth->execute || print FILE "$query\n";
      my @status_array = $sth->fetchrow_array;
      $sth->finish;
         
      unless (@status_array) {
              $query = "insert into greylist.relaytofrom values (NULL, '$host', '$sender_addr', '$lp', DATE_ADD(NOW(), INTERVAL 15 MINUTE), DATE_ADD(NOW(), INTERVAL 36 DAY), 1, 0, 0, 'AUTO', NOW(), NULL)";
              $sth = $isp->prepare($query)
              || print FILE "$query\n";
              $sth->execute || print FILE "$query [".$sth->errstr()."]\n";  
              close(FILE);
              $isp->disconnect;
              return 1;
      } else {
              if ($status_array[0]>0) {
                      $query = "update greylist.relaytofrom set blocked_count=blocked_count+1 where rcpt_to='$lp' and mail_from='$sender_addr' and relay_ip='$host'";
                      $sth = $isp->prepare($query)
                      || print FILE "$query\n";
                      $sth->execute || print FILE "$query\n";
                      close(FILE);
                      $isp->disconnect;
                      return 1;
              } else {
                      $query = "update greylist.relaytofrom set passed_count=passed_count+1,  record_expires=DATE_ADD(NOW(), INTERVAL 36 DAY) where rcpt_to='$lp' and mail_from='$sender_addr' and relay_ip='$host'";
                      $sth = $isp->prepare($query)
                      || print FILE "$query\n";
                      $sth->execute || print FILE "$query\n";
              }
      }
      
          close(FILE);
          $isp->disconnect;
      return "yes";
      }
      }
    • Create a file called /etc/clear_greylist and paste the following in:

      Code:
      #!/usr/bin/perl -w
      
      use strict;
      use DBI;
      
      my $date = localtime;
      my $LOGFILE = "/var/log/exim_greylist";
      my $i = 0;
      my $host = "";
      my $mail_from = "";
      my $rcpt_to = "";
      my $block = "";
      my $pass = "";
      
      # put your mysql connect options here
      my $mydb = "greylist";
      my $myhost = "localhost";
      my $myuser = "greylist";
      my $mypasswd = "greylist";
      
      
      open(FILE, ">>$LOGFILE");
      
      my $isp= DBI->connect("DBI:mysql:$mydb:$myhost", "$myuser", "$mypasswd") || print FILE "$date Can't connect\n";
      my $query = "select relay_ip, mail_from, rcpt_to, blocked_count, passed_count from greylist.relaytofrom where (passed_count=0 and (UNIX_TIMESTAMP() - UNIX_TIMESTAMP(block_expires)) > 28850) or (UNIX_TIMESTAMP() - UNIX_TIMESTAMP(record_e
      xpires) > 0) ";
      my $sth = $isp->prepare($query)
        || print FILE "$query - Can't prepare retrieving dates";
      $sth->execute || print FILE "$query - Can't execute retrieving dates";
      
      my $lines = $sth->fetchall_arrayref || &error("Can't fetch all period data");
      $sth->finish;
      
      
      foreach $i (0 .. $#{$lines}) {
      $host = $lines->[$i][0];
      $mail_from = $lines->[$i][1];
      $rcpt_to = $lines->[$i][2];
      $block = $lines->[$i][3];
      $pass = $lines->[$i][4];
      
      $host =~ s/\'/\\\'/g;
      $mail_from =~ s/\'/\\\'/g;
      $rcpt_to =~ s/\'/\\\'/g;
      $block =~ s/\'/\\\'/g;
      $pass =~ s/\'/\\\'/g;
      
      $query = "insert into greylist.spam_records values ('$host', '$mail_from', '$rcpt_to', $block, $pass, NULL)";
      $sth = $isp->prepare($query)
        || print FILE "$query - Can't prepare inserting triplet\n";
      $sth->execute || print FILE "$query - Can't execute inserting triplet [".$sth->errstr()."]\n";
      $query = "delete from greylist.relaytofrom where rcpt_to='$rcpt_to' and mail_from='$mail_from' and relay_ip='$host'";
      $sth = $isp->prepare($query)
        || print FILE "$query - Can't prepare retrieving dates";
      $sth->execute || print FILE "$query - Can't execute retrieving dates";
      }
      
      $isp->disconnect;
      close(FILE);
    • Save and exit and chmod 755 this file.
    • Enter something along these lines into your crontab to run this script at midnight every night:

      Code:
      0 0 * * * /etc/clear_greylist.pl
    • Go to WHM/Exim Configuration Editor/Advanced Editor and enter the following into the top box:

      Code:
      perl_startup = do '/etc/exim_greylist.pl'
    • Enter the following into your ACL's section under "#if it gets here it isn't mailman":

      Code:
      warn set acl_m0 = ${perl{greylist}}
      
      defer message = Greylisted for 15 minutes, please try again later
               log_message = Greylisted for 15 minutes, please try again later
               !hosts = +relay_hosts
               !authenticated = *
               condition = ${if eq{$acl_m0}{1}{1}}
    • Save and immediately check your /var/log/exim/mainlog or /var/log/exim_mainlog (system depending) and you should start to see this message pop up:

      Code:
      Greylisted for 15 minutes, please try again later
    • If anything breaks simply remove the ACL's section that you added and post a message in this thread and i'll see what i can do to help, i've been running it without problems for ~2 years now.
    • Final note is that the greylist time can be adjusted by editing the messages in the ACL's area and editing the following line in /etc/exim_greylist.pl

      Code:
              $query = "insert into greylist.relaytofrom values (NULL, '$host', '$sender_addr', '$lp', DATE_ADD(NOW(), INTERVAL 15 MINUTE), DATE_ADD(NOW(), INTERVAL 36 DAY), 1, 0, 0, 'AUTO', NOW(), NULL)";
    Last edited by tprice42; 10-31-2007 at 07:54 PM.

  2. #2
    Member
    Join Date
    Feb 2003
    Posts
    176

    Default

    I already have a line at the top of my exim.conf:
    Code:
    perl_startup = do '/etc/exim.pl'
    Another perl based spam solution required that. How will adding your version of this line affect that? The other perl routine comes from this solution.

    Looks interesting, I want to try it

  3. #3
    Member
    Join Date
    Sep 2004
    Posts
    37

    Default

    cPanel implements /etc/exim.pl in it's default installation then overwrites it at every possible opportunity hence why i called the script something else and pointed the exim config there instead. Your implementation should still be to copy the exim.pl file to exim_greylist.pl, insert the function where specified and change the
    Code:
    perl_startup = do '/etc/exim.pl'
    to
    Code:
    perl_startup = do '/etc/exim_greylist.pl'
    This way both systems will work fine.

  4. #4
    big
    big is offline
    Member
    Join Date
    Aug 2001
    Location
    Earth
    Posts
    232

    Default

    what do you mean the sender re-tries in 15 minutes?

    this means emails won't go through unless the sender resend his message within 15 minutes buffer time so it can be delivered?

    or there is an automatic message that will be sent to the sender (like boxtrapper) notifies them to resend the message?

  5. #5
    Member
    Join Date
    Sep 2004
    Posts
    37

    Default

    Basically it will defer the message for 15 minutes, the sender can retry as many times as they like within those 15 minutes but the message will only be accepted after those 15 minutes are up. This means that most spam messages that try to deliver once, maybe twice to a receiving MTA will give up. But genuine sending MTAs will retry on a schedule for up to 7 days.

  6. #6
    Member
    Join Date
    Jan 2005
    Location
    /dev/null
    Posts
    770

    Default

    greylisting - the easy way to piss your customers off and lose legitimate e-mail!

    how many of you guys actually adopt this method out of curiousity?

    Can I also add that 15 mins seems a bit short, Especially on cPanel hosts given that the default queue retry is an hour.....

  7. #7
    Member
    Join Date
    Sep 2004
    Posts
    37

    Default

    Greylisting combined with fake primary MX records is currently the most effective way of not only reducing spam but reducing the load of your MTA as the spamassassin daemon doesn't have to process those messages, stats don't lie:


    The default retry times for cPanel exim are as follows:

    Code:
    *                      *           F,2h,15m; G,16h,1h,1.5; F,4d,8h
    Which in plain english means that any defered message will be retried every 15 minutes for 2 hours, then with intervals starting at one hour and increasing by a factor of 1.5 up to 16 hours, then every 8 hours up to 4 days.

    This however is besides the point as what you are really relying on is the sending MTA's retry configuration, not your receiving MTA's retry configuration. The greylisting interval could be as low as a couple of minutes because all you are really trying to do is reject the first couple of attempts made by spam bots as they usually give up after that. Every genuine SMTP sending MTA should be compliant with RFC2821 and under section 4.5.4.1 of that RFC titled 'Sending Strategy' it specifies guidelines for retry intervals ranging from 30 minutes to 5 days.

    As far as annoying customers is concerned: As an ISP we have been using this method on our receiving MTA for the last two years and have yet to receive a complaint about our imposed filtering methods. We have received inquiries, sure, but all customers have happily accepted that there is a tradeoff to having a good spam filtering service.

  8. #8
    Member
    Join Date
    Jan 2005
    Location
    /dev/null
    Posts
    770

    Default

    Then you are very lucky my friend, My customers simply would not let me get away with delaying their e-mail, regardless of the benefits of doing so.

    I have had mixed experiences with greylisting, getting the intervals properly staged so that mail comes in from all over the place (so many conflicting ISP's setups) is an absolute nightmare, as such I can only see greylisting as being worthwhile on MX's that are just there to queue and forward if the primary is down.

    Your findings are based on the assumption that the servers that are sending to you are RFC compliant, now I know they all SHOULD be, but I could stream off an endless list of servers that are not.
    Last edited by nickp666; 11-29-2007 at 10:58 AM. Reason: typo

  9. #9
    Member
    Join Date
    Nov 2003
    Posts
    67

    Default

    I've seen it work extremely well, but you do need to keep a close eye on it initially because people like AOL use server farms to send their e-mail and you often won't get the same recipient : senderIP combo for what is a legit message.

    Also I'm told it's not very compatible with the callback system as used often with Cpanel and others.

    As for the defer times - it's very handy for getting rid of the spambots running on peoples computers which don't understand "I'm busy, please wait" codes (can't remember the exact code) so won't retry. So as mentioned setting it very low will work, although it depends totally on the remote SMTP server as to when it retries. You get legit delays anyway if a server is down for example.

    Trev

  10. #10
    Member
    Join Date
    Jul 2003
    Posts
    38

    Default

    Great greylisting solution tprice42, thanks.
    It runs on half of my servers but on the second half I got this error in my exim_log:

    2007-12-03 17:42:22 H=corpmailer04.prod.mesa1.secureserver.net [68.178.232.202] F=<offers@godaddy.com> temporarily rejected RCPT <mktg@adv-media.net>: failed to expand ACL string "${perl{greylist}}": install_driver(mysql) failed: Can't locate DBD/mysql.pm in @INC (@INC contains: /usr/local/cpanel /usr/lib/perl5/5.8.5/i386-linux-thread-multi /usr/lib/perl5/5.8.5 /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl/5.8.4 /usr/lib/perl5/site_perl/5.8.3 /usr/lib/perl5/site_perl/5.8.2 /usr/lib/perl5/site_perl/5.8.1 /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl/5.8.4 /usr/lib/perl5/vendor_perl/5.8.3 /usr/lib/perl5/vendor_perl/5.8.2 /usr/lib/perl5/vendor_perl/5.8.1 /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl . /etc/exim/perl /usr/lib/perl5/5.8.1 /usr/lib/perl5/5.8.0) at (eval 4) line 3.


    I have the same config on my all servers RHEL4 + cpanel, DBD and DBI perl modules installed and looks the same on both - working and not working servers.
    Any idea?
    Last edited by cinusik; 12-03-2007 at 05:05 PM.

  11. #11
    Member
    Join Date
    Sep 2004
    Posts
    37

    Default

    It looks like you've been through a lots of perl upgrades since the server was installed and it's possible that your mysql.pm file has been left behind in one of your old versions. Try locating the file and make sure it's in one of those paths for a start at least.

  12. #12
    Member
    Join Date
    Nov 2004
    Posts
    58

    Default

    Doesn't ASSP already do this?

  13. #13
    Member
    Join Date
    Sep 2004
    Posts
    37

    Default

    Not from my initial browse of their features page, no.

  14. #14
    Member
    Join Date
    Jan 2007
    Posts
    170

    Default

    ASSP does do greylisting, and much more intelligently, to boot. One of ASSP's most important features, IMO, is the auto-whitelist capability. Anyone that you send mail out to is whitelisted, and they will never be stopped by the spam filter (aside from virus scanning) again.

    This way you get the benefits of greylisting, while also not delaying whitelisted addresses (the people you send mail out to).

    Personally, I think greylisting is bunk. There are wayyy too many servers out there that will not retry mail properly for me to even consider greylisting. It's also such a hassle to explain to people that they're not going to get that message for another 1-24hrs, if the sending mta even tries anyways.

  15. #15
    Member
    Join Date
    Sep 2004
    Posts
    37

    Default

    Thats cool. I've never tried ASSP but if it works then thats the main thing eh? In my experience i find that all anti-spam techniques only work for a limited amount of time. They all launch with claims of low IO load time and greater efficiency than the last. But at the end of the day all techniques used these days will eventually be their undoing as spammers adapt.

    As well as adapting to what spammers are doing these days you also have to beer in mind how cpanel updates are going to affect any customizations that you have implemented and for me at least there have been no updates that have stuffed up my mail system in this configuration.

    This technique has worked well for me for the last 24 months and all i want to do is share it with the cpanel community as there seemed to be limited well documented information about greylisting in the forums.

    There may well be better ways to do this out there but thats besides the point, this is my method and i'm sharing it with the community at large. Use it if you want to, or not. I'm only interested in questions regarding this post, not whether something else might or might not do the same thing or do it better.

Similar Threads & Tags
Similar threads

  1. Replies: 4
    Last Post: 08-22-2011, 02:45 PM
  2. Exim queue runner and retry times - greylisting
    By internetfab in forum cPanel and WHM Discussions
    Replies: 0
    Last Post: 10-27-2008, 08:02 AM
  3. cPanel/Exim dealing with other servers that implement greylisting
    By ichthus in forum cPanel and WHM Discussions
    Replies: 6
    Last Post: 03-04-2006, 07:15 PM
  4. bsmtp solution for cpanel/exim server wanted asap
    By Domenico in forum cPanel and WHM Discussions
    Replies: 1
    Last Post: 11-09-2005, 12:10 PM
  5. Exim and/or mail filter for UNIQUE solution
    By oneguy in forum cPanel and WHM Discussions
    Replies: 1
    Last Post: 01-28-2005, 03:27 AM
Linkedin       Facebook       Twitter       RSS       Flickr       YouTube