The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

My Mail Archiving Solution - Care to Validate?

Discussion in 'E-mail Discussions' started by rocksolidhq, Nov 4, 2011.

  1. rocksolidhq

    rocksolidhq Active Member

    Joined:
    Sep 13, 2011
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    DataCenter Provider
    Hey,

    So i've come up with a way to create a real-time archive of all incoming and outgoing messages stored in such a way that they are available to the users via webmail. It seems to be working with only a few minor issues.

    The end result of my config is a pair of sub-folders (Incoming & Outgoing) available in webmail with a parent folder named Archive which contain the last 30 days of mail appropriately seperated.

    I'm not positive that there aren't holes and bad ideas here so feel free to make corrections/suggestions for improvement.

    So the first step is creating custom routers in exim.conf. As cPanel overwrites exim.conf don't edit it directly. You _can_ edit it through the WHM Exim Configuration Editor's Advanced Editor. Find the text block after "ROUTERS CONFIGURATION" and add this:

    Code:
    # Dean's Archiver
    incoming_archiver:
    unseen
    no_expn
    no_verify
    transport = incoming_local_copy
    driver = accept
    domains = +local_domains
    
    outgoing_archiver:
    unseen
    no_expn
    no_verify
    transport = outgoing_local_copy
    driver = accept
    domains = ! +local_domains 
    
    Now that we have additional routes for out mail we need to set up the transports that are on the other end of these routes so that they can process it. Go down to the section named "TRANSPORTS CONFIGURATION" and add the following to the text block:

    Code:
    # Dean's Archiver
    incoming_local_copy:
      driver = appendfile
      delivery_date_add
      envelope_to_add
      directory = "${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${domain}/${local_part}/.Archive.Incoming"
      maildir_use_size_file
      maildir_quota_directory_regex = ^(?:cur|new|\.(?!Trash$)[^\@]+)$
      maildir_format
      maildir_tag = ,S=$message_size
      quota_size_regex = ,S=(\d+)
      mode = 0660
      quota = "${if exists{${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${domain}/quota} {${lookup{$local_part}lsearch*{${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${domain}/quota}{$value}}} {}}"
      quota_is_inclusive = false
      quota_directory = "${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${domain}/${local_part}"
      return_path_add
      user = "${lookup{$domain}lsearch* {/etc/userdomains}{$value}}"
      group = ${extract{3}{:}{${lookup passwd{${lookup{$domain}lsearch* {/etc/userdomains}{$value}}}{$value}}}}
    #  shadow_condition = ${if exists {${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/.cpanel/rim/bis/$local_part@$domain}{1}{0}}
    #  shadow_transport = rim_bis_notifier_virtual_user
    
    outgoing_local_copy:
      driver = appendfile
      delivery_date_add
      envelope_to_add
      directory = "${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${sender_address_domain}/${sender_address_local_part}/.Archive.Outgoing"
      maildir_use_size_file
      maildir_quota_directory_regex = ^(?:cur|new|\.(?!Trash$)[^\@]+)$
      maildir_format
      maildir_tag = ,S=$message_size
      quota_size_regex = ,S=(\d+)
      mode = 0660
      quota = "${if exists{${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${sender_address_domain}/quota} {${lookup{$sender_address_local_part}lsearch*{${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${sender_address_domain}/quota}{$value}}} {}}"
      quota_is_inclusive = false
      quota_directory = "${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${sender_address_domain}/${sender_address_local_part}"
      return_path_add
    #  user = "${lookup{$domain}lsearch* {/etc/userdomains}{$value}}"
       user = "${extract{2}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}"
       group = "${extract{2}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}"
    #  group = ${extract{3}{:}{${lookup passwd{${lookup{$domain}lsearch* {/etc/userdomains}{$value}}}{$value}}}}
    #  shadow_condition = ${if exists {${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/.cpanel/rim/bis/$local_part@$domain}{1}{0}}
    #  shadow_transport = rim_bis_notifier_virtual_user
    
    This is the area that i could use some help with.

    First, i don't really understand the shadow stuff. I'm guessing that it has something to do with the Blackberry Fastmail Service which i haven't played with yet. If i read this correctly, i'd say that this is just CC'ing messages to the Blackberry. For that reason these lines should be commented out (or just removed) to avoid duplication. Anyone care to corroborate?

    Second, i see in the exim logs that it tries to process messages from system accounts (such as root) and errors out because the folder structure doesn't exist. This doesn't cause me any real problems but it would be nice to be able to omit that kind of mail to clean up the logs and reduce the overhead.

    Lastly, i have a cron job that runs each night that removes the messages that are older than 30 days. If you need this for legal reasons you may want to adjust/ignore this part. Be sure that there is space on the drive that you're storing your mail on.

    Code:
    find /home/*/mail/*/*/.Archive* -type f -mtime +30 -delete
    
    Your mileage may vary using this but i'd love some feedback and suggestions for improvement.

    later,
    dean
     
  2. rocksolidhq

    rocksolidhq Active Member

    Joined:
    Sep 13, 2011
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    DataCenter Provider
    Hey,

    Two adjustments are necessary. The incoming routers above do not account for forwarders and generate a mountain of NDR's when trying to save messages directed at the forwarded addresses. E.g. a message is sent to bobthefrog@domain.com which is a forwarder for bob@domain.com the message will be delivered to bob@domain.com and properly archived in /home/domain/mail/domain.com/bob/.Archive.Incoming but Exim will also try to store the message in the directory /home/domain/mail/domain.com/bobthefrog/.Archive.Incoming which does not exist and shouldn't.

    The way to correct this is to change the incoming router to include the a check to ensure that the alias is a user ala:

    Code:
    incoming_archiver:
    unseen
    no_expn
    no_verify
    transport = incoming_local_copy
    driver = accept
    domains = +local_domains
    local_parts = lsearch;${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/$domain/passwd
    
    On the ougoing side the router is mistakenly limited to only capture messages that are destined for addresses that are not local thanks to the line:

    Code:
    domains = ! +local_domains
    
    This results in any inter-domain and likely any inter-server messages not being archived. i don't have a solution to this yet but i'm working on it.

    later,
    dean
     
  3. rocksolidhq

    rocksolidhq Active Member

    Joined:
    Sep 13, 2011
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    DataCenter Provider
    nevermind. thought i had a solution but still missing something. :(
     
    #3 rocksolidhq, Nov 10, 2011
    Last edited: Nov 10, 2011
  4. rocksolidhq

    rocksolidhq Active Member

    Joined:
    Sep 13, 2011
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    DataCenter Provider
    Hey,

    Gee, i thought that this topic might generate a bit more traffic. I had searched for archiving and found alot of people trying to find a way to do it but not succeeding but the lack of replies on this thread suggest that i may be wrong. :(

    Regardless, i think that this is a valueable feature so hopefully someone can benefit from my pain.

    As far as i can see there is only one minor issue that i cannot seem to resolve. Messages sent from localhost (so webmail) to an alias on localhost are duplicated in the Outgoing Archive folder. I tried everything that i can think of without success. The closest i got was using a headers_add directive to add a header to the incoming, archived message and then a matching condition in the outgoing router to ignore messages with the header but it turns out the headers_add can't be used in routers that are set as unseen. i tried a bunch of other things but am out of time and this is a pretty minor issue that won't cause me alot of grief.

    That said, if anyone can show me a way to eliminate this last problem i'd be grateful. I'm also not certain that this is the most efficient solution so if someone that is more exim savvy can improve it, please post here.

    This config is the result of countless hours of screwing around and researching exim (which i knew nothing about) over the course of two weeks so even if you just benefit from this, let me know by replying. It's good for my Id and your chi.

    Code:
    # Dean's Archiver Router
    outgoing_archiver:
    #condition = ${if !def:h_X-Spam-Flag:}
    condition = ${if !eq{$sender_address_local_part}{root}}
    domains = ! +local_domains
    debug_print = true
    unseen
    no_expn
    verify
    transport = outgoing_local_copy
    driver = accept
    
    incoming_archiver:
    #headers_add = "Archive_Copy: YESr"
    condition = ${if !def:h_Archive_Copy:}
    no_expn
    no_verify
    unseen
    domains = +local_domains
    local_parts = lsearch;${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/$domain/passwd
    transport = incoming_local_copy
    driver = accept
    


    Code:
    # Dean's Archiver Transport
    incoming_local_copy:
      driver = appendfile
      delivery_date_add
      envelope_to_add
      directory = "${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${domain}/${local_part}/.Archive.Incoming"
      maildir_use_size_file
      maildir_quota_directory_regex = ^(?:cur|new|\.(?!Trash$)[^\@]+)$
      maildir_format
      maildir_tag = ,S=$message_size
      quota_size_regex = ,S=(\d+)
      mode = 0660
      quota = "${if exists{${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${domain}/quota} {${lookup{$local_part}lsearch*{${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${domain}/quota}{$value}}} {}}"
      quota_is_inclusive = false
      quota_directory = "${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${domain}/${local_part}"
      return_path_add
      user = "${lookup{$domain}lsearch* {/etc/userdomains}{$value}}"
      group = ${extract{3}{:}{${lookup passwd{${lookup{$domain}lsearch* {/etc/userdomains}{$value}}}{$value}}}}
      headers_add = "Archive_Copy: YESt"
    
    
    outgoing_local_copy:
      driver = appendfile
      delivery_date_add
      envelope_to_add
      directory = "${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${sender_address_domain}/${sender_address_local_part}/.Archive.Outgoing"
      maildir_use_size_file
      maildir_quota_directory_regex = ^(?:cur|new|\.(?!Trash$)[^\@]+)$
      maildir_format
      maildir_tag = ,S=$message_size
      quota_size_regex = ,S=(\d+)
      mode = 0660
      quota = "${if exists{${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${sender_address_domain}/quota} {${lookup{$sender_address_local_part}lsearch*{${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${sender_address_domain}/quota}{$value}}} {}}"
      quota_is_inclusive = false
      quota_directory = "${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${sender_address_domain}/${sender_address_local_part}"
      return_path_add
       user = "${extract{2}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}"
       group = "${extract{2}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}"
    
     
  5. rocksolidhq

    rocksolidhq Active Member

    Joined:
    Sep 13, 2011
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    DataCenter Provider
    Hey,

    So my last iteration generated NDR's and exim log errors if messages were received for non-existent email accounts/aliases. While this is no big deal, i hate having errors in the logs.

    Below are the updated routers/transports. Now the only deficiency is that duplication that occurs if a messages is sent from a local account to a local alias. I've looked in to resolving this but the only way i can see to do it is to set up a cache of message ID's that can be used to compare messages as they pass through the router. As much fun as that would be, i just don't have time given the severity of the problem. It's been 10yrs since i wrote a line of PERL and that, in combination with memcache, seems to be the best way to accomplish it. Perhaps someone else could take a kick at it and complete this solution??

    i'm actually also unsure of what impact this has on quotas but as they are disabled on our system, i just don't care about this either.

    later,
    dean


    Code:
    ROUTERS
    # Dean's Archiver Router
    outgoing_archiver:
    # works but duplicates messages sent from local to a local alias
    condition = ${if and {{exists{${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/$sender_address_domain/$sender_address_local_part}}{!eq{$sender_address_local_part}{root}}{!def:h_Archive_Copy:} }  }
    debug_print = true
    unseen
    no_expn
    verify
    transport = outgoing_local_copy
    driver = accept
    
    incoming_archiver:
    require_files = "+${extract{5}{::}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/$domain/passwd"
    condition = ${lookup {$local_part} lsearch{${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/$domain/passwd}{true}{false}}
    no_expn
    no_verify
    unseen
    domains = +local_domains
    transport = incoming_local_copy
    driver = accept
    
    TRANSPORTS
    # Dean's Archiver Transport
    incoming_local_copy:
      driver = appendfile
      delivery_date_add
      envelope_to_add
      directory = "${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${domain}/${local_part}/.Archive.Incoming"
      maildir_use_size_file
      maildir_quota_directory_regex = ^(?:cur|new|\.(?!Trash$)[^\@]+)$
      maildir_format
      maildir_tag = ,S=$message_size
      quota_size_regex = ,S=(\d+)
      mode = 0660
      quota = "${if exists{${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${domain}/quota} {${lookup{$local_part}lsearch*{${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${domain}/quota}{$value}}} {}}"
      quota_is_inclusive = false
      quota_directory = "${extract{5}{:}{${lookup passwd{${lookup{$domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${domain}/${local_part}"
      return_path_add
      user = "${lookup{$domain}lsearch* {/etc/userdomains}{$value}}"
      group = ${extract{3}{:}{${lookup passwd{${lookup{$domain}lsearch* {/etc/userdomains}{$value}}}{$value}}}}
    
    
    outgoing_local_copy:
      driver = appendfile
      delivery_date_add
      envelope_to_add
      directory = "${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${sender_address_domain}/${sender_address_local_part}/.Archive.Outgoing"
      maildir_use_size_file
      maildir_quota_directory_regex = ^(?:cur|new|\.(?!Trash$)[^\@]+)$
      maildir_format
      maildir_tag = ,S=$message_size
      quota_size_regex = ,S=(\d+)
      mode = 0660
      quota = "${if exists{${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${sender_address_domain}/quota} {${lookup{$sender_address_local_part}lsearch*{${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/etc/${sender_address_domain}/quota}{$value}}} {}}"
      quota_is_inclusive = false
      quota_directory = "${extract{5}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}/mail/${sender_address_domain}/${sender_address_local_part}"
      return_path_add
       user = "${extract{2}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}"
       group = "${extract{2}{:}{${lookup passwd{${lookup{$sender_address_domain}lsearch*{/etc/userdomains}{$value}}}{$value}}}}"
     
    #5 rocksolidhq, Dec 3, 2011
    Last edited: Dec 3, 2011
  6. joehanz30

    joehanz30 Registered

    Joined:
    Jan 9, 2009
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    I just wanna congratulate rocksolidhq as able to come out with this kind of method solution. I have tried it personally and it works like a charm! :)

    BTW, I just wondering if you still continue on this method enhancement? I would like to know if you are able to help me and others on below issue:

    a. Get the BCC recipient list being displayed in the log. I do believe that we need to tweak the exim itself but I not tried yet.
    b. Instead of the archiving email being copied to a folder, can it being configure to sent to an email account? Below URL have some kind of the solutions that able to send the copy to an email. Maybe we can fine tune the current code with that code. :)

    - http://forums.cpanel.net/f43/hidden-copies-incoming-outgoing-emails-201801.html
    - /http://www.unixsurgeon.com/kb/how-to-monitor-all-outgoing-emails-in-exim.html
     
  7. rocksolidhq

    rocksolidhq Active Member

    Joined:
    Sep 13, 2011
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    DataCenter Provider
    Thanks joehanz30,

    We've been pretty happy with it as well.

    As for your changes, you _won't_ be able to get the BCC list for incoming messages that's pretty much guaranteed but the outbound messages...that should be possible though i see that this does not happen with the current ruleset. Exim logs the outbound messages but i suspect they are not hitting the router.

    It would certainly be possible to send messages to an e-mail address. There were plenty of solutions out there for that when i was working on this. My requirements diverged in that i wanted the messages stored in the users' mailbox for easy access. If you want to e-mail the the archive copy are you planning on e-mailing each users' mail to individual mail addresses or is it one address that will be used for collection?

    Let me know and i'll try to help out.

    thanks,
    dean
     
Loading...

Share This Page