JohnConover_medium.jpg 
john@email.johncon.com
http://www.johncon.com/john/

Securing E-Mail


Home | John | Connie | Publications | Software | Correspondence | NtropiX | NdustriX | NformatiX | NdeX | Thanks



home.jpg
john.jpg
connie.jpg
publications.jpg
software.jpg
correspondence.jpg
ntropix.jpg
ndustrix.jpg
nformatix.jpg
ndex.jpg
thanks.jpg

The following example procmail(1) ~/.procmailrc script utilizes the E-mail "Received: " Header IP Address Auditing database software for auditing "Received: " record headers of e-mail, the Quarantining Malicious Outlook Attachments procmail fragment to protect users from potentially malicious Microsoft Outlook® attachments, the Stochastic UCE Detection, and Confirmed Mail Delivery to minimize the spam seen by users and offers reasonable security, without being overly invasive, with complete audit trails of distribution of e-mail within an organization.

The script is offered as a template, or example, of what can be done to secure corporate e-mail using procmail-some knowledge of how procmail works will be required to adapt the script to a specific environment-pop-3, imap, etc..


Description and Walk Through of the Script

The word "theuser" should be substituted for the user's account name, and "thedomain.com" substituted for the user's fully qualified domain name. The boiler plate:


        MAILDIR="${HOME}/Mail"
        DEFAULT="theuser"
        # LOGFILE="${HOME}/PROCMAIL.LOG"
        # VERBOSE="on"
        # LOGABSTRACT="all"
        MSGIDCACHESIZE=262144

        

where the user's mail directory is ~/Mail, and the "Message-ID:" cache size for procmail's formail is 262144 bytes.


Macros, variables, header records, and architecture


        ws='[    ]*($[   ]+)*'

        

Folded whitespace, (the characters between the block braces are a tab character, hex 09, followed by a space character, hex 20.)


        dq='"'

        

Double quote, (to avoid problems caused by how the procmail shell expands conditions).


        ext = '(a(d[ep]|r[cj]|s[dmxp]|u|vi)|\
                b(a[st]|mp|z[0-9]?)|\
                c(an|hm|il|lass|md|om|(pp|\+\+)?|sv)|\
                d(at|e?b|ll|o[ct])|\
                e(ml|ps?|xe)|\
                g(if|z?)|\
                h(lp|t(a|ml?)|(pp|\+\+)?)|\
                in[ci]|\
                j(ava|pe?g|se?|sp|tmpl)|\
                kbf|\
                l(ha|nk|og|yx)|\
                m(d[abew]|p(e?g|[32])|s[ip])|\
                ocx|\
                p(a(tch|s)|c[sx]|df|h(p[0-9]?|tml?)|if|[lm?]\
                    |n[gm]|[po][st]|p?s)|\
                r(a[mr]|eg|pm|tf)|\
                s(c[rt]|h([bs]|tml?)|lp|ql|ys)?|\
                t(ar|ex|gz|iff?|xt)|\
                u(pd|x)|\
                vb[es]?|\
                w(av|m[szd]|p(d|[0-9]?)|s[cfh])|\
                x(al|[pb]m|l[stw])|\
                z(ip|oo))'

        

List (sorted and optimized), of Microsoft file name extensions that can be included as e-mail attachments that are potentially malicious executable code, and includes the following file name extensions:

.ade .adp .amp .arc .arj .asd .asm .asp .asx .au .avi .bas .bat .bz .bz0 .bz1 .bz2 .bz3 .bz4 .bz5 .bz6 .bz7 .bz8 .bz9 .c .c++ .can .chm .cil .class .cmd .com .cpp .csv .dat .db .deb .dll .doc .dot .eml .ep .eps .exe .g .gif .gz .h .h++ .hlp .hpp .hta .htm .html .inc .ini .java .jpeg .jpg .js .jse .jsp .jtmpl .kbf .lha .lnk .log .lyx .mda .mdb .mde .mdw .mpeg2 .mpg2 .mpg3 .msi .msp .ocx .os .ot .pas .patch .pcs .pcx .pdf .phtm .phtml .php .php0 .php1 .php2 .php3 .php4 .php5 .php6 .php7 .php8 .php9 .pif .pl .plm .png .pnm .pps .ps .pt .ram .rar .reg .rpm .rtf .s .scr .sct .shb .shs .shtm .shtml .slp .sql .sys .tar .tex .tgz .tif .tiff .txt .upd .ux .vb .vbe .vbs .wav .wmd .wms .wmz .wp .wp0 .wp1 .wp2 .wp3 .wp4 .wp5 .wp6 .wp7 .wp8 .wp9 .wpd .wsc .wsf .wsh .xal .xbm .xls .xlt .xlw .xpm .zip .zoo

Further details can be found at Quarantining Malicious Outlook Attachments.


Variables


        DAEMON=false
        KNOWN=true
        MALICIOUS=false
        SPAMSCORE="0"
        FROM=""
        SENDER=""
        DOMAIN=""
        CMDADDRESS=""
        CMDFQDN=""
        CMDUSER=""
        ENVELOPE_TO=""
        TEMP_TO=""
        BOUNCE=false
        REJECT=false
        FOLDER=""
        TRUSTED=false

        
  • If the message is from a known list, or a daemon, $DAEMON is set to true, otherwise, false.
  • If the message is from a known sender, (someone the user has corresponded with before,) $KNOWN is set to true, otherwise, false.
  • If the message contains potentially malicious content in attachment(s), $MALICIOUS is set to true, otherwise, false.
  • The variable $SPAMSCORE contains score value of the message; a high positive value means the message is probably spam, a low positive, (or negative,) value means it probably isn't.
  • $FROM contains the trusted return address of the message.
  • $SENDER contains the machine generated return address of the sender.
  • $DOMAIN contains the domain name of the return address of the sender.
  • $CMDADDRESS contains the address of the user when using Qmail's qmail-local addressing scheme for confirmation replies.
  • $CMDFQDN contains the fully qualified domain name of the user when using Qmail's qmail-local addressing scheme for confirmation replies.
  • $CMDUSER contains the user name of the user when using Qmail's qmail-local addressing scheme for confirmation replies.
  • $ENVELOPE_TO contains the sendmail envelope-to address.
  • $TEMP_TO is temporary storage for a matching envelope-to address.
  • If the message is to be unconditionally bounced for approval, $BOUNCE is set to true, otherwise, false.
  • If the message is to be unconditionally rejected, $REJECT is set to true, otherwise, false.
  • If the message is to be filed in a folder, $FOLDER contains the folder's filename, otherwise, the null string, "".
  • If the sender is trusted, $TRUSTED is set to true, otherwise, false.

Header records

The following header records can be inserted into a message.

  • X-Loop: user@fqdn, the address of the recipient.
  • Approved: user@fqdn, where user@fqdn approved the message for delivery.
  • X-Approved: user@fqdn, renamed from Approved:.
  • Approval: bounce to user@fqdn, the address of the recipient.
  • X-Delivered-To: user@fqdn, renamed from Delivered-To:, but may contain the $ENVELOPE_TO address if it can be ascertained.
  • X-Delivered-Sender: user@fqdn, the machine generated address of the sender, $SENDER.
  • X-Delivered-From: user@fqdn, the trusted return address of the sender, $FROM.
  • X-Name: first last, the name of the recipient.
  • X-Audit-Log: Potentially malicious, if the message contains potentially malicious attachments.
  • X-Audit-Log: Message from/to a daemon, if the message machine generated, from a daemon, mailing list agent, etc.
  • X-Audit-Log: $SPAMSCORE, the spam score-positive if from an unknown sender, negative if from known.
  • X-Diagnostic: Message is looping, if a mail loop occurs.
  • X-Diagnostic: No machine generated return address, if the message has no machine generated return address.
  • X-Diagnostic: No domain name, if the message has no domain name in the return address.
  • X-Diagnostic: No trusted return address, if the message has no trusted return address.
  • X-Diagnostic: Malformed CMD address, if the CMD address is not recognized.

The following header records will be removed from a message if it is approved:

Resent-Date:
Resent-Message-ID:
Resent-From:
Resent-To:
Approved:
X-Approved:

Architecture

General; a message that contains an "X-Audit-Log: $SPAMSCORE" record has already been checked for both malicious attachments and spam content, (and a host of other things.) The addition of the "X-Audit-Log: $SPAMSCORE" record is the last operation before disposition of the message commences. No message may be delivered, (except under exception conditions, or rejected to /dev/null, or diverted to an agent,) without having an "X-Audit-Log: $SPAMSCORE" record. Disposition of the message can not include delivery unless the message is not spam, and does not contain malicious attachments, etc.

  • First, handle any "X-Diagnostic:" fatal errors in the message; mandatory abort if present-two options; bounce the message to a mail handler, (which may present looping issues,) or, file immediately.

  • Then, immediately, handle any looping issues, checking for header records containing "signatures" that a message is looping from an agent, or looking at a recurrence of the "X-Loop:" record. (Note: this can be complicated, since a message that has been approved is a valid loop.) Under exception, abort with an "X-Diagnostic:" record.

  • Handle any approvals, (mail handler only-the stream is diverted to a file, immediately.) This interacts with the handling of looping issues.

  • Immediately add an "X-Loop:" record.

  • Establish assorted variables, such as, FROM, SENDER, DOMAIN, etc. Under exception, add an "X-Diagnostic:" record, and abort.

  • Handle confirmation replies. A message from an unknown sender may have been diverted to an agent for delivery confirmation. If the message is a confirmation, divert the message back to the agent to send the original message to the user.

  • Mandatory for all messages that are not aborted, (with an "X-Diagnostic:" record,) approvals, (mail handler only,) rejected into /dev/null, or diverted confirmation messages-the message is verified as not having potentially malicious attachments. If the message has a "X-Audit-Log: SPAMSCORE" record, then it has already been checked for malicious attachments, don't do so again.

  • Handle messages from all known daemons, (known mailing list agents, machine generated messages, etc.,) not by filing them, (they may contain malicious attachments,) but by setting the filename of the folder in which they will be delivered, and setting the DAEMON variable true. A true DAEMON variable will prohibit automatically responding to daemons in the disposition of the message.

  • Query various databases to verify whether the sender of the message is known, or not.

  • Attempt to deduce the envelope-to e-mail address that the message was sent to, and if fails use the address in the e-mail headers; validate that this address exists by querying various databases-also, validate that the sender's address is not in the "kill" databases.

  • Construct the cumulative SPAMSCORE, followed, immediately, by disposition of the message; disposition means, that if necessary, do something with the message besides letting it fall through to delivery. If the message has an "X-Audit-Log: SPAMSCORE" record, then it has already been checked for its SPAMSCORE, don't do so again-simply fall through to delivery; a message with a pre-existing "X-Audit-Log: SPAMSCORE" has already had its disposition verified, and was not diverted from delivery.

  • Deliver the message.


Preamble

X-Diagnostic mandatory aborts, anti-looping mandatory aborts, and setting of variables: $FROM, (trusted return address,) $SENDER, (machine generated return address,) and $DOMAIN, (domain of return address,) of the sender of the message. All variables must be set, and in addition, if the corresponding "X-" header records do not exist, "Delivered-To:" (the recipient's address,) renamed to "X-Delivered-To:", "X-Delivered-Sender:" containing $SENDER, "X-Delivered-From:" containing $FROM, and the "X-Name:" record containing the user's name. The "X-" records will not be replaced, if they exist, but variable values extracted from them. It is the responsibility of the first account receiving a message to set these headers in the message. The "X-" records are used by various agents.


        :0
        * ^x-diagnostic:
        {
            :0 B
            * -+OAB32076.910133664/
            /dev/null
            #
            :0:
            x-diagnostic
        }

        

"OAB32076.910133664/" is the "signature" of a mime encapsulation from noattachment@thedomain.com, and cmd@thedomain.com; with an "X-Diagnostic:" record, the message is a bounce from one of those agent's that received the message from a daemon or mailer-probably an invalid address.


Anti-looping


        :0
        * 1^0 ^content-type:.*multipart/mixed;.*boundary.*\
               -+OAB32076.910133664/
        * 1^0 ^x-mime-audit-id:.*OAB32076.910133664
        {
            :0 wfh
            | formail -A "X-Diagnostic: Message is looping"
            #
            :0:
            x-diagnostic
        }

        

"OAB32076.910133664/" is the "signature" of a mime encapsulation from noattachment@thedomain.com, and cmd@thedomain.com; any message with these signature in the header records is looping, mandatory abort.


        :0
        * ^approval:
        * !^(x-)?approved:.*thedomain.com
        {
            :0 wfh
            | formail -I "Approved: theuser@thedomain.com" \
                      -I "Delivered-To:"
            #
            :0:
            approve
        }

        

A message that needs approval for delivery will have an "Approval: Bounce to ..." record containing the address from which it was forwarded for approval. For approval, bounce it back to the user with an "Approved: ... thedomain.com" record. (The existence of a "X-Approved:" or "Approved:" record in conjunction with an "Approval:" record is an exception condition-also, the "Delivered-To:" record, that was inserted by Qmail's anti-looping mechanism, when the message was delivered here for approval has to be removed; otherwise, Qmail will refuse delivery of the approval.) The construct is only required for those accounts that do approval.

An "X-Loop:" record containing this address is looping, unless it has an approved record-which is an intentional loop. If it has an approved record, remove all traces that the message was approved, and allow it to continue. If it does not have an approved record, mandatory abort. (Note that once an "X-Loop:" record is inserted, it is never removed; it is, however, over ridden if the message came back to the user after being approved.)

If the message has been approved, it was bounced with a set of "Resent-.*:" headers from the approver, remove them. Also, if the message has been approved, rename the "Approved:" record to "X-Approved:".


        :0
        * ^x-loop:.*theuser@thedomain.com
        {
            :0 wfh
            * ^approved:.*thedomain.com
            * !^x-approved:.*thedomain.com
            | formail -I "Resent-Date:" -I "Resent-Message-ID:" \
                      -I "Resent-From:" -I "Resent-To:" \
                      -R "Approved:" "X-Approved:"
            #
            # Else, the message is looping, mandatory abort.
            #
            :0 E
            {
                :0 wfh
                | formail -A "X-Diagnostic: Message is looping"
                #
                :0:
                x-diagnostic
            }

        }

        

However, if the message has an "Approved:" header, then rename it to "X-Approved:". The "X-Approved:" is the anti-looping construct. If it exists, then the message was sent for approval, came back approved, and is back again, for unknown reasons-i.e., the message is looping; it is a simple counter construct: 1) initial reception, (no "X-Loop:" record for this account,) guarantee no "X-Approved:" record, add "X-Loop:" record; 2) if approval required, then send the message to the approver with a "Approval:" record-if approved; 3) the message returns with an "Approved:" record, which is renamed to "X-Approved:", (which is the audit of who approved it,) and if; 4) the message returns with an "X-Approved:" record, it is looping, mandatory abort. (Note: a mandatory abort means save the message, and stop; the message can not be sent to an admin; that's a loopable construct.)


        :0 wfhE
        | formail -A "X-Loop: theuser@thedomain.com"
                  -I "X-Approved:"

        

Else, the message has never been through this account, add an "X-Loop:" record, and make sure there is no "X-Approved:" record.


Maintenance and establishment of variables


        :0 wfh
        * ^return-path:[ ]+[<] *[>]
        | formail -I "Return-Path:"

        

Qmail's mailer daemon inserts a "Return-Path: <>" record, which is favored by formail -rzx; if it exists, remove it.


        :0 wfh
        * ^delivered-to:
        * !^x-delivered-to:
        | formail -R "Delivered-To:" "X-Delivered-To:"

        

If there is no "X-Delivered-To:" record, rename the "Delivered-To:" record; this is the address of the first recipient of this message.


        :0 wfh
        * ^delivered-to:
        | formail -I "Delivered-To:"

        

If there is a "Delivered-To:" record, (i.e., the preceding statement did not execute because of a pre-existing "X-Delivered-To:" record,) then remove it-the "Delivered-To:" record is used by Qmail as an anti-looping mechanism-its existence means the message is looping, and delivery will be aborted; anti-looping is handled by an other mechanism; the "X-Loop:" record.


        :0
        * !^x-delivered-sender:
        {
            :0 wh
            SENDER=| formail -rzx To:
            #
            :0 wfha
            | formail -A "X-Delivered-Sender: ${SENDER}"
            #
            :0 E
            {
                :0 wfh
                | formail -A "X-Diagnostic: No machine generated \
                              return address"
                #
                :0:
                x-diagnostic
            }

        }
        #
        :0 whE
        SENDER=| formail -zx "X-Delivered-Sender:"

        

If an "X-Delivered-Sender:" record exists, use its contents as the machine generated return address of the message's sender, $SENDER. If not, create the record, and if that fails, mandatory abort.


        :0
        * ^x-delivered-sender:.*\/[^@.]+\.[^.]+$
        * MATCH ?? ^^\/.+
        {
            DOMAIN="${MATCH}"
        }
        #
        :0 E
        {
            :0 wfh
            | formail -A "X-Diagnostic: No domain name"
            #
            :0:
            x-diagnostic
        }

        

Extract the domain name, $DOMAIN, from the machine generated return address of the message's sender, and if that fails, mandatory abort.


        :0
        * !^x-delivered-from:
        {
            :0 wh
            FROM=| formail -rtzx To:
            #
            :0 wfha
            | formail -A "X-Delivered-From: ${FROM}"
            #
            :0 E
            {
                :0 wfh
                | formail -A "X-Diagnostic: No trusted return address"
                #
                :0:
                x-diagnostic
            }

        }
        #
        :0 whE
        FROM=| formail -zx "X-Delivered-From:"

        

If an "X-Delivered-From:" record exists, use its contents as the trusted return address of the message's sender, $FROM. If not, create the record, and if that fails, mandatory abort.


        :0 wfh
        * !^x-name:
        | formail -A "X-Name: Theuserfirst Theuserlast"

        

Add the user's name in an "X-Name:" header.


Confirmation replies

Messages addressed to the user, using Qmail's qmail-local addressing scheme, user-local@thedomain.com, of the form "To: user-123-123.123@thedomain.com", (which is the "signature" of a reply to a confirmation sent from cmd@thedomain.com; base acount addresses of the form user123@domain.com are not permitted,) were directed here by the users ~/.qmail-default file. If so, then extract the domain, and user, (i.e., without the -local part,) and validate that it is a legitimate address in this system, (the ~/.procmail.address file is a Unix flat file database, made with sort -u, of lower case characters, consisting of all legitimate addresses in this system,) using the bsearchtext database program. This validates that the message is is a reply to a confirmation sent from cmd@thedomain.com, and if so, fetch the message by sending this message to cmd-request@thedomain.com-it will be returned with an "Approved:" record.


        :0
        * ^((x-)?delivered-to|to|cc):.*\/[-0-9a-z._]+-[0-9]+\
           -[0-9]+\.[0-9]+@[-0-9a-z._]+
        {
            CMDADDRESS="${MATCH}"
            #
            :0
            * CMDADDRESS ?? ^^\/[-0-9a-z._]+[^-0-9.@]
            {
                CMDUSER="${MATCH}"
                #
                :0
                * CMDADDRESS ?? ^^.*@\/[-0-9a-z._]+
                {
                    CMDFQDN="${MATCH}"
                    #
                    :0
                    * CMDADDRESS ?? ^^[-0-9a-z._]+-[0-9]+\
                                    -\/[0-9]+\.[0-9]+[^@]
                    {
                        :0
                        * CMDADDRESS ?? ^^[-0-9a-z._]+-\/[0-9]+[^-]
                        {
                            :0
                            * ? /usr/local/bin/bsearchtext -r n -f \
                                "${HOME}/.procmail.addresses" \
                                "${CMDUSER}@${CMDFQDN}"
                            ! cmd-request@thedomain.com
                        }

                    }

                }

            }
            #
            :0 wfh
            | formail -A "X-Diagnostic: Malformed CMD address"
            #
            :0:
            x-diagnostic
        }

        

Further details can be found at Confirmed Mail Delivery.


Filter MS Outlook attachments for possible malicious code


        :0
        * !^x-audit-log:.*-?[0-9]+$
        {
            #
            # Encrypted, application, files, and base64 attachements
            # can not be searched:
            #
            :0
            * 1^0 $ ^content-type:${ws}(multipart/(signed|encrypted))\
                     |(application/)
            * 1^0 $ ^content-disposition:${ws}attachment;${ws}.*\
                     name${ws}=${ws}${dq}.*\.${ext}(\..*)?${dq}${ws}${eol}
            * 1^0 $ ^content-transfer-encoding:${ws}base64
            { MALICIOUS=true }
            #
            # All other mime mail can contain embedded, uuencode,
            # or html malicious code:
            #
            :0 BE
            * -3^0
            * 4^0 $ name${ws}=${ws}${dq}.*\.${ext}(\..*)?${dq}${ws}${eol}
            * 4^0 $ begin${ws}[0-9]+${ws}.*\.${ext}(\..*)?${ws}${eol}
            * 4^0 $ ^content-transfer-encoding:${ws}base64
            * 2^0 [<](!doctype|[sp]?h(tml|ead)|title|body)
            * 2^0 [<](app|bgsound|div|embed|form|i?l(ayer|ink)|img\
                    |i?frame(set)?|meta|object|s(cript|tyle))
            * 2^0 =3d
            { MALICIOUS=true }
            #
            :0
            * MALICIOUS ?? true
            {
                :0 wfh
                * !^x-audit-log:[ ]+potentially[ ]+malicious
                | formail -A "X-Audit-Log: Potentially malicious"
                #
                BOUNCE=true
            }

        }

        

Further details can be found at Quarantining Malicious Outlook Attachments.


Messages from daemons

Mailing lists

Note: the else-if construct, (:0 ... :0 E ... :0 E ...,) for daemons starts here, and ends exclusive of the detection for general daemons.

Note: ":0:\n* ^from:.* ...\n| formail -ds >> myfile" uses myfile.lock as a lockfile while un-digestifying and writing myfile.


        :0
        * ^to:.*theuser@(.*\.)*thedomain\.com
        * ^from:.*networksolutions\.com
        {
            DAEMON=true
            FOLDER="theuser"
        }

        

Example: E-mail from networksolutions that may not have a valid "To: " header in the address. (The thedomain.com domain has theuser@thedomain.com, registered with Network Solutions; domain business will be sent to those addresses.)


        :0 E
        * ^x-mailing-list:.*cmd@(.*\.)*thedomain.com
        {
            :0 B
            * -+OAB32076.910133664/
            /dev/null
            #
            DAEMON=true
            FOLDER="cmd"
        }

        

The agents noattachment@thedomain.com, and cmd@thedomain.com, send copies of messages handled to this account for auditing via their "dist" files. File them.

"OAB32076.910133664/" is the "signature" of a mime encapsulation from noattachment@thedomain.com, and cmd@thedomain.com; the message is a bounce from one of those agent's that was received from a daemon or mailer-probably an invalid address.


        :0 E
        * ^x-mailing-list:.*noattachment@(.*\.)*thedomain.com
        {
            :0 B
            * -+OAB32076.910133664/
            /dev/null
            #
            :0
            DAEMON=true
            FOLDER="noattachment"
        }

        

Additional examples:


        #
        :0 E
        * ^x-mailing-list:.*debian-user@lists\.debian\.org
        {
            DAEMON=true
            FOLDER="debian-user"
        }
        #
        :0 E
        * ^mailing-list:.*qmail-help@list\.cr\.yp\.to
        {
            DAEMON=true
            FOLDER="qmail"
        }
        #
        # Lists that are administered on thedomain.com.
        #
        :0 E
        * ^x-mailing-list:.*(bod|members)@(.*\.)*(thedomain.com)
        {
            DAEMON=true
            FOLDER="theuser"
        }

        

Machine generated messages

Note: the else-if construct, (:0 ... :0 E ... :0 E ...,) for daemons continues from mailing lists, and ends exclusive of the detection for general daemons.


        :0 E
        * ^from:.*fax@(.*\.)*thedomain.com
        {
            :0 c
            * ^Subject:[ ]+facsimile[ ]+received[ ]+from
            * B ?? ^\/recvq[^:]*
            |/usr/local/bin/metasend -b -t theuser@thedomain.com\
                -F fax@thedomain.com\
                -s "Facsimile mail transmission"\
                -S 5000000\
                -m application/fax\
                -f "/usr/spool/uucppublic/fax/${MATCH}"
            #
            DAEMON=true
            FOLDER="fax"
        }

        

Example: Fax notification from Hylafax.


        :0 E
        * 1^0 ^to:.*faxmaster@(.*\.)*thedomain.com
        * 1^0 ^subject:.*Cron.*--report
        * 1^0 ^subject:.*fax.*stats
        * 1^0 ^subject:.*output.*anacron
        {
            DAEMON=true
            FOLDER="system"
        }

        

Example: System notification.


        :0 E
        * ^from:.*(root|theuser)@(.*\.)*thedomain.com
        * ^subject:[ ]+reminders[ ]+for
        {
            DAEMON=true
            FOLDER="calendar"
        }

        

Example: Reminder service from remind(1) supplied program, remind-all.


        :0 E
        * ^subject:[ ]+((returned[ ]+mail:|delivery[ ]+report:) \
          [ ]+return[ ]+receipt|receipt[ ]+of[ ]+.*[ ]+message|registered:[ ]+)
        {
            DAEMON=true
            FOLDER="daemon"
        }

        

Example: Receipts.


        :0 E
        * ^from:.+-request@(.*\.)*thedomain\.com
        {
            DAEMON=true
            FOLDER="archive-request"
        }

        

Example: E-mail returned from archive requests. (Specifically, Nformatix.)

The Daemon macro is based on the original ^MAILER_DAEMON macro from the bottom of "man(1) procmailrc(5)," with additions for RFC822 paragraph 6.3, and, RFC2142 paragraphs 2-6, which define standard required accounts: billing, mailer-daemon, nobody, and, root added; ftp, info, marketing, news, sales, usenet, uucp, webmaster and, www omitted.

Additions for mailing list traffic from RFC2369 paragraph 3, and, RFC2919, standard required headers: list, mailing-list and x-mailing-list added, none omitted.


        :0
        * DAEMON ?? false
        * (^(((x-)?mailing-)?list(-(archive|help|id|name|owner|post|\
           (un)?subscribe))?:|beenthere:|precedence:.*(junk|bulk|list)|\
           (to|cc):[ ]+multiple[ ]+recipients[ ]+of|(((resent-)?(reply|from|\
           sender)|(return-receipt|errors|reply)-to|(return-)?path|\
           x-envelope-from):|>?from )([^>]*[^(.%@a-z0-9])?\
           (post(ma?(st(e?r)?|n)|office)|(send)?mail(er)?|daemon|\
           m(mdf|ajordomo)|n?uucp|list(serv|proc)|n(etserv|obody|\
           oc)|o(wner|ps)|r(e(quest|sponse)|oot)|b(ounce|bs\.smtp|\
           illing)|echo|mirror|s(erv(ices?|er)|mtp(error)?|ystem)|\
           abuse|hostmaster|s(ecurity|upport)|a(dmin(istrator)?|\
           mmgr|utoanswer))(([^).!:a-z0-9][-_a-z0-9]*)?[%@>\t ]\
           [^<)]*(\(.*\).*)?)?$([^>]|$)))
        {
            :0 B
            * -+OAB32076.910133664/
            /dev/null
            #
            :0
            * !^x-audit-log:.*-?[0-9]+$
            { BOUNCE=true }
            #
            DAEMON=true
            FOLDER="daemon"
        }
        #
        :0 wfh
        * DAEMON ?? true
        * !^x-audit-log:.*message.*from.*a.*daemon
        | formail -A "X-Audit-Log: Message from/to a daemon"

        

E-mail from a daemon; this must handle all daemons and mailing list distribution agents, otherwise noattachment@thedomain.com, or cmd@thedomain.com will respond to daemons and mailing lists. Also will detect messages addressed "To:" a daemon-since those may be forwarded/aliased/miss-addressed here.

The regular expression was taken from the bottom of procmailrc(5), but has root omitted, and several mailing list agent's "signatures" added. It is important that this construct protect the spam detection section, below, from mailing list messages, which will almost certainly create a false positive spam detection for list messages.

"OAB32076.910133664/" is the "signature" of a mime encapsulation from noattachment@thedomain.com, and cmd@thedomain.com; the message is a bounce from one of those agent's that was received from a daemon or mailer-probably an invalid address.


Message from a known address


        :0
        * DAEMON ?? false
        * !? /usr/local/bin/bsearchtext -r n -f \
             "${HOME}/.procmail.accept" "${FROM}" "${SENDER}"
        {
            :0
            * !? /usr/local/bin/bsearchtext -r n -f \
                 "${HOME}/.procmail.addresses" "${FROM}" "${SENDER}"
            {
                :0
                * !? fgrep -i -s -e "${FROM}" "${HOME}/.mailrc" \
                     "${HOME}/.address/maillog"
                {
                    KNOWN=false
                }

            }

        }

        

The ~/.procmail.accept and ~/.procmail.address files are Unix flat file databases, made with sort -u, of lower case characters, consisting of all legitimate addresses that the user has had previous correspondence with, and all legitimate addresses in this system, respectively-they will be searched using the bsearchtext database program for both the trusted and machine generated addresses of the sender. The ~/.mailrc and ~/.address/maillog files have no structure, and will be searched by fgrep using the trusted return address of the sender. Known daemons and list agents are assumed to be known addresses.


Message to a known address


        :0 wh
        ENVELOPE_TO=| formail -zx X-Delivered-To:
        #
        :0 Wh
        TEMP_TO=| /usr/local/bin/receivedAddressdb -r n -d -e \
                  forward.thedomain.com \
                  "${HOME}/.procmail.domains"
        #
        :0 whfa
        | formail -I "X-Delivered-To: ${TEMP_TO}"
        #
        :0 a
        {
            ENVELOPE_TO="${TEMP_TO}"
        }
        #
        :0 wfh
        * ^x-delivered-to:.*forward\.thedomain.com
        | formail -I "X-Delivered-To:"

        

The ~/.procmail.domains file is a Unix flat file database, made with sort -u, of lower case characters, consisting of all legitimate domains that this system uses which provide smtp services for the user. (The subdomain forward.thedomain.com is always used as a ~/.forward from other shell accounts-and should be omitted from the search for e-mail addresses in "Received: " records.) The receivedAddressdb program searches the messages "Received:" records for any address that contains any domain listed in ~/.procmail.domains, (specifically, a "Received: ... From ... For ... user@domain.com" construct, e.g., attempts to find the sendmail(1) specific ENVELOPE_TO address.) If successful, this address replaces the "X-Delivered-To:" address, which was derived from the "Delivered-To:" address. If a valid "X-Delivered-To:" record can not be generated, the record should be removed, entirely, from the e-mail header records.

This is, also, a convenient place to verify that the user is not rejecting e-mail from the message's sender or domain. The ~/.procmail.kill file is is a Unix flat file database, made with sort -u, of lower case characters, consisting of all the e-mail addresses, and/or, domains that the user wants to reject.


        :0
        * 1^0 !? /usr/local/bin/bsearchtext -r m -f \
                 "${HOME}/.procmail.addresses" \
                 "${ENVELOPE_TO}"
        * 1^0 ? /usr/local/bin/bsearchtext -r n -f \
                "${HOME}/.procmail.kill" \
                "${FROM}" "${SENDER}" "${DOMAIN}"
        { REJECT=true }

        

Validate that the ENVELOPE_TO address is legitimate by querying the ~/.procmail.address file, which is a Unix flat file database, made with sort -u, of lower case characters, consisting of all legitimate addresses in this system, using the bsearchtext database program. The ~/.procmail.kill is the same kind of database file, but consists of the addresses of all senders for which messages are to be discarded. The database program bsearchtext is used to query the file for both the machine generated and trusted addresses of the sender.


Messages from a trusted sender


        :0
        * ? /usr/local/bin/bsearchtext -r n -f \
            "${HOME}/.procmail.trusted" \
            "${FROM}" "${SENDER}" "${DOMAIN}"
        { TRUSTED=true }

        

The ~/.procmail.trusted file is a Unix flat file database, made with sort -u, of lower case characters, consisting of all trusted user e-mail addresses from which e-mail will be unconditionally delivered. The file will be queried for both the trusted and machine generated sender's e-mail address using the bsearchtext database program.

Alternative implementations could be:


        * ^(errors-to:|from:?|(return-)?path:|return-receipt-to:\
            reply-to:|sender:|resent-(from:|reply-to:|sender:))
           [ ]+.+@(.*\.)*thedomain.com

        

which would trust everyone in the domain thedomain.com, and no one outside of it, or:


        * ^return_path:[ ]+.+@(.*\.)*thedomain.com

        

which, using the Qmail specific "Return-Path:" header record, would do the same.


Spam detection


        :0
        * !^x-audit-log:.*-?[0-9]+$
        {

        

(A record "X-Audit-Log: -123" is the "signature" that the message already has a spam score-the number can be zero, positive, or negative.)


Evaluation of message header construction


            :0
            * $${SPAMSCORE}^0
            * 3361741^0 !^to:
            * 6454846^0 ^to:.*[<] *[>]
            * 6258282^0 ^to:.*undisclosed.*recipient
            * 4448203^0 ^cc:.*recipient.*list.*not.*shown
            * 5135798^0 ^received:.*microsoft exchange
            * 2167692^0 ^received:.*microsoft smtpsvc
            * 1272966^0 $ !^received:.*"${DOMAIN}"
            * 1257903^0 $ !^message-id:.*"${DOMAIN}"
            * 2217521^0 ^subject:.*!
            * 10361956^0 ^x-advertisement:
            * 5855766^0 ^subject:.*adv(ertise(ment)?.*)?\
                         ([ .:-]|$)
            * 5750007^0 ? test "${SENDER}" != "${FROM}"
            * 1683922^0 MALICIOUS ?? true
            * 1989573^0 !? /usr/local/bin/receivedTodb -r m \
                           "${HOME}/.procmail.addresses"
            * 5213281^0 ? /usr/local/bin/receivedIPdb -r n \
                          "${HOME}/.procmail.reject"
            * 2663031^0 ? /usr/local/bin/receivedMSGIDdb -r n -f \
                          "${FROM}" "${HOME}/.procmail.domains"
            * 4563378^0 ? /usr/local/bin/receivedUnknowndb -r n \
                          "${HOME}/.procmail.domains"
            { SPAMSCORE="$=" }

        

Evaluation of message body construction


            :0
            * < 1000000
            {
                :0 B
                * $${SPAMSCORE}^0
                * 1409686^0 base64
                * 847052^0 delete
                * 4750287^0 mailing
                * 2342018^0 $ ${dq}mailto:
                * 2125098^0 remove
                * 1468567^0 unsolicited
                * 8449986^0 unsubscribe
                { SPAMSCORE="$=" }

            }

        

If the sender's address is a known address, the message is probably not spam. If so, the message is treated as if it has a very low spam score; the negative of the spam score.


            :0
            * KNOWN ?? true
            {
                SPAMSCORE="-${SPAMSCORE}"
            }
            #
            :0 wfh
            | formail -A "X-Audit-Log: ${SPAMSCORE}"

        

Disposition of message commences

Evaluation of spam score. A spam score more than 15325533 is almost certainly spam, and less than 1175912 almost certainly not.

Message to be rejected, unless from a trusted user?


            :0
            * REJECT ?? true
            * TRUSTED ?? false
            /dev/null

        

Spam score large enough to trash the message, unless from a trusted user, or a daemon?


            :0
            * -15325533^0
            * $${SPAMSCORE}^0
            * DAEMON ?? false
            * TRUSTED ?? false
            /dev/null

        

Spam score still large enough to trash the message, unless from a trusted user?


            :0
            * -1175912^0
            * $${SPAMSCORE}^0
            * TRUSTED ?? false
            {
                :0 wfh
                * !^approval:.*bounce.*to.*\
                    theuser@thedomain.com
                | formail -A \
                  "Approval: bounce to theuser@thedomain.com"
                #
                :0
                * DAEMON ?? false
                ! cmd@thedomain.com
                #
                BOUNCE=true
            }

        

The Smartlist scripts are available for download that are used in the following construct; if a message has potentially malicious attachments, and is from the Internet, then the message is bounced back to the sender with a polite message explaining why the message was not delivered to the recipient. Note that attachments can still be used on the local network; only messages from the Internet, and, that contain potentially malicious attachments are bounced.

Message contain potentially malicious attachments, unless from a trusted user?


            :0
            * MALICIOUS ?? true
            * TRUSTED ?? false
            {
                :0 wfh
                * !^approval:.*bounce.*to.*\
                    theuser@thedomain.com
                | formail -A \
                  "Approval: bounce to theuser@thedomain.com"
                #
                :0
                * DAEMON ?? false
                * KNOWN ?? false
                ! noattachment@thedomain.com
                #
                BOUNCE=true
            }

        

Message to be bounced for approval, unless from a trusted user?


            :0
            * BOUNCE ?? true
            * TRUSTED ?? false
            {
                :0 wfh
                * !^approval:.*bounce.*to.*\
                    theuser@thedomain.com
                | formail -A \
                  "Approval: bounce to theuser@thedomain.com"
                #
                :0
                * DAEMON ?? false
                ! theuser@thedomain.com
            }

        

Message to be bounced for approval if sender is unknown, unless from a trusted user? (Only recommended to protect children's accounts from the I-sex people.)


            #
            # :0
            # * KNOWN ?? false
            # * TRUSTED ?? false
            # {
            #     :0 wfh
            #     * !^approval:.*bounce.*to.*theuser@thedomain.com
            #     | formail -A "Approval: bounce to theuser@thedomain.com"
            #     #
            #     :0
            #     ! theuser@thedomain.com
            # }

        }

        

Message is probably valid, file it in folder(s).


Reject duplicate messages


        :0 Wh :msgid.cache.lock
        | formail -D "${MSGIDCACHESIZE}" "${MAILDIR}/.msgid.cache"

        

Message from a unknown user

If the message is from an unknown user, then save the machine generated and trusted return addresses in the ~/.address/maillog file.


        :0 Wic :"${HOME}/.address/maillog.lock"
        * KNOWN ?? false
        * DAEMON ?? false
        | echo -e "${FROM}\n${SENDER}" >> "${HOME}/.address/maillog"

        

Message to be filed in folder


        :0:
        * ? test "${FOLDER}" != ""
        "${FOLDER}"

        

Messages that will automatically be archived


        :0
        * ^from:.*theuserfriend@(.*\.)*thedomain.com
        {
            :0 c
            ! theuser-archive@thedomain.com
            #
            :0:
            theuserfriend
        }

        

Emacs/RMAIL/VM generated forward and Bcc from this account

If forward, or archive request, file in folder; if not, it is a Bcc from an e-mail, archive it.


        :0
        * ^reply-to:.*theuser@(.*\.)*thedomain.com
        {
            :0:
            * 1^0 ^to:.+-request@(.*\.)*thedomain.com
            * 1^0 ^subject:[ ]+forwarded[ ]+message[ ]+from[ ]+.*|\
                  \[theuser@(.*\.)*thedomain.com:[ ]+.*\]
            not-filed
            #
            :0
            ! theuser-archive@thedomain.com
        }

        

If emacs/RMAIL/VM generated forward and Bcc from user's account(s) on the Internet, file the message, (the ~/.procmail.address file is a Unix flat file database, made with sort -u, of lower case characters, consisting of all legitimate addresses in this system,) using the bsearchtext database program.


        :0:
        * ? /usr/local/bin/bsearchtext -r n -f \
            "${HOME}/.procmail.addresses" \
            "${FROM}" "${SENDER}"
        not-filed

        

Final default disposition, (folder, pop-3, imap, etc.)

Anything that has not been handled will be filed in ${MAILDIR}/${DEFAULT}, using $MAILDIR/$DEFAULT.lock as a lock file.


Performance

The script requires, on average, about 0.11 seconds of CPU time, (from the time(1) command,) plus about 1 second per MB to scan for malicious attachments, to process an e-mail, on a 433 MHz. Pentium class machine.


Thanks

A special note of appreciation to Stephen R. van den Berg, (AKA BuGless,) the author of procmail, who for nine years developed and supported the procmail program, (the "e-mail system administrator's crescent wrench,") for the Internet community. And, a special thanks to Philip Guenther the current maintainer of procmail, and moderator of the procmail mailing list for providing the search optimization for the procmail "recipe" described above.


License

A license is hereby granted to reproduce this software for personal, non-commercial use.

THIS PROGRAM IS PROVIDED "AS IS". THE AUTHOR PROVIDES NO WARRANTIES WHATSOEVER, EXPRESSED OR IMPLIED, INCLUDING WARRANTIES OF MERCHANTABILITY, TITLE, OR FITNESS FOR ANY PARTICULAR PURPOSE. THE AUTHOR DOES NOT WARRANT THAT USE OF THIS PROGRAM DOES NOT INFRINGE THE INTELLECTUAL PROPERTY RIGHTS OF ANY THIRD PARTY IN ANY COUNTRY.

So there.

Copyright © 1992-2005, John Conover, All Rights Reserved.

Comments and/or problem reports should be addressed to:

john@email.johncon.com

http://www.johncon.com/john/
http://www.johncon.com/ntropix/
http://www.johncon.com/ndustrix/
http://www.johncon.com/nformatix/
http://www.johncon.com/ndex/



Home | John | Connie | Publications | Software | Correspondence | NtropiX | NdustriX | NformatiX | NdeX | Thanks


Copyright © 1992-2005 John Conover, john@email.johncon.com. All Rights Reserved.
Last modified: Sat Aug 20 02:05:44 PDT 2005 $Id: index.html,v 1.0 2005/08/20 09:06:13 conover Exp $
Valid HTML 4.0!