Home > Articles > Procmail Mail Filtering

Procmail Mail Filtering

I've toyed around with open-source mail filtering packages, most of which work pretty well; but decided that the best way to learn procmail was to dive straight into it, warts and all. Hopefully there are one or two tips on this page that may be of use to other lost souls out there in procmail-land.

Build

Go ahead and fetch the source from http://www.procmail.org

$ cd procmail-src $ make install

We need to secure the files. This next command will chown root /usr/bin/procmail and chmod 4755 /usr/bin/procmail

$ make install-suid

If you are a system administrator you should consider integrating procmail into the mail-delivery system for advanced functionality, speed and security. To do this, you'll have to have sendmail configured to use procmail as your local delivery agent, and create your rules (if you have any) in /etc/procmailrc. Otherwise, you can create individual ~/.procmailrc files for each user, and set up their .forward file to read:

"|IFS=' ' && exec /usr/bin/procmail -f- || exit 75 #david"

Configure

Create a standard template file which will hold some very useful variables:

SHELL=/bin/sh

Define the shell to be used. Very useful if you're used to writing (for example) in the bash shell, but your environment uses the c-shell.

PATH=/usr/bin

Define the paths to be used; good for security

LOGFILE="$HOME/procmail.log"

Store the logfile in each user's home directory. In quotes since we don't know if the $HOME variable will have spaces in it or not.

LOGABSTRACT=false

Don't use log abstracts (we'll create our own)

EOL=" "

Define an End-Of-Line (very useful in creating log entry lines that by default don't end in a newline)

DATE=`date`

Define the current system date. Notice the use of the backticks (`)

DQ='"'

Define a double-quote character, since procmail has trouble matching it

SPC="[ ]*"

Define one or more spaces/tabs. Note that the asterix is included here; don't include it later in your regex

NOSPC="[^ ]"

The opposite -- not a space/tab. In brackets is exactly one space and one tab.

XLOOP="X-Loop: xloop@$HOST"

A header used to prevent endless mail loops

:0 * $ ^Subject${SPC}: ${SPC}\/.+ { SUBJECT="$MATCH" }

Grab the subject line. Note the extra literal space before the second $SPC variable.

:0 * $ ^From${SPC}: ${SPC}\/.+ { APPARENTLY="$MATCH" }

Grab the name of the sender, which may or may not be accurate (i.e., faked).

:0 * $ ^^From $SPC\/.+ { FROM="$MATCH" }

The envelope "From" field. Note the extra literal space before the variable. The double caret (^^) links the regex to the top of the input.

LOG="${EOL}${DATE}${EOL}Processing: '$SUBJECT' from '$FROM'$EOL"

Start our logging entry.

:0 fh | formail -cz | sed -e 's/^[ ][ ]*/ /g' -e 's/^[ ][ ][ ]*/ /g'

Flatten the headers and remove all extra spaces, replacing all tabs with spaces which makes it easier to parse. See RFC822 if you're confused about multi-line whitespace. Note that if we replace all instances of extra spacing in the header our mailbox format will become invalid when dates such as "March 1" are changed into "March 1" (single space); go figure.

Let's assume that the file above is stored in /etc/procmail-globals. It's very unlikely that this file will need to change, thus we put it in it's own location. Just make sure to add the following to the top of your /etc/procmailrc file:

INCLUDERC=/etc/procmail-globals

After this, you can put in any assortment of scripts, conditions, etc. Although not strictly necessary, make sure that the bottom of your script has the following bit to deliver all e-mail that hasn't yet been filtered by any of your scripts:

:0 ${DEFAULT}

Sample Script:

I don't keep this script up to date, so I don't recommend it for use in production environments. However, it gives some good insight into how to write a script. Note that file extension filtering is inherintly a bad idea without additional scripting; imagine your suprise when someone e-mails you a file attachment named 'badfile.%"ex"$e' which your e-mail client transforms into 'badfile.exe'. Or worse, a file attachment with no name but a Content-Type: definition, and the e-mail program generates a name automatically for it. Sigh...

# Turn on Verbose logging VERBOSE=1

# Make sure we haven't already processed this mail. # The first $ tells procmail to interpret variables as variables, not literals # The \ after the second $ tells procmail to escape all meta characters that might be in the variable. :0 * $ ! ^$\XLOOP { LOG="Checking for bad file extensions$EOL"

:0 HB * 0^0 * 1^0 $ ^${SPC}name$SPC=$SPC$DQ?\/[^$DQ][^DQ]*\.$EXT * 1^0 $ ^${SPC}begin$SPC([0-9][0-9]*)?$SPC\/.*\.$EXT { LOG="*** Found bad file attachment named $MATCH!$EOL" :0 fwh | formail -A"$XLOOP" -i"Subject: $SUBJECT ($LOGNAME/$MATCH)" :0 fwbh | sed -e 's/name[ ]*=[ ]*"*[^"][^"]*\.[^ "][^ "]*/&.DEFANGED/g' :0 fwbh | sed -e 's/begin[ ]*([0-9][0-9]*)?[ ]*.*\.[^ "][^ "]*/&.DEFANGED/g' :0: ${DEFAULT} } }

Tips:

Remember that procmail is supposed to be egrep-syntax familiar for regex's, but it's not (fyi). Here's a general procmailrc script which you can hash with. I don't keep it up to date here on the web, but it will give you a few good general ideas.
If you want to use sed in your script, anything you match can be pasted into your replacement script by using the & symbol. Thus sed -e 's/fred/&dy/g' would change "fred" into "freddy".
Another nifty trick (which was a major pain in my side until I came across it) with sed is case-insensitive matching. sed -e 's/WINMAIL\.DAT/WINMAIL.BAD/g' would match "WINMAIL.DAT" but not "winmail.dat" (lower-case). Instead, use sed -e 's/WINMAIL\.DAT/WINMAIL.BAD/I'
If you want to match one or more items in procmail, you might be tempted to try using something like [A-Z]+, but you'll find that your scripts will not always work as expected. Instead, use this slight workaround: [A-Z][A-Z]*
Using :0 D as the start of your script line will make the test case-sensitive.
You can match word-boundaries (anything not in [-_A-Z0-9]) using \< and \> for start/end (respectively).
The built-in macro ^TO_ expands out to: (^((Original-)?(Resent-)?(To|Cc|Bcc)|(X-Envelope|Apparently(-Resent)?)-To):(.*[^a-zA-Z])?)
If you use echo statements in your script, make sure to use a lockfile!
To evaluate a condition based on the body or header, use B and H respectively.
To filter the contents of the body or header, use b and h respectively.

Anatomy of E-Mail Attachments

It's important to get a good overview of the structure of MIME e-mail before you attempt to write a script for it.

MIME e-mail without attachment (huh?)

Yes, it is possible to send MIME e-mail without sending any file attachments. Simply declairing "MIME-Version: 1.0" in the header makes it so. Which can confuse your scripts if you are anticipating attachments with all MIME e-mails. Ok, technically it's not a MIME-email but your script won't know that!

From fred@localhost Thu Jan 10 16:41:11 2002 Return-Path: <fred@localhost> Received: from station.fluffygerbil.com by boss.fluffygerbil.com (8.12.1/8.12.1) with SMTP id g0B0fB4v027415 for <fred@localhost>; Thu, 10 Jan 2002 16:41:11 -0800 From: "Fred" <fred@localhost> To: "Fred" <fred@localhost> Subject: MIME e-mail without attachment Date: Thu, 10 Jan 2002 16:42:37 -0800 Message-ID: <JOELLNDAIKGHCPPDJMPMEEAKCAAA.fred@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700

This is a MIME e-mail message. Is it not nifty?

MIME e-mail with attachment

This is the standard boilerplate MIME e-mail attachment message. If it looks confusing to you, you will probably want to review some bits at http://cr.yp.to/immhf.html which gives a good overview.

From fred@localhost Thu Jan 10 16:46:34 2002 Return-Path: <fred@localhost> Received: from station.fluffygerbil.com by boss.fluffygerbil.com (8.12.1/8.12.1) with SMTP id g0B0kY4v027459 for <fred@localhost>; Thu, 10 Jan 2002 16:46:34 -0800 From: "Fred" <fred@localhost> To: "Fred" <fred@localhost> Subject: MIME e-mail with attachment Date: Thu, 10 Jan 2002 16:48:00 -0800 Message-ID: <JOELLNDAIKGHCPPDJMPMMEAKCAAA.fred@localhost> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0026_01C199F6.9030B400" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700

This is a multi-part message in MIME format.

------=_NextPart_000_0026_01C199F6.9030B400 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit

This is a MIME e-mail message with a file attachment. Is it not nifty? ------=_NextPart_000_0026_01C199F6.9030B400 Content-Type: application/octet-stream; name="Nifty.png" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="Nifty.png"

Qk22AAAAAAAAAD4AAAAoAAAAHwAAAB4AAAABAAEAAAAAAHgAAAATCwAAEwsAAAIAAAACAAAAAAAA AICAgAAAAAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAA kkkAAEkk//wAAAAAgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgA AJJIgACSSH/+SSQ=

------=_NextPart_000_0026_01C199F6.9030B400--

MIME e-mail with attachment without using MIME Boundaries (suprise!)

Ah, you say. You've built this nifty procmail script to block e-mail attachments, and someone managed to send an e-mail with a file attachment right past your script! See if you can pick up the reasons why, and compare to the example above. In other words, should I have highlighted it red or blue to keep with the scheme from above? (answer - both!). This tactic is often overlooked when building scripts.

From fred@localhost Thu Jan 10 16:23:17 2002 Return-Path: <fred@localhost> Received: from station.fluffygerbil.com (hugo.fluffygerbil.com [192.168.51.119]) by boss.fluffygerbil.com (8.12.1/8.12.1) with SMTP id g0B0NH4v027303 for <fred@localhost>; Thu, 10 Jan 2002 16:23:17 -0800 From: "Fred" <fred@localhost> To: "Fred" <fred@localhost> Subject: test Date: Thu, 10 Jan 2002 16:24:43 -0800 Message-ID: <JOELLNDAIKGHCPPDJMPMAEAKCAAA.fred@localhost> MIME-Version: 1.0 Content-Type: application/octet-stream; name="Nifty.png" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="Nifty.png" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700 Status: O X-Status: X-Keywords: X-UID: 579

Qk22AAAAAAAAAD4AAAAoAAAAHwAAAB4AAAABAAEAAAAAAHgAAAATCwAAEwsAAAIAAAACAAAAAAAA AICAgAAAAAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAA kkkAAEkk//wAAAAAgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgA AJJIgACSSH/+SSQ=

E-mail Clients Dissected

This is actually handy for reference when I'm trying to determine issues that different mail clients might have.

UNIX command-line e-mail (no MIME)

From root@station.fluffygerbil.com Thu Jan 10 17:19:56 2002 Return-Path: <root@station.fluffygerbil.com> Received: from station.fluffygerbil.com (station.fluffygerbil.com [192.168.51.103]) by boss.fluffygerbil.com (8.12.1/8.12.1) with ESMTP id g0B1Ju4v027733 for <fred@localhost>; Thu, 10 Jan 2002 17:19:56 -0800 Received: from station.fluffygerbil.com (localhost [127.0.0.1]) by station.fluffygerbil.com (8.12.1/8.12.1) with ESMTP id g0B1LQrZ022290 for <fred@localhost>; Thu, 10 Jan 2002 17:21:26 -0800 Received: (from root@localhost) by station.fluffygerbil.com (8.12.1/8.12.1/Submit) id g0B1LQm3022289 for fred@localhost; Thu, 10 Jan 2002 17:21:26 -0800 Date: Thu, 10 Jan 2002 17:21:26 -0800 From: ROSIE <root@station.fluffygerbil.com> Message-Id: <200201110121.g0B1LQm3022289@station.fluffygerbil.com> To: fred@localhost Subject: Another e-mail

This is a simple e-mail.

MIME with elm (sort of)

From fred@station.fluffygerbil.com Thu Jan 10 17:02:53 2002 Return-Path: <fred@station.fluffygerbil.com> Received: from station.fluffygerbil.com (station.fluffygerbil.com [192.168.51.103]) by boss.fluffygerbil.com (8.12.1/8.12.1) with ESMTP id g0B12r4v027631 for <fred@localhost>; Thu, 10 Jan 2002 17:02:53 -0800 Received: from station.fluffygerbil.com (localhost [127.0.0.1]) by station.fluffygerbil.com (8.12.1/8.12.1) with ESMTP id g0B14NrZ022171 for <fred@localhost>; Thu, 10 Jan 2002 17:04:23 -0800 Received: (from fred@localhost) by station.fluffygerbil.com (8.12.1/8.12.1/Submit) id g0B14NAv022170 for fred@localhost; Thu, 10 Jan 2002 17:04:23 -0800 From: Fred <fred@station.fluffygerbil.com> Message-Id: <200201110110.g0B1AAb2027675@boss.fluffygerbil.com> Subject: Another e-mailTo: fred@localhost Date: Thu, 10 Jan 2002 17:10:10 -0800 (PST) X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit

This is written with elm - everyone loves elm.

MIME with mutt

From fred@station.fluffygerbil.com Thu Jan 10 17:02:53 2002 Return-Path: <fred@station.fluffygerbil.com> Received: from station.fluffygerbil.com (station.fluffygerbil.com [192.168.51.103]) by boss.fluffygerbil.com (8.12.1/8.12.1) with ESMTP id g0B12r4v027631 for <fred@localhost>; Thu, 10 Jan 2002 17:02:53 -0800 Received: from station.fluffygerbil.com (localhost [127.0.0.1]) by station.fluffygerbil.com (8.12.1/8.12.1) with ESMTP id g0B14NrZ022171 for <fred@localhost>; Thu, 10 Jan 2002 17:04:23 -0800 Received: (from fred@localhost) by station.fluffygerbil.com (8.12.1/8.12.1/Submit) id g0B14NAv022170 for fred@localhost; Thu, 10 Jan 2002 17:04:23 -0800 Date: Thu, 10 Jan 2002 17:04:23 -0800 From: Fred <fred@station.fluffygerbil.com> To: fred@localhost Subject: Another e-mail Message-ID: <20020110170423.A22146@station.fluffygerbil.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="n8g4imXOkfNTN/H1" Content-Disposition: inline User-Agent: Mutt/1.2.5i

--n8g4imXOkfNTN/H1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline

This is written with mutt, which sucks less.

--n8g4imXOkfNTN/H1 Content-Type: image/png Content-Disposition: attachment; filename="Nifty.png" Content-Transfer-Encoding: base64

Qk22AAAAAAAAAD4AAAAoAAAAHwAAAB4AAAABAAEAAAAAAHgAAAATCwAAEwsAAAIAAAACAAAAAAAA AICAgAAAAAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAA kkkAAEkk//wAAAAAgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgA AJJIgACSSH/+SSQ=

--n8g4imXOkfNTN/H1--

MIME with Microsoft Outlook

From fred@localhost Thu Jan 10 16:46:34 2002 Return-Path: <fred@localhost> Received: from station.fluffygerbil.com by boss.fluffygerbil.com (8.12.1/8.12.1) with SMTP id g0B0kY4v027459 for <fred@localhost>; Thu, 10 Jan 2002 16:46:34 -0800 From: "Fred" <fred@localhost> To: "Fred" <fred@localhost> Subject: MIME e-mail with attachment Date: Thu, 10 Jan 2002 16:48:00 -0800 Message-ID: <JOELLNDAIKGHCPPDJMPMMEAKCAAA.fred@localhost> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0026_01C199F6.9030B400" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700

This is a multi-part message in MIME format.

------=_NextPart_000_0026_01C199F6.9030B400 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit

This is written with Microsoft Outlook. Sigh... ------=_NextPart_000_0026_01C199F6.9030B400 Content-Type: application/octet-stream; name="Nifty.png" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="Nifty.png"

Qk22AAAAAAAAAD4AAAAoAAAAHwAAAB4AAAABAAEAAAAAAHgAAAATCwAAEwsAAAIAAAACAAAAAAAA AICAgAAAAAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAA kkkAAEkk//wAAAAAgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgA AJJIgACSSH/+SSQ=

------=_NextPart_000_0026_01C199F6.9030B400--

MIME with Netscape Messenger

From fred@localhost Thu Jan 10 17:16:32 2002 Return-Path: <fred@localhost> Received: from boss.fluffygerbil.com (hugo.fluffygerbil.com [192.168.51.119]) by boss.fluffygerbil.com (8.12.1/8.12.1) with ESMTP id g0B1GW4v027717 for <fred@localhost>; Thu, 10 Jan 2002 17:16:32 -0800 Message-ID: <3C3E3D46.FFE125DA@boss.fluffygerbil.com> Date: Thu, 10 Jan 2002 17:17:58 -0800 From: Fred <fred@localhost> X-Mailer: Mozilla 4.79 [en] (Win95; U) X-Accept-Language: en MIME-Version: 1.0 To: fred@localhost Subject: Another e-mail Content-Type: multipart/mixed; boundary="------------50B956BD89D099378DE6F998"

This is a multi-part message in MIME format. --------------50B956BD89D099378DE6F998 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit

This is written with Netscape Messenger 4.7x --------------50B956BD89D099378DE6F998 Content-Type: image/png; name="Nifty.png" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="Nifty.png"

Qk22AAAAAAAAAD4AAAAoAAAAHwAAAB4AAAABAAEAAAAAAHgAAAATCwAAEwsAAAIAAAACAAAA AAAAAICAgAAAAAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI//ySSAAAkkkAAJJI //ySSAAAkkkAAEkk//wAAAAAgACSSH/+kkgAAJJIgACSSH/+kkgAAJJIgACSSH/+kkgAAJJI gACSSH/+kkgAAJJIgACSSH/+SSQ= --------------50B956BD89D099378DE6F998--

Additional Resources:

The single best book on Procmail is The Procmail Companion by Martin McCarthy (currently out of print, but available used on Amazon). It's definately worth it, and helps to counter some of the conflicting off-the-shelf advice you'll come across when researching your procmailrc script. It explains the basics of how email works and demonstrates: how to set up simple and advanced mail processing and filtering, procmail scoring, invoking external processing programs from within procmail, procmail regular expressions, and procmail-specific syntax, macros, variables and diagnostic messages.