URN:NBN:fi-fe20051953 Copyright © 1999- by Prof. Timo Salmi    Last modified Sat 13-May-2023 17:34:17

1. I want to filter my email automatically. How do I get started with procmail?
2. Building a testbench. How can I test individual procmail recipes?
3. I know how to make "and" rules in procmail recipes, but how do I make "or" rules?
4. How can one perform multiple shell commands on the action line?
5. How can I find out what the subject of a posting is?
6. How do I get a copy of the headers of all the incoming email into a separate file?
7. Would you give some further hints for spam foiling recipes?
8. I have limited disk space. How can I truncate long messages?
9. How can I quickly test if my rules with regular expressions match?
10. How can I detect if the email comes, say, from the .com domain?
11. What alternatives do I have to detect a sender all through the various header-fields?
12. How can I extract a valid address from the Reply-To field?
13. How can I extract the address of the sender's postmaster?
14. How can I weed out an inordinately long recipient list?
15. What is this procmail scoring? How can I utilize it?
16. How can I test if the subject is empty or if the subject field is missing altogether?
17. How can I modify the "To:" field of the email I received?
18. I have a long list of spammers in a separate file. How can I utilize it?
19. How do I forward certain messages that I get, and preserve myself a copy?
20. How do I forward certain messages to two different addresses?
21. How do I automatically return certain email messages?
22. My address has changed. How do I forward a copy to myself and tell the sender?
23. How can I set variable values based on the text in the body of the email message?
24. How can I insert some token text in front of the body of incoming email?
25. Do you have any useful tips for regular expression matching?
26. How can I test if two procmail variables have the same contents?
27. I am having difficulties with "<". How does one match it?
28. How can I insert identification text to the beginning of the subject line?
29. I tried out your tips, but some of them failed on my system. What next?
30. Is there a cure for the echo and grep blues?
31. How do I know which of my many procmail recipes has been enacted?
32. How can I detect Korean, Cyrillic, or Chinese to avoid such frequent spam?
33. How can I change the subject line and include part of the message body into it?
34. How can I remove the signature from the incoming email?
35. What unix manuals relating to procmail should I get?
36. Is it possible to use procmail to call the vacation program?
37. How can I avoid duplicate messages sent in rapid succession?
38. How can I skip logging a certain, matched recipe?
39. Could you please solve for me this procmail problem of mine?
40. I liked this material. Do you have anything else on programming?
41. Exercises
42. Acknowledgements for useful advice and/or feedback

#Preliminaries
SHELL=/usr/bin/sh               #Use the Bourne shell (check your path!)
MAILDIR=${HOME}/Mail            #First check what your mail directory is!
LOGFILE=${MAILDIR}/procmail.log
LOG="--- Logging ${LOGFILE} for ${LOGNAME}, "

#Whatever recipes you'll use
#The order of the recipes is significant
:0
* ^From: scam@cyberspam\.com
/dev/null

# Accept all the rest to your default mailbox
:0:
${DEFAULT}

"|IFS=' ' && exec /usr/local/bin/procmail || exit 75 #myid"

SHELL=/bin/sh
TESTDIR=/home/myid/test
MAILDIR=${TESTDIR}
LOGFILE=${TESTDIR}/Proctest.log
LOG="--- Logging for ${LOGNAME}, "

#Troubleshooting:
VERBOSE=yes
LOGABSTRACT=all

#Let's test stripping lines from the email message's header
:0 fwh
| egrep -vi "(^Content-|^MIME-Version:.)"

#If it is from myself, store the email message
:0:
* $ ^From:.*${LOGNAME}
${TESTDIR}/Proctest.mail

#Otherwise, discard the email message
:0
/dev/null

#Trivial catching of potential spam towards the end of a ~/.procmailrc
#Place only after accepting all the mailing lists you want to receive
:0:
* ! ^TO_ts@([-a-z0-9_]+\.)*uvasa\.fi
* ! ^TO_timo\.salmi
${HOME}/.mail/PotentialSpam.mail

#Accept email from Era Eriksson, the author of the major procmail FAQ
:0:
* ^From:.*reriksso@([-a-z0-9_]+\.)*helsinki\.fi|\
  ^From:.*era@iki\.fi
${DEFAULT}

• The "^TO_" in the first recipe is a procmail reserved predefined special expression "which should catch all destination specifications containing a specific address." It must be written in upper case.
   
  
• The "!" in the first recipe is the familiar operator indicating a negation.
   
  
• If "${HOME}/.mail" is your mail directory you don't need to spell out the entire path "${HOME}/.mail/PotentialSpam.mail". Just "PotentialSpam.mail" will be sufficient.
   
  
• The first detail of the "or" example is complicated and is per se unrelated to the "or" issue at hand. The "([-a-z0-9_]+\.)*" expression in "reriksso@([-a-z0-9_]+\.)*helsinki\.fi" sees to it that if Era has several machines in his domain (as I do under mine), all will be matched by the recipe. The "[-a-z0-9_]" matches any of the characters within the brackets "[]", the trailing "+" tells that there must be at least one repeat of those characters, the "\." matches a dot, and the "*" tells that there has to be zero or more repeats if the preceding expression within the parentheses "()". [This item owes heavily to Era's friendly guidance.]
   
  
• The backslash "\" in "helsinki\.fi" sees to it that the actual dot (.) is matched. This is because if the "quote next character" "\" is omitted, the "." is taken as a regular expression matching any (exactly one) character.
   
  
• The "|" in the "|\" indicates an "or" condition, and the "\" quotes the embedded end of line, i.e. tells that the rule is continued on the next line.
   
  
• The "|" or condition sees to it that the recipe matches email coming from Era either from the "helsinki.fi" or the "iki.fi" domain.
   
  
• The "${DEFAULT}" puts the email in the regular mailbox.
   
  
• The trailing ":" in the recipe start line ":0:" tells procmail to use temporary file locking to avoid writing simultaneously arriving potential email on top of each other at your "${DEFAULT}" mailbox. Since no lock file name is given after the ":0:", procmail will provide the lockfile name. Always use this format when delivering to a mail folder, unless the target folder is /dev/null. That is, unless you want the email is discarded.

:0:
* 1^0 ^From:.*reriksso@([-a-z0-9_]+\.)*helsinki\.fi
* 1^0 ^From:.*era@iki\.fi
${DEFAULT}

:0:
* ^From:.*(\
reriksso@([-a-z0-9_]+\.)*helsinki\.fi|\
era@iki\.fi)
${DEFAULT}

1. The potentially existing "Test.zip" zip-file is unzipped to obtain the earlier email messages that already might be within Test.zip.
    
   
2. The incoming email is appended to the extracted Test.mail file.
    
   
3. The updated Test.mail file is compressed back into the Test.zip zip-file.
    
   
4. The uncompressed Test.mail is deleted.

SHELL=/bin/sh
TESTDIR=/home/myid/test
MAILDIR=${TESTDIR}
LOGFILE=${TESTDIR}/Proctest.log
LOG="--- Logging for ${LOGNAME}, "

#Troubleshooting:
VERBOSE=yes
LOGABSTRACT=all

#An auxiliary regular expression to detect text,
#The brackets [] contain a space and a tab
GETTEXT="[  ]*\/[^  ].*"

• The first part "[ ]*" of the regular expression matches any number of spaces and tabs (even the case of none) which can lead the subject.
   
  
• The "\/" is a special procmail-only operand which puts a (possible) match found by the rest of the expression into a variable named MATCH.
   
  
• "[^ ]" means all other characters but the one's within the brackets. The ".*" means that a match of non-tab, non-space characters is sought for.

• The "c" flag in ":0wc" tells that the processing should continue also after this particular recipe has been acted upon. (When the "c" flag is not present, the all the rest of the recipes in proctest.rc are all skipped.) The "w" tells to wait until the "|" pipe has finished.
   
  
• The ":${TESTDIR}/Proctest.mail.lock" tells which lockfile to use in order to avoid the confusion from the possibility of simultaneous arrival of several email messages. Note that since we use a pipe "|" in the actions part, it is prudent to explicitly give the name of the lock.
   
  
• Note the first "$" on the "$ ^Subject:${GETTEXT}" condition line. It tells that the environment variables (in this case "GETTEXT") on the line are to be expanded, not to be taken as literal text.

• The "h" flag in ":0hc" tells that the header should be accessed.
   
  
• The "c" flag in ":0hc" orders the processing to continue also after this recipe. In other words, you put your other recipes, after the header-catching, in the ordinary fashion. The email will reach them.
   
  
• The ":${HOME}/.mail/Procmail.head.lock" tells which particular lockfile to use.
   
  
• Since there are no condition lines (lines starting with *) this item will always be acted upon when it is reached. You wanted to log the headers from all the incoming email, right?
   
  
• The "| cat >> ${HOME}/.mail/Procmail.head" appends the headers to the ${HOME}/.mail/Procmail.head file.

#Catch potential spam
:0
* !^TO_(ts|timo\.salmi)@([-a-z0-9_]+\.)*uwasa\.fi
{
  :0 fwh
  * ^Content-Length:
  | formail -IContent-Length:
  :0:
  Spam.mail
}

#Spam avoidance of certain routes and if not for me personally
:0:
* -1^0
*  1^0 ? formail -x"Received:" | egrep -is "dialsprint\.net"
*  1^0 ! ^TO_(myid|myFirstName[ _\.]myLastName)@([-a-z0-9_]+\.)*myhost\.mydom
Spam.mail

• The "?" at the start of the condition executes and evaluates what is on the condition line instead of searching for a literal match.
   
  
• Procmail's companion program formail is used to extract all the "Received:" routing information from the posting's header. Then "dialsprint.net" is sought for using Unix egrep via the "|" pipe.
   
  
• This is a sideline, but the simpler, less general form of the last condition line would, of course, be just "*  1^0 ! ^TO_myid@myhost\.mydom"
   
  
• The scoring system is explained elsewhere on this page, but in brief the score is initialized at -1. Each explicit condition is given a weight of 1. If the total score is at least 1 (i.e. positive) then the action (storing to the Spam.mail file) is initiated.

:0B:
* (remove@|removeme@)
PotentialSpam.mail

• The "B" flag tells the recipe to search through the body of the email message.
   
  
• Note the "or" testing on the conditions line.
   
  
• Note again the file locking (the trailing : in ":0B:"). Since the email message is directed to a folder, we do not need explicitly to name the lockfile. We can let procmail do it. As a default it will use the name PotentialSpam.mail.lock
   
  
• The "B" means the body and only the body of the message. The header is not included. However, I have as hearsay that some procmail versions have a bug in this respect, but I have not been able to test that situation myself.

:0D:
* ^Subject:.*ADV
PotentialSpam.mail

• The "D" flag tells to distinguish between the lower and the upper case in testing for a match.

:0:
* (^Subject:.*make.*money.*fast|^Subject:.*\$\$\$)
PotentialSpam.mail

• Note, not "^Subject:.*$$$", but "^Subject:.*\$\$\$" because, if not quoted with "\", a "$" is taken as a regular expression indicating the end of line.
   
  
• Other typical subjects which you might wish to catch include such as
  • cable descrambler
  • FOR SALE
  • laser printer toner
  • million email addresses
  • ONLY $
  • Quit Your Job
• Other typical contents include such as
  • absolutely no obligation
  • call now 24 h
  • to be taken off our list

:0:
* ^From:.*[0-9].+@([-a-z0-9_]+\.)*respectable.net
PotentialSpam.mail

Date: Thu, 19 Dec 2002 10:44:44 +1000
From: Philip Gunter
To: Timo Salmi
Subject: A procmail tidbit

Hi Timo, thanks for your excellent procmail reference.

Here is a small recipe you might like to add to your site.
It limits the number of emails being forwarded from an account,
useful to stop sms storms.

Cheers,
Philip.

:0
{
  :0
  {
    # remove any sms-alert files older than 5 minutes
    GLOP_=`find /var/tmp/sms -name sms-alert\* -cmin +5 -exec rm -f {} \;`

    # Create an sms-alert file for this message.
    GLOP_=`touch /var/tmp/sms/sms-alert$$`

    # Count the number of sms-alert files
    COUNT=`ls /var/tmp/sms | grep sms-alert | wc -l`
    COUNT1=`expr ${COUNT}`

    # Check if number of alerts in the last 5 minutes is less than 2

  }
  :0:
  # if the expression is true then forward the email
  * ISLT ?? ^^1^^
    ! 0123456789@pager.net
}

• The "* > 4000" matches email messages longer than 4000 bytes.
   
  
• The already familiar set of flags "fwh" tells to treat the email's header.
   
  
• Use formail to ensure removing even complicated "Content-Length:" lines.
   
  
• The above also serves as an example of "block nesting", i.e. the rules and actions between the braces "{ }".

• The first 100 lines are included. So are the last 10.
   
  
• The above also exemplifies giving multiple commands. Recall that a standard recipe only allows one action line.

#The executable file named "greptest"
#!/bin/sh
egrep -i '(ts|timo\.salmi)@([-a-z0-9_]+\.)*uvasa\.fi' mail.msg
#
#Allow a quick visual comparison on the screen
echo ""
cat mail.msg

#The mail.msg target file with the trial text for the matching
ts@uvasa.Fi
ts@loisto.uvasa.fi
Timo.Salmi@uvasa.Fi
Timo.Salmi
null@uvasa.fi

• There are some special differences between procmail extended matching rules and the egrep expressions. Thus under special circumstances they do not match the regular expressions quite the same way. This might raise occasional confusion. See "man procmailrc" for the details.
   
  
• You can also test egrep regular expressions on your PC since egrep clones are available from the Garbo program archives. For example you might try gnuegrep.zip, egrep.zip and dgrep.zip. Even better, get the entire, very useful Windows 32-bit UNIX-clone sets UnxUtils.zip and UnxUpdates.zip 

From: scam@cyberspam.com (The Big Bad Spammer)

:0:
* ^From:.*\.com
* !^From:.*\.com\.
* !^TO_(ts|timo\.salmi)@([-a-z0-9_]+\.)*uwasa\.fi
ProbableComSpam.mail

• The first condition line matches a ".com" anywhere on the "From:" address line. It would match, for example, email from "someone@my.company.net".
   
  
• The second condition line tries narrow the condition down, but it still would match e.g. "someone@my.ispcom.net". (Or would it? Anyway, the recipe is not quite accurate.)
   
  
• The third condition line is just standard spam avoidance, not necessarily related to the task at hand. It is just that much, if not the majority of spam appears to involve .com addresses.

# Get the sender's address
# Discard any leading and trailing whitespaces
FROMADDR_=`formail -rt -xTo: \
           | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

# Test if the email came from the .com domain
:0:
* $ ? echo ${FROMADDR_} | egrep -is '\.com$'
ComDomain.mail

• Let formail take care of finding out from the headers what the sender's address is. Get rid of any leading and/or trailing white spaces using "expand" for tabs and "sed" for the remaining spaces. You should have this definition high up in your ~/.procmailrc
   
  
• The "$" on the condition line tells to expand any variables on the line. In this case the "${FROMADDR_}" instead of taking in literally.
   
  
• As far as I understand, the "?" executes a line (and tells to transmit an exit code, but that is beside the current point). BTW, if you have the procmail extended diagnostics on ("VERBOSE=yes") you can get in your procmail logfile a sinister looking "Program failure (1)". Don't panic. It just is egrep's exit code telling that no match was found for that particular email message, i.e. that it was not from the ".com" domain.
   
  
• The condition line echoes the stripped email address to "egrep" in order to test if there is a match. The "-i" switch is used since email addresses are case insensitive. The essence of the "egrep" is the trailing "$" matching the end of the extracted address. The "-s" switch tells egrep to work silently, i.e. only to give the return code.

:0:
* ^From:.*\.hk|\
  ^From:.*\.kr|\
  ^From:.*\.tr
* !^From:.*\.hk\.|\
  !^From:.*\.kr\.|\
  !^From:.*\.tr\.
* !^TO_myid@([-a-z0-9_]+\.)*myhost\.mydom
ProbableSpam.mail

* ^From:.*\.(hk|kr|tr)

#Define getting the sender's address
#Discard any leading and trailing whitespaces
FROM_=`formail -rt -xTo: \
         | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

#Whatever other recipes in between.

#Spam screening of certain susceptible domains
:0:
* -1^0
*  1^0 $ ? echo ${FROM_} | egrep -is '\.hk$'
*  1^0 $ ? echo ${FROM_} | egrep -is '\.kr$'
*  1^0 $ ? echo ${FROM_} | egrep -is '\.tr$'
*  1^0 !^TO_myid@([-a-z0-9_]+\.)*myhost\.mydom
ProbableSpam.mail

#Accept all email from myself, weed out autoreplies
:0:
* ^From:.*myid@([-a-z0-9_]+\.)*myhost\.mydom
* ! ^X-Loop: myid@myhost\.mydom
${DEFAULT}

:0
* ? formail -x"From" -x"From:" -x"Reply-To:" -x"Errors-To:"\
    | egrep -i "scam@cyberspam\.com"
/dev/null

• The "?" at the start of the condition executes and evaluates what is on the condition line instead of searching for a literal match.
   
  
• Use formail to extract from the headers.
   
  
• The "-x" switch means extract the contents a header-field from the header. Formail is convenient (also) because it can concatenate the potential continuation lines in a header-field.
   
  
• Pipe the results to "egrep" regular expression search. The "-i" switch tells egrep to ignore the lower/uppercase status of the target string.
   
  
• Incidentally: Since we discard the email message to "/dev/null", file locking ":0:" must not be used.
   
  

• The first "$" on the condition line tells that the environment variable(s) on the line are to be expanded, instead of taking all the text on the condition line literally.
  

:0
* ? formail -x"Received:"\
    | egrep -i "cyberspam\.com"
/dev/null

# Define getting the sender's address
# Discard any leading and trailing whitespaces
FROM_=`formail -rt -xTo: \
         | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

# Test if the sender could not be identified at all
:0:
* FROM_ ?? foo@bar
NoSender.mail

:0:
* ^From[  ]*MAILER-DAEMON
* ! ? formail -x"From:" | egrep -is "[a-z]"
ProbableSpam.mail

• The first condition is to check the first From line in the header.
   
  
• The [] contains a space and a tab.
   
  
• In the second condition the "!" is the familiar operator indicating a negation.
   
  
• The "?" tells to execute and evaluate what is on the condition line instead of searching for a literal match.
   
  
• formail's -x"From:" extracts the From: header contents (without the field name).
   
  
• Unix egrep is used to test whether the "From:" field exists and contains at least one ordinary letter, upper or lower case ("i"), working silently ("s").
  

REPLYTO_=`egrep "^Reply-To:" | head -1 \
         | formail -c -rt -xTo: \
         | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

• Assume that indeed you strictly want the address from a "Reply-To:" header. No address in any other header will do. Use egrep to extract the "Reply-To:" header-field from the incoming email.
   
  
• head -1 ensures that only the first occurrence of a "Reply-To:" in the message counts.
   
  
• formail -c -rt -xTo: is a standard, special trick to form a return address. The key is the -r switch which "generates an autoreply header". The "-c" switch concatenates any continued fields in the header.
   
  
• If no "Reply-To:" header is found in the email message, foo@bar will be returned as the address.
   
  
• The last line removes any leading and trailing tabs and blanks from the address.
  

REPLYTO_=`formail -cXReply-To: | head -1 | formail -rtzxTo:`

• "formail -cX" rather than "egrep" in case the header has a different capitalization -- or if the real address is on a continuation line.
   
  
• formail "-z" flag to avoid "expand" and "sed".

• The "-c" switch concatenates continuation lines.
   
  
• The "-X" switch extracts the header-field, preserving the field name.
   
  
• The "-rt", "-x" and "To:" trick prepare a return address.
   
  
• The "-z" switch ensures that a whitespace exists between field name and content.
   
  
• If the Reply-To: header-field is empty or missing, the value of the REPLYTO_ variable will be foo@bar

#Get the sender's address, the generic version
FROM_=`formail -rt -xTo: \
         | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

#Get the sender's host
FHOST_=`echo "${FROM_}" | awk -F@ '{ print $2 }'`

#Build the postmaster's address
FMAST_="postmaster@${FHOST_}"

• At e quick glance it may appear that the "From:" header and the "To:" header have been confused in the formula, but this is not the case. The formail program is asked to ("-r") to prepare a reply header to send email back to the sender. Then that return address is extracted. That is why we have a "-xTo:" since we want to extract where the reply would be sent. That is where we assume that the email came from.
   
  
• In the pipe "expand" is used to replace potential tabs with spaces, and "sed" is used to omit any leading and trailing white spaces.
  

#Get the sender's address, ignore Reply-To:
FROM_=`formail -I"Reply-To:" -rt -xTo: \
         | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

#Ensure a whitespace exists between field name and content
#Comment "Old-" the Content-Length field from all the headers
:0 fwh
| formail -z -i"Content-Length:"

#(whatever else in between)

:0
* From:.*the-mailing-list-maintainer
* ^TO_the@first\.recipient\.edu
{
  :0 fw
  | formail -I"To:" -I"X-" -I"Content-Type:" -I"MIME-Version:"\
    -A "To: Maintainer's long recipient list suppressed" \
  | sed -e '/^This is a multi-part /,/^Content-Transfer-Encoding: /d' \
        -e '/------=_NextPart_/,$d'

  :0:
  ${DEFAULT}
}

• There are two condition lines.
  • Match if it is from the mailing list maintainer.
  • Match if it is for the full mailing list and not only to me personally from the maintainer.
     
    
• Feed ("f") the email message to a pipe of several lines. Tell procmail to wait ("w") for the pipe to finish.
  • Let formail weed out superfluous fields.
  • Append a very brief "To:"-field for your information.
  • Let sed take out any special format information.
  • Let sed weed from the start of the HTML part to the end of the message
     
    
• This example shows the principles, but it is based on the established format of the postings on the particular mailing list. Therefore it is not applicable as such, but you'll have to customize and test it for your own situation. (See the items on test methods on this page.)

:0:
* 1^0 ^Subject:.*make money fast
* 1^0 ^Subject:.*\$\$\$
ProbableSpam.mail

* 1^0 ^Subject:.*make money fast
* 2^0 ^Subject:.*\$\$\$

:0:
* -1^0
*  1^0 ^Subject:.*money
*  1^0 ^Subject:.*fast
*  1^0 ^Subject:.*\$\$\$
ProbableSpam.mail

• The initial score is set at -1. Thus at least two of the subsequent conditions have to be met in order for the entire recipe to match. If none or only one of the conditions is met, the score will not rise above zero.

• procmail \/ operand is used to extract the subject of the email into the reserved MATCH variable.
   
  
• Variables testing "??" is used.
   
  
• Word matching is used applying the word boundaries "\<". Thus "fast" would be matched, but not "faster".
   
  
• If both the words "cash" and "money" appear on the subject line no more than one score point will be awarded.

#Catch potential spam by examining the email route
:0:
* 1^0 ? formail -x"Received:" | egrep -i "157\.161\.140\.2"
* 1^0 ? formail -x"Received:" | egrep -i "199\.217\.231\.46"
* 1^0 ? formail -x"Received:" | egrep -i "212\.106\.213\.36"
* 1^0 ? formail -x"Received:" | egrep -i "216\.154\.1\.82"
ProbableSpam.mail

• As usual, the "?" executes and evaluates what is on the rest of the condition line instead of searching for a literal match. Note the syntax order.
   
  
• Incidentally, there is a subtle catch in using the IP numbers. Assume that you wish to detect the nodes from 216.154.1.74 through to 216.154.1.86. This rule won't work quite right: "216\.154\.1\.[74-86]". Why? The "[74-86]" will match 4-8. (The 7 and 6 would be superfluous since they already are within the 4-8 range.) The rule would find matches outside the intended range. E.g. "216\.154\.1\.72" would be matched. Instead, applying both "216\.154\.1\.7[4-9]" and "216\.154\.1\.8[0-6]" would match correctly.

#Avoid a specific forgery spam
:0:
* -1^0
*  1^0 ^From:.*mikerobbins2000@hotmail\.com
*  1^0 ? formail -x"Received:" | egrep -is "psi\.net"
Spam.mail

:0:
* -1^0
*  1^0 ? formail -c -x"Received:" | fgrep -is 'alladvantage.com'
*  1^0 ? formail -c -x"Received:" | fgrep -is 'ameritech.net'
*  1^0 ? formail -c -x"Received:" | fgrep -is 'bellatlantic.net'
*  1^0 ! ^TO_(myid)@([-a-z0-9_]+\.)*myhost\.mydom
ProbableSpam.mail

:0:
* ! ^TO_(myid)@([-a-z0-9_]+\.)*myhost\.mydom
* 1^0 ? formail -c -x"Received:" | fgrep -is 'alladvantage.com'
* 1^0 ? formail -c -x"Received:" | fgrep -is 'ameritech.net'
* 1^0 ? formail -c -x"Received:" | fgrep -is 'bellatlantic.net'
ProbableSpam.mail

• -c Concatenate continued fields in the header.
• -x Get the contents of the said header-field. Do not include the field name.

• -i Ignore upper/lower case distinction during comparisons.
• -s Silent (only produce error messages) in order to check the return status without any output.

:0:
* ! ^TO_(myid)@([-a-z0-9_]+\.)*myhost\.mydom
* ^Received:.*(\
alladvantage\.com|\
ameritech\.net|\
bellatlantic\.net)
ProbableSpam.mail

:0:
* 1^0 ^Subject:([  ]$|$)
* 1^0 !^Subject:
NoSubject.mail

#Get the subject discarding any leading and trailing blanks
SUBJ_=`formail -xSubject: \
       | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

#Test for an empty or missing subject
:0:
* SUBJ_ ?? ^^^^
NoSubject.mail

• ^^^^ denotes empty contents. The trick is adopted from procmail material of some other authors where the ^^ anchor is better explained than what I can do. Also see procmarc.man for it.
   
  
• Likewise, see procmarc.man for the ?? definition.

:0 fhw
* To.*myoldid@myoldhost.myolddom
| formail -i "To: mynewid@mynewhost.mynewdom"

MAILDIR=/home/myid/Mail   #The location of your own mail directory
# Whatever other preliminaries

# Whatever other recipes

# Test if the email's sender is in the blacklisted
:0
* ? formail -x"From" -x"From:" -x"Sender:" \
    -x"Reply-To:" -x"Return-Path:" -x"To:" \
    | egrep -is -f black.lst
/dev/null

• All the common email sender headers are covered.
   
  
• Also the "To:" field is covered in the recipe, since spammers often name their mailing lists as phony addresses.
   
  
• Continuation lines ("\") are utilized. Incidentally, ensure that there are no trailing whitespaces after the "\" on a line.
   
  
• The "-i" option in egrep tells to ignore upper/lower case distinction. The "-s" is for silence. The "-f file" option tells to take the list of the regular expressions from file.
  

abc23@airnewz.ccn
abdu@advis.com.tr
adexec@mail.com
dinner@dine.com
friend@public.com
helpingyou@mail.com
mk1977@ms1.kingnet.com.tw
nb8MAMxhq@mail.com
no@body.com
owieuj@peterlink.ru
patkline00@usa.net
promotions@web-vertise.com
unknown@unknown.com

• The black.lst file should reside in your "${MAILDIR}" mail directory (unless you explicitly include the path in your "~/.procmailrc").
   
  
• The problem with such lists is that most of the spam related addresses are very transient by nature. I do not think such lists alone are a very effective method, as I have explained in my Foiling Spam with an Email Password System measures medley.
   
  
• For an exact matching you might wish to use e.g. "no@body\.com" instead of "no@body.com". Alternatively, one could use fgrep (fixed grep) or grep -F
  

#Get the sender's bare email address from the first "From" line
FROM_=`formail -c -x"From " \
         | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g' \
         | awk '{ print $1 }'`

#Get the original subject of the email
#Discard superfluous tabs and spaces
#On some systems -xSubject: has to be -x"Subject: "
SUBJ_=`formail -c -xSubject: \
         | expand \
         | sed -e 's/  */ /g' \
         | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

#Whatever other recipes you'll use

:0
* ^From:.*infolist@([-a-z0-9_]+\.)*infohost\.infodom
# Avoid email loops
* ! ^X-Loop: myid@myhost\.mydom
{
  :0c:   #Preserve a copy of the email
  Infolist.mail
  :0fwh  #Adjust some headers before forwarding
  | formail -A"X-Loop: myid@myhost.mydom" \
            -A"X-From-Origin: ${FROM_}" \
            -i"Subject: $SUBJ_ (fwd)"
  # Forward the email
  :0
  !mydept@myhost.mydom
}

SENDMAIL=/usr/sbin/sendmail
FROM_=`formail -c -I"Reply-To:" -rt -xTo: \
  | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`
SUBJ_=`formail -xSubject: \
       | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

# Make a copy of all email to my second address
:0
* ! ^X-Loop: myid@myhost\.mydom
{
  :0c:${HOME}/procmail.lock
  | formail -A"X-Loop: myid@myhost.mydom" \
    -I"Subject: ${SUBJ_} [autofwd]" \
    | ${SENDMAIL} -f"${FROM_}" my2ndId@my2ndHost.mydom
}

• The spamfoiling page has a further example.
   
  
• The "-r" option tells formail to generate an auto-reply header.
  

#Note: On some systems -xSubject: has to be -x"Subject: "
SUBJ_=`formail -c -xSubject: \
       | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

:0
# Was it to me
* ^TO_myoldid@myoldhost\.myolddom
# Ignore messages for daemons
* ! ^FROM_DAEMON
# Avoid email loops
* ! ^X-Loop: myid@myhost\.mydom
{
  :0 c
  ! myid@myhost.mydom
  :0:dejatold.lock
  | formail -rD 8192 dejatold.cache
  :0 eh
  | (formail -r \
     -A"X-Loop: myid@myhost.mydom" \
     -I"Subject: Changed email address" ; \
     echo "Dear Sender," ; \
     echo "" ; \
     echo "Thank you for your email about" ; \
     echo "\"${SUBJ_}\"" ; \
     echo "" ; \
     echo "My email address has changed." ; \
     echo "Old: myoldid@myoldhost.myolddom" ; \
     echo "New: myid@myhost.mydom" ; \
     echo "Your email has been forwarded to my new address." ) \
     | /usr/lib/sendmail -oi -t
}

• The "-r" switch prepares s reply header for sending email back to the sender.
   
  
• The "-D maxlen idcache" switch in "-rD" controls the message identification cache. For more see "man formail"
   
  
• The "c" flag in ":0 c" tells that the processing should continue also after this particular recipe has been acted upon.
   
  
• The "e" flag in ":0 eh" decrees that recipe only executes if the immediately preceding recipe failed
   
  
• The "h" flag in ":0 eh" tell to feed the header to the pipe. Put since it is the default, it is not compulsory.
  

Flags can be any of the following:

B   Egrep the body.

:0 B:
* mypassword
password

All the best, Timo

Category: aaa

Subcategory: bbb

File: ccc

All the best, Timo

:0
* ^Subject:.*Whatever
{
  :0
  {
  MAILTO_=`sed -e '1,/^$/ d' \
           | egrep "mailto:" \
           | tail -1 \
           | expand \
           | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g' \
           | sed -e 's/[^o]://g' -e 's/^://g' \
           | awk -F: '{ print $2 }' | awk '{ print $1 }'`
  }
  :0:
  WhichEverFolderYouWant
}

• The sed -e '1,/^$/ d' extracts the body of the email message (i.e. the headers are ignored).
   
  
• The egrep "mailto:" finds all the lines containing mailto:.
   
  
• If there are several mailto: lines the tail -1 gets the last of them.
   
  
• The expand expands any TAB characters to SPACE characters.
   
  
• The sed -e 's/^[ ]*//g' -e 's/[ ]*$//g' omits any leading and trailing blanks.
   
  
• The sed -e 's/[^o]://g' -e 's/^://g' weeds out from the same line the possible preceding colons (:) which might cause confusion. It is not perfect, though.
   
  
• The awk -F: '{ print $2 }' gets the rest (until the end of line or the next colon) after the colon (:), i.e. the email address from the mailto: line and what may come after it. The awk '{ print $1 }' discards the potential rest of the line starting with the first blank after the address. What should thus be left is the email address in the mailto: field.
  

:0
{
  :0 fhw
  | cat - ; \
  echo "===== Filtered email ====="
  :0:
  ${DEFAULT}
}

:0
{
  :0c:
  ${DEFAULT}
  :0 fhw
  | cat - ; \
  echo "======= Forwarded Mail =========="
  :0
  !forward@myhost.mydom
}

# Discard loops
:0
* ^X-Loop: myid@myhost\.mydom
/dev/null

:0
{
  :0c:
  ${DEFAULT}
  :0 fhw
  | cat - ; \
  echo "======= Forwarded Mail =========="
  :0 fhw
  | formail -A"X-Loop: myid@myhost.mydom"
  :0
  !forward@myhost.mydom
}

# Matching a few undelivery and such reports
:0:
* ^Subject:.*Undeliver(ed|able) (e)?mail|\
  ^Subject:.*Returned (spam )?(e)?mail
* ^TO_(myid|firstname\.lastname)@([-a-z0-9_]+\.)*myhost\.mydom
Returned.mail

• Undelivered mail
• Undeliverable mail
• Undelivered email
• Undeliverable email
• Re: Undelivered mail
• etc...

• Returned mail
• Returned email
• Returned spam mail
• Returned spam email
• Re: Returned mail
• etc...

* ^Subject:[  ]+Undeliver(ed|able) (e)?mail

• cyber.com
• cyber.com.au

:0:
* ^From:.*cyber.com([^\.]|$)
ProbableSpam.mail

* ^From:.*myid@([-a-z0-9_]+\.)*myhost.mydom
* ^From:.*myid@([-a-z0-9_]+\.)?myhost.mydom
* ^From:.*myid@([-a-z0-9_]+\.)+myhost.mydom

1. myid@myhost.mydom
2. myid@subhost1.myhost.mydom
3. myid@mypc.subhost1.myhost.mydom

• The first one does not match myid@.myhost.mydom (and neither should it!).
• The second one matches 1 and 2, but not 3.
• The third one matches 2 and 3, but not 1.

VAR1_=Whichever expression you devise
:0:
* VAR1_ ?? regexp
wherever

VAR1_=Whichever expression you devise
VAR2_=whatever
:0:
* $ VAR1_ ?? ${VAR2_}
wherever

        * VAR1_ ?? $ ^^$\VAR2_^^

* ()</body>

or

* (<)/body>

or

* (</body>)

or

* \</body>

#Get the subject discarding any leading and trailing blanks
#Note: On some systems -xSubject: has to be -x"Subject: "
SUBJ_=`formail -xSubject: \
       | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

:0
* YourFirstSelectionCriterion
{
  :0 fwh
  | formail -I"Subject: WhateverYouAdd_1 ${SUBJ_}"
  :0:
  YourFirstFolder
}

:0
* YourSecondSelectionCriterion
{
  :0 fwh
  | formail -I"Subject: WhateverYouAdd_2 ${SUBJ_}"
  :0:
  YourSecondFolder
}

1. Have you copied right? For example:
    
   
   • If you cut and paste, the brackets [] containing tabs will not be copied correctly, since on this page the assumed tabs aren't true tabs.
      
     
   • Make sure that you have not misinterpreted the meaning of the quotation (") marks anywhere in the advice.
      
     
   • If you have a backslash \ at the end of the line to continue the line, it is very important to ensure that you do not have white spaces after the \ backslash.
      
     
2. Have you customized all your file-paths right? Some of the recipes may require a slightly different setup in your environment than assumed in this FAQ.
    
   
3. Check that procmail is getting your proper path. Try "echo ${PATH}" and then include "PATH=WhatYouGot" high up in your ~/.procmailrc recipe file.
    
   
4. Include "VERBOSE=yes" high up in your ~/.procmailrc recipe file. Then see what is in the logfile procmail produced for debugging. The testbench is a useful aid in the debugging.
    
   
5. The shell you use may affect some actions. Check where your Bourne shell sh is with "which sh". If it is e.g. /bin/sh then include "SHELL=/bin/sh" at the beginning ~/.procmailrc recipe file and see if anything changes. Bourne shell is the shell I have used in preparing this tips page.
    
   
6. Work systematically. Try to pinpoint which particular line is causing the offense and how. If the problem is with the condition part make general enough a version to get it match. Then narrow it down towards what you wanted until the recipe fails. If the problem is with an action, try to separate whether the problem is with the actual action or your procmail syntax. For example if you pipe the email to a program, try to separate if it is the call syntax that is in error (e.g. do you manage to convey the parameters right) or if it the actual program you called that fails.
    
   
7. If you have a procmail problem which you can't solve after trying properly, post your problem to the comp.mail.misc Usenet newsgroup and/or your corresponding local newsgroup. If you have genuine feedback about my procmail tips, your email is most welcome, but please refrain from using email for private consultation requests.

#Note: On some systems -xSubject: has to be -x"Subject: "
SUBJ_=`formail -c -xSubject:`

# Responses to filter reports
:0:
* -1^0
*  1^0 $ ? echo \"${SUBJ_}\" | fgrep -is 'Re: Filter report'
*  1^0 ^TO_myid@([-a-z0-9_]+\.)*myhost\.mydom
Response.mail

• In the example the email's subject header is put into a "SUBJ_" variable utilizing formail "-x" option.
   
  
• The "-c" option is used to concatenate the potential continuation lines, since occasionally the headers are divided onto several lines. This is more common on the "Received:" line, but can also occur on the "Subject:" line.
   
  
• If the quoted quotes (\") are not used in the echo, the special characters on the email's Subject line in the header will be processed as shell related operators. This must not be allowed, since it will result in errors that may be hard to trace. For example operators such as "(", ")", "`", "'", "<", ">" and "|" all have a special meaning to the shell.
   
  
• It is safer to use fgrep (the fixed-character expression search) because fgrep interprets also the regular expression special characters literally. For example, for fgrep you could use fgrep 'myhost.mydom' instead of egrep "myhost\.mydom". BTW, as you gather from the example above, procmail uses egrep-like syntax.
  

#Note: On some systems -xSubject: has to be -x"Subject: "
SUBJ_=`formail -c -xSubject: \
         | expand | tr '\;\|\$\`\\]/' '     ' \
         | sed -e 's/  */ /g' \
         | sed -e 's/(/\\\(/g' -e 's/)/\\\)/g' \
         | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

• The potential tabs are expanded.
   
  
• Some of the problem special characters are substituted with spaces.
   
  
• Multiple spaces are substituted with a single space.
   
  
• Parentheses are covered with backslashes "\". Here things can get really complicated, since the number of backslashes must be compatible with the number of interpretation rounds through procmail and the shell.
   
  
• The last sed gets rid of any leading and trailing whitespaces.
  

SHELL=/usr/bin/sh                 # Use Bourne shell
MAILDIR=${HOME}/Mail              # Customize as appropriate
LOGFILE=${MAILDIR}/procmail.log   # Your procmail log
VERBOSE=yes                       # Produce full information
LOGABSTRACT=all                   #       - " -

# Discard probable spam mail, set 1
:0:
* ! ^TO_(myid)@([-a-z0-9_]+\.)*myhost\.mydom
* 1^0 ^From:.*alladvantage.com
* 1^0 ^From:.*ameritech.net
* 1^0 ^From:.*bellatlantic.net
ProbableSpam.mail

:0
* ! ^TO_(myid)@([-a-z0-9_]+\.)*myhost\.mydom
* 1^0 ^From:.*alladvantage.com
* 1^0 ^From:.*ameritech.net
* 1^0 ^From:.*bellatlantic.net
{
  :0
  { RULE="Discard probable spam mail, set 1" }
  :0:
  ProbableSpam.mail
}

#!/usr/bin/sh
grep "Assigning \"RULE=" ${HOME}/Mail/procmail.log

• The location of the lockfile invocation ":".
   
  
• Above the RULE="..." line there is no cloning "c" flag in ":0" since setting a variable is a non-delivering action. The next line will be reached anyway. In fact, it would be a mistake to use a "c" there. It would lead to complications.
   
  
• In setting the RULE variable ensure that there is space after the "{" and prior the "}". Otherwise the email will go to a folder with rather a long and complicated name.
  

:0
* ^TO_my-mailing-list
{
  :0
  * ^From:.*@([-a-z0-9_]+\.)*myhost\.mydom
    {
      :0
      { RULE="To my-mailing-list, probably legitimate" }
      :0:
      ${DEFAULT}
    }
  :0E
    {
      :0
      { RULE="To my-mailing-list, probably spam" }
      :0:
      Spam.mail
    }
}

#Troubleshooting:
VERBOSE=yes
LOGABSTRACT=all

VERBOSE=no

LOGABSTRACT=no

BODY_=`sed -e '1,/^$/ d' | head -5 | tr -d '\n'`

BODY_=`sed -e '1,/^$/ d' | tr -d '\n' | dd bs=1 count=160`

• The b flag means feed the body to the pipe.
• The f flag means use the pipe as a filter.
• The w flag means wait for the filter or program to finish.
• This is not a sed FAQ, but in brief:
  • In the sed script the /^-- / matches the first occurrence of the signature designator string "-- ".
  • In sed, a lone $ stands for the last line.
  • The d denotes deleting the "pattern space" found.

• The sed FAQ Eric Pement
• SED - the stream editor Sven Guckes
• Manipulating Strings with sed
• SED Patrick Hartigan
   
  
• comp.lang.awk FAQ Russell Schulz
• Awk -- A Pattern Scanning and Processing Language Aho, Kernighan, Weinberger
• Awk programming links The University of Edinburgh
• How to Use AWK Patrick Hartigan

Programming
	Timo's assorted NT/2000/XP/.. CMD.EXE Script Tricks  NT/2000/XP command line programming material links
	MS-DOS batch programming material
	Turbo Pascal programming material
	Unix Bourne shell scripts programming material
Etc
	More links to Timo's FAQ materials

# Discard potential email loops
:0
* ^X-Loop: myid@myhost\.mydom
/dev/null

:0
* Any rule(s) you might wish to have
{
  # Keep a copy, but don't stop yet ( the c )
  :0c:
  ${DEFAULT}

  # Comment with "Old-" the Content-Length field from the header
  # Ensure that a whitespace exists between field name and content
  :0 fwh
  * ^Content-Length:
  | formail -z -i"Content-Length:"

  # Add the loop avoidance
  # ( f for piping; w for waiting for completion; h for headers )
  :0 fwh
  | formail -A"X-Loop: myid@myhost.mydom"

  # Truncate the body ( the b ) to five lines
  :0 fwb
  | head -5

  # Forward to the other account
  :0
  ! myid2@myhost.mydom
}

:0 fwbi

| head -10

:0
* ^Subject:.*\/\[TS[0-9]+\]
{
  :0 fhw
  | cat - ; \
  echo "Subject id: ${MATCH}"
  :0:
  ${DEFAULT}
}

* ^Subject:.*\/\[TS[0-9]+\].*

# Probable spam mail, by message body
:0B
* $ ? fgrep -is -f BlackList.lst
{
  :0
  * KEEPSPAM ?? ^^yes^^
  {
    :0:MyProcmail.lock
    | sed -n 1,100p >> Spam.mail
  }
  :0E
  {
    :0
    /dev/null
  }
}