view · edit · print · history

I receive a lot of emails with attachments which are essentially encrypted bank and other financial statements on a special account that I had setup. All these statements have to be cataloged and archived. This was something I used to do every weekend and finally I decided to get it done right - ie have my \SLUG take care of this administrative detail of downloading and cataloging it every day and preparing a report that I can look at later.

The solution involved orchestration of fetchmail + procmail + scripting of my own. The fetchmail downloads the mail from the POP3? account and forwards it to procmail. Procmail then applies its rules to forward the mail to my custom program which then downloads the attachment and saves it to the right directory. It also generates a summary file that contains information about the mail received and the attachments. This summary can be reviewed by me once a week to check if everything is in order.

If you are still curious and read along so far, then here is how you set things up.

Setup Fetchmail and Procmail

First you have to install fetchmail and procmail on your slug. You will find the instructions for installation and configuration of fetchmail at SetupFetchMailWithGMail. If your ISP mail server is more easier to deal with than google you should look at SetUpAnEmailServer and follow the instructions for Fetchmail setup.

Configure Fetchmail and Procmail

This section has to be a rant since the amount of pains that procmail put me through to configure and manage is too hard to explain. Finally I had to accept defeat and made procmail forward all emails to a custom script that I wrote which would do the bulk of the "real" processing. So my procmail rule looks something like this

# .procmailrc
# routes incoming mail to appropriate mailboxes


I suspect that this kind of usage pretty much makes procmail useless and can probably be removed from the chain but that is not something I did and decided to live with procmail configured.

Processing attachments

Ideally I wanted each attachment to be catalogued, stored in a separate directory and a report prepared on a monthly basis that tracked the statements that were received for the month. So instead of trying to create a full blown report, I decided to just generate a flat file that I could process further to get the kind of report that I needed. Essentially for each email, I unpacked it and then write out a log about the email.

The shell script that does all the processing is given below.


TMPFILE=`/bin/mktemp /tmp/filtermail.XXXXXX`
DATEDIR=`/bin/date +%Y-%m-%d`
export SENDER=`tee $TMPFILE | formail -zxFrom: -zxReply-To:|cut -f2- -d'<' | cut -f1 -d'>'`
SUBJECT=`/bin/cat $TMPFILE  | formail -zxSubject:`

/bin/mkdir -p $ATTACHDIR/$SENDER/$DATEDIR >/dev/null
while read decodedfile;do
        /bin/echo $SENDER~$DATEDIR~$SUBJECT~$ATTACHDIR/$SENDER/$DATEDIR/$decodedfile >>$ATTACHDIR/mails
/bin/cat $TMPFILE
/bin/rm $TMPFILE

Tying it all together

Finally, we need all this to happen magically on a daily basis. So I tried to configure fetchmail on a new rc scipt called rc.fetchmail which is supposed to start fetchmail directly. The contents of the script are



Thats all there is to it. This script should be called rc.fetchmail and should be saved in /etc/rc.d/ directory.

Hope this article helps you setup a good email download capability. Ideas suggestions ? just drop me a note - I have an account on gmail called venugopalan.vivek

view · edit · print · history · Last edited by vivekv.
Originally by vivekv.
Page last modified on January 17, 2009, at 10:52 AM