Frustrate Hackers – Not Your Customers

E-Mail should be obsolete by now according to many social media proponents, yet we still rely on e-mail as a means of communicating complex concepts and expanding our network of contacts. It remains an indispensable tool at work and home, allowing us to filter and prioritize communications and manage actions against them accordingly. This makes e-mail an irresistible medium for scammers and con-artists to try to snare new victims, hence Spam volumes are always rising. We introduced some pretty aggressive filtering via Spamassassin to address this trend, but we’d lose an occasional mail response and needed a solution to allow responses to pass amidst the torrent of unwanted Spam.

Our core messaging infrastructure is based on open source software, which is eminently customizable, but generally not very well integrated as distributed. We needed a way to automatically white list responses to mail sent from our servers, but mail was sent through sendmail, cataloged in dovecot, and incoming mail was filtered through spamassassin. The simplest way of addressing this was to extract the mail “To” and “BCC” addresses in all the “Sent” folders and add them to a whitelist that is referenced by spamassassin.

Initially, we thought of reading the mail from the mailbox files for those folders on the filesystem, but this relies on the folders remaining in an mbox format. We have been considering moving to an mdbox format for more efficient indexing and higher performance and ruled out reliance on the mbox formats on concerns of future compatibility.

We settled on using the doveadm search and fetch tools available in dovecot to get the envelope information on emails in the IMAP Sent folders maintained in dovecot. We could then extract the relevant header information, apply the data to the whitelist, then restart sendmail. We implement this as a script (see below) and run it in a scheduled job on a daily basis An optimization to only read from mail sent since the last whitelist update allows for more frequent execution without introducing extreme overhead on the system.

Initial load took a very long time – in retrospect, running the sed filter in the script against an extract of all the sent mail in mbox file format reduces the initial load time over 95% by eliminating the individual header extract on each mail independently by dovecot.