There are many solutions providing encryption, anti-spam, authentication and others available on top of the venerable SMTP protocol. Some of these require management overhead, others require end-user input. But the holy grail is to provide all these features with no user input and low management overhead.
Basics
The most important information needed before starting with any anti-spam or advanced email solution is an understanding of the message triplet. The triplet is made up of:
- sender email address
- sender server ( IP/host-name )
- recipient
You will see later on in this document how important the triplet is but in essence, these 3 values constitute the limits of what can be used for address-based services ( eg. rDNS, blacklist, etc. ).
TLS SMTP encryption
TLS is an extension of SSL designed for, among others, SMTP traffic.
It’s important to understand that TLS does not provide any form of anti-spam or authentication. It’s primarily there to encrypt communications between clients and servers, as well as between servers. It doesn’t care who is sending to whom, it simply takes the MIME envelope ( a binary composition of the email ) and encrypts it with an agreed upon cipher, if the recipient system supports it.
Anti-spam options
Let ‘s take a look at some of the AS options that are available to us.
address-based AS options
There are quite a few address-based options available for AS purposes and these should be implemented as the very first step in a full AS solution.
- black-/white-list – this is a simple list which can block specific sender or recipient addresses based on either the full email address or the domain.
- reverse DNS (rDNS) – when the sender server connects, it’s DNS host-name and IP address are provided as part of the SMTP transaction; the recipient server can do a rDNS/PTR lookup for the host-name ( ie. for a particular host-name, give me the IP address ), and then compare that to the IP address of the connecting server; if the 2 do not match then either the recipient server is misconfigured or is being spoofed.
- Real-time Blackhole Lists (RBLs) – these are internet-based lookup lists which maintain known spam sources ( in IP address form ); when the sender server connects to you, your server checks to see if the recipient server’s IP address is in the configured RBLs; if it is then the connection is blocked before the setup is complete.
- SASL authentication – usually used for client to server connections to send email; if you are not authenticated, then you can’t send email.
- invalid host-name/non-fqdn host-name – if the sender server does not provide a fully formed and valid DNS host-name, then the connection is terminated before setup completion.
- unknown sender domain – the recipient server checks that the sender domain is in fact valid; if not, then terminate the connection.
- unknown recipient domain – same as above but for the recipient.
- HELO/EHLO checks – determine if the sender server is who they say they are according to DNS
The above options can provide a powerful solution for address-based AS. However spammers have become very clever over the years and a large percentage operate valid mail servers which can successfully bypass some of the protections above.
content AS
Most MTAs have the ability to block email based on content – eg. phrase or words
heuristic AS options
Heuristics play an important and powerful role in AS as these systems can intelligently ( based on email address and content behaviours ) identify spam.
Heuristic-based solutions use a scoring system and advanced statistical analysis, breaking the email down into component pieces and then assigning sub-values to these pieces. These are then totaled and if greater than a predefined score, the email will be marked as spam. Some of the component checks are:
- local and network tests to identify spam signatures – the AS can determine the difference between users sending email from inside or outside your network
- DCC, Razor, Pyzor – online email hash sharing databases – email hashes are checked against online databases; if a hash matches, then the email is dropped or tagged.
- Bayesian learning ( the big one ) – the AS can automatically learn ( without external input ) what is and isn’t spam
- integration with AV
- integration with SMTP authentication solutions like SPF, DKIM
- URI (RHSBL) blacklists – a blacklist served via DNS to identify UCE/UBE specifically
- RBLs – a blacklist served via DNS to identify spam servers
- training of the BAYES statistical analysis engines can improve the accuracy of the AS
Remember that heuristic AS uses a combination of the above to compute a total score, not the individual items.
grey-listing
Grey-listing makes use of the message triplet to identify a unique sender/recipient combination. If the grey-listing system receives an email where the triplet has not been seen previously, then the recipient server will reject the email with a temporary defer code. If the greylisting system has seen the triplet previously, then it accepts the email.
This system works on the premise that spammers do not operate receiving SMTP servers and as a result will never receive the defer message. As such, they will never resend the email to be accepted by the recipient’s grey-listing system the 2nd time around.
Grey-listing on average will effectively block 90% of spam.
Greylisting systems also support spam traps, where if designated email addresses receive emails, then the sender servers are blacklisted.
client-based AS
Some email clients ( eg. Thunderbird ) have their own built-in AS solutions which can be very useful in conjunction with the server-side solutions. Once again, training will significantly increase the accuracy of the solution.
Message signing/encryption
This is not specifically an AS solution so is listed here in a separate section. This function is performed at the client side ( and in some solutions, a combination of the client and sending server ).
The first requirement for this solution is that people who want to communicate with each other using signed emails, need to:
- each generate a private/public key set
- send their respective public keys to each other
If you don’t know the sender, then you can’t authenticate that the sender is in fact who they say they are. Anyone can generate a private/public key set using any information. This is a major disadvantage to message signing – it’s only useful for people who know/trust each other.
There is also significant management overhead associated with the PKI ( Public Key Infrastructure ) required for message signing. Provision needs to be made for secure storage and retrieval of keys. Lost private keys will mean that you can’t sign or read signed messages.
Public stores/directories are available for storage of public keys.
S/MIME ( Secure MIME ) is the most widely accepted method of digital message signing. Microsoft has the following to say about message signing:
A digital signature attached to an email message offers another layer of security by providing assurance to the recipient that you—not an imposter—signed the contents of the email message. Your digital signature, which includes your certificate and public key, originates from your digital ID. And that digital ID serves as your unique digital mark and signals the recipient that the content hasn’t been altered in transit. For additional privacy, you also can encrypt email messages.
The authentication provided by digital signatures is predicated on the fact that the person who originally generated the keys, is who they say they are so take the above with a pinch of salt.
For use of message signing in Windows AD environments:
As an administrator, you can enable S/MIME-based security for your organization if you have mailboxes in either Exchange 2013 SP1 or Exchange Online, a part of Office 365. To use S/MIME in supported versions of Outlook or ActiveSync, with either Exchange 2013 SP1 or Exchange Online, the users in your organization must have certificates issued for signing and encryption purposes and data published to your on-premises Active Directory Domain Service (AD DS). Your AD DS must be located on computers at a physical location that you control and not at a remote facility or cloud-based service somewhere on the internet. For more information about AD DS, see Active Directory Domain Services.
Outside of Windows AD environments, you can use Enigmail with Thunderbird.
Authentication
As we’ve seen above with the AS and digital signature options, there is no clear way of confirming the authenticity of the sender.
From Wikipedia:
The need for this type of validated identification arose because spam often has forged addresses and content. For example, a spam message may claim to be from sender@example.com, although it is not actually from that address or domain or entity, and the spammer’s goal is to convince the recipient to accept and to read the email. It is difficult for recipients to establish whether to trust or distrust any particular message or even domain, and system administrators may have to deal with complaints about spam that appears to have originated from their systems but did not.
SMTP authentication solutions are specifically designed with this in mind.
SPF -Sender Policy Framework
SPF is a simple and elegant solution to authentication and does not require any configuration or adjustment of clients.
To implement SPF, one needs 2 items:
- a TXT DNS record with SPF arguments
- an SPF plugin or ability on your email server ( when receiving email )
The primary purpose of SPF is to generate the TXT DNS record with a list of the IP addresses of all servers which can send email for your domain.
When an email is initiated, the recipient server will do a DNS lookup on the sender domain’s SPF record to determine which servers are responsible for sending email for that domain. If that lookup matches the actual IP address of the sending server, then the transaction is authenticated.
In this case, the IP address of the sending server is the 1 item in an email exchange that can not be spoofed.
SPF provides options for different levels of authentication ( PASS, NEUTRAL, SOFTFAIL, FAIL ). These can be used in conjunction with testing or for sender domains which do not publish SPF information.
DKIM – Domain Keys Identified Mail
DKIM provides a similar solution as SPF except it uses digital certificates instead of IP addresses to confirm authenticity.
Digital certs are arguably more secure ( ie. the cert is authenticated by a CA ) than using IP addresses but in practice, the 2 solutions are very similar.
There are 2 parts to DKIM
- signing – the sender server, with the help of a DKIM plugin or feature, signs the outbound email with its private certificate
- verifying – the recipient server compares the provided certificate against the certificate published in DNS – if a match, then the email is authentic
DMARC – Domain Message authentication, reporting and conformance
DMARC is essentially a combination of SPF and DKIM. The admin for a domain will create a policy defining SPF, DKIM or both, and how failures are managed. It also provides a reporting mechanism ( aggregate and forensic ) on the actions performed in the policy, allowing the recipient to detail messages that pass and/or fail.
These reports are an advantage of DMARC, as neither SPF or DKIM will provide feedback on passes or failures ( except for the SMTP logs ).
The purpose of these authentication systems is to make sure that the sender is who they say they are. In practice, these work quite especially considering that up to 50% of all email servers support 1 or the other solution.
3rd party
There are some 3rd party email client plugins that perform a form of grey-listing.
What can you do to protect yourself?
If you are running your own email server(s), then you need to at the very least, implement address-based AS options including grey-listing and RBLs. The next step is to implement heuristics-based AS – if you’re running MS Exchange then you can use a Linux-based front-end with SpamAsssasin to do this. Alternately there are some commercial solutions that run on Exchange servers.
The next step is domain authentication with SPF, DKIM or DMARC. Note that MS Exchange itself does not have the ability to check SPF/DKIM for incoming emails from other domains. If you run Exchange then you can only create an SPF/DKIM record for your own domains so that others can check it when receiving email from you.
Exchange Online/Office 365 and many other online services like GMail, Yahoo mail, MimeCast, Messagelabs, etc. support SPF and DKIM.
If you need to authenticate incoming email from other domains, then you need to run a FOSS SMTP server that supports SPF/DKIM/DMARC plugins ( eg. postfix ).
There are also other solutions like PGP/SMIME gateways, certified email ( eg. OpenPec )and other libraries but they are beyond the scope of this article.
What else can you do?
- increase the intensity of the AS heuristics solution
- learn to read message headers – user training (SAT) is crucial for end-users to identify spam email
- don’t click on attachments or links
- use additional AS solutions like Fortinet’s FortiMail or Cisco IronPort
- push your email through a 3rd party message scrubbing solution like Mimecast
Conclusion
There are many solutions available to solve a number of issues with email including spam, authenticity and delivery. Email as a system, was never designed with reliability in mind, and the spectre of spammers and malware actors, means that the industry needed to respond in kind.