We've been delivering MDJ and MWJ to paying subscribers by E-mail for over ten years now, so we've become familiar with the rules by necessity. Much of what you read online about "E-mail deliverability" is aimed at marketers who are trying to send millions of E-mail messages at once. Most of those are requested (or at least tolerated, from companies you've done business with), but some are not—some are just spam.

Those discussions rarely apply to MDJ and MWJ because our subscribers have paid to receive our journals. Yet E-mail is much like the US Mail: once you send your letter, the actual delivery is out of your hands.

Think about it. When you mail a letter or parcel, the act of putting it in the mailbox or handing it to the clerk is the last step you take. You don't travel with your letter, you don't accompany the delivery person, and you certainly can't make sure that the recipient actually opens your letter once it's delivered. It might sit on a desk somewhere for weeks while the recipient yells at you, "Where is it?" All you can do is verify that you sent it—and, if you have package tracking, that it was delivered. The rest is up to other people.

So it is with E-mail as well. When you send E-mail, you don't send it directly to the recipient's computer—it might be offline, or the E-mail program might not be running, or something like that. Instead, you send E-mail to your outgoing E-mail server. Your server, in turn, talks to your recipient's incoming E-mail server, and your recipient fetches mail from that server. For example, if we sent E-mail to a certain CEO, the sequence looks something like this:

Most ISP users control only the first of these four steps. As a company, we run our own mail server, so we control the first two steps (in fact, we use custom SMTP delivery software based on qmail to deliver issues and track bounces).

Having set the stage, we'd like to make a few points.

1. Once your mail server accepts a message on your behalf, we have no more say in the delivery.

   When our server contacts another server to deliver a message, the two servers speak SMTP (Simple Mail Transfer Protocol) to each other. Our server says, "We have a message for <johndoe@example.com>." The responding server either says "OK, I handle mail for johndoe, send the message" or returns an error, anything from "I don't have a user named johndoe" to "Sorry, johndoe's mailbox is full" to "I'm restarting; try again in 30 minutes."

   If we get an error in the transaction, we either try again later (if that's what the error code says to do), or we're notified that the message bounced—could not be delivered. That's how it's supposed to work. More and more often, though, in the name of "spam filtering," E-mail servers are accepting messages on your behalf and then deleting them—"dropping them on the floor," as it's called—because the contents may be commercial.

   This is not just inconvenient—it's a violation of SMTP standards. If a server accepts a message for you, the standard requires it to deliver that message to you. If it declines a message for you, it is required to tell the sender that it did so, and ideally, why. Yet in the name of spam fighting, way too many server admins allow these standards violations.

   Even worse, we've found that in many cases, postmasters don't even know it's happening. When readers contact us because they haven't received issues, we look at our server logs. At least nine times out of ten, we find confirmation that their E-mail server accepted delivery of the issue on their behalf. It looks something like this:

   Aug 5 03:15:05 mercury qmail: 1154765705.682244 starting delivery 340: msg 1133114 to remote reader@example.com

   Aug 5 03:15:48 mercury qmail: 1154765748.631981 delivery 340: success: WW.XX.YY.ZZZ accepted message.

   Remote host said: 250 2.6.0 message received OK

   Note a total time of 43 seconds from start to finish—not too fast, but still decent. Sometimes we get a 4xx error, meaning "try again later." That's how the antispam technique graylisting works: legitimate mail servers will try again for a few days to deliver a message. Spammers, usually taking advantage of compromised PCs, do not. Here's how that looks:

   Aug 5 03:17:31 mercury qmail: 1154765851.497157 starting delivery 506: msg 1133188 to remote reader2@example.com

   Aug 5 03:17:47 mercury qmail: 1154765867.280820 delivery 506: deferral: AAA.BB.CCC.DDD does not like recipient.

   Remote host said: 450 <reader2@example.com>: Recipient address rejected: Policy Rejection- We are using greylisting. Please try again in 3 minutes.

   Giving up on AAA.BB.CCC.DDD.

   Aug 5 03:31:04 mercury qmail: 1154766664.374165 starting delivery 827: msg 1133188 to remote reader2@example.com

   Aug 5 03:31:14 mercury qmail: 1154766674.884008 delivery 827: success: AAA.BB.CCC.DDD accepted message.

   Remote host said: 250 Ok: queued as 0A2381FE51C

   In this example, the "450" status code told the server to try again later. Our server tried again after about 14 minutes, not in 3 minutes as requested (because servers don't parse those human-readable messages), and true to its word, the remote server accepted the message the second time.

   You would be dismayed at how many times we've had to go hunting through server logs to find these transcripts and send them to readers, who then in turn have to send them to their postmasters, all to prove to the server admin that his server accepted a message and then did not deliver it. Even when confronted with the logs, some admins refuse to admit that their server would ever do that, something we take as a strong indication that they don't even know it's happening.

   This is why we added RSS feeds for MDJ and MWJ subscribers. For a lot of organizations, E-mail is officially broken. We mean large organizations: government agencies, Fortune 500 companies, computing societies, you name it. If the messages bounced, we could track them. If the server refused delivery, we'd know it. When it accepts E-mail for you and then deletes it, we can't fix it. All we can do is provide a way other than E-mail to be notified of issues, and that's RSS.

2. A huge number of mail servers are misconfigured in other ways.

   We fight the war on incoming spam just like every other organization does. We use DNS-based and content-based spam filtering, but keeping in line with the standards, we never accept messages and then drop them. If we reject a message, the sender gets an error. We recognize that, unfortunately, this can cause problems for people whose addresses were stolen by spammers for use as return addresses on spam, but given our experiences described above, we strongly feel that we should continue to obey the standards even in the face of non-standard behavior.

   As it turns out, it's not easy for spammers to obey the standards. Checking for standards compliance can be a simple and powerful way to combat spam. We already mentioned graylisting: spammers usually give up on a 4xx "try again" error code, while legitimate mail servers never do.

   Servers are supposed to identify themselves when sending messages. When an SMTP server sends its "hello" command (HELO for SMTP, or EHLO for extended SMTP, or ESMTP), it's supposed to be followed by the name of the server. Ours says "EHLO mail.gcsf.com". Sure enough, if you look up that name, you get the same IP address as the server sending the mail.

   The standards do not require that this be true, and for good reason. Many machines host multiple domains, and there's no guarantee that looking up a domain name will always return the same IP address as the sending server. Large domains might use round-robin DNS, for example, so looking up something like "mail.apple.com" could return a different IP address each time. But the name should resolve to something.

   Spammers have a hard time faking that for various reasons, so they often use SMTP host names like "friend", "localhost", or even your own server's name or IP address. Simply detecting these host names and blocking those connections can drastically reduce the amount of spam your server receives.

   As a small organization, we try to be aggressive but reasonable in our spam filtering. Since spammers have difficulty faking a reasonable HELO/EHLO name, examining it can be a great way to filter out the obviously bogus connections. Alas, we've tried examining both that name and the name we get from looking up an SMTP connection's IP address, and it's usually too inconsistent to use as a filter.

   • Although the HELO/EHLO name should be the sending mail server's advertised name, a huge number of Microsoft Exchange servers are misconfigured, and instead return their local name as this parameter. So, instead of mail.example.com, Exchange servers seem just as likely to say example-exchange-server.local. You therefore can't look up the SMTP host name and act based on the results—it could be garbage. (It is, however, supposed to be some kind of fully-qualified domain name, so if it doesn't even have a "." in it, it's almost certainly bogus.)

   • One easy filter is to block connections when the SMTP host name is your own server's name. If you run mail.example.com, no one should be connecting to your server claiming to be mail.example.com. Too many servers let this pass, thinking that clients might be misconfigured—but clients shouldn't pretend to be the server. No one connecting to your server should pretend to be your server's name or IP address.

   • Since the vast majority of spam comes from compromised PCs, one easy check should be to see if the message is coming from another SMTP server. Check the diagram above—one client shouldn't talk directly to a destination server. Unfortunately, there's no real way to check to see if a connection is from a "real" SMTP server. We've tried rules that do reverse lookups on the IP address of the connection and try to filter out dynamic IP addresses, but huge ISPs from Pacific Bell to Comcast have this habit of selling static IP addresses but leaving the reverse DNS the same, so looking up 192.168.1.10 would get 10-1-168-192-dynamic-bigISP.com or something similar. We put these rules into production from time to time, but they block too much legitimate E-mail to leave running all the time.

So what can you do to make sure your mail flows as reasonably as can be expected?

• Get your DNS in order!

  At the very least, make sure your mail servers have PTR records that resolve to the primary name of the mail server. Use a free formerly-free tool like DNSReport.com to check all the intricacies of the DNS system for your domains. If you have failures or warnings, make sure you understand them and are OK with them. (We use EasyDNS for our DNS service, and our DNSReport shows (or would, if you could see it) a few warnings related to exactly how EasyDNS runs its servers—they're not functional errors, just a difference from suggested norms, and they're fine.)

• Make your mail server behave

  Section 4.3 of RFC 821 and Section 4.3.1 of RFC 2821, the SMTP RFCs, say that the SMTP host name should be either a fully-qualified domain name or an IP address adhering to a specific format, since not all mail servers are "known to the domain name system."

  Make sure your server does this. If it's a one-word answer, it's wrong. If it's a local domain name that doesn't resolve, make it one that resolves on the Internet. Don't use a ".local" name when you can use something real. Postini, for example, defaults to using the SMTP host name "Postini". That's wrong, and yet you'd be surprised at some servers that haven't changed it didn't change it for a long time.

• Use SMTP authentication in your E-mail client

  When you send mail via SMTP, you're connecting to your server the same way spammers do. Help your server figure out that you're a legitimate user by always using SMTP authentication when you send mail. An increasing number of servers use this as a clue to bypass unnecessary and expensive spam filtering—after all, spammers don't have accounts on your server, so they can't be authenticating with passwords.

• ALWAYS return error messages

  Sysadmins everywhere are grumbling, "But we don't want to bounce spam! It just lets the spammer know that we figured them out and wastes bandwidth!" Maybe so, but the standards require it. A huge number of mail admins don't even know that their servers are dropping messages on the floor when filtering spam. MDJ and MWJ are not the only paid-subscription E-mail services on the Internet—loads of people pay good money to receive E-mail from companies ranging from the humongous to the merely clever. If you're going to throw those messages away, you owe it to your users and to their legitimate senders to report the error.

  Even if you think your server isn't doing this, check on it. We don't have the several hours per week it takes to chase down these log entries and prove to you that your server is dropping messages, and a huge percentage of the E-mail admins we've had to contact denied it was happening until they saw the logs. Some of them even argued with the logs, saying that their servers would never do that—until they tracked the message IDs and found them in their own logs. Know what your server is doing.

Thanks to spam, E-mail is all but broken in a lot of cases, but it doesn't have to be that way. Make sure your E-mail uses best practices so that more servers can filter on things like the SMTP host name and the reverse lookup of the IP address. Graylisting is working out pretty well for the standards-compliant, too, and that's just another reason to obey the standards. The more closely you comply with the standards, the more likely it is that your E-mail will go through and that you'll be able to detect non-standard users like spammers.

Spammers often can't obey the standards because the machines they've compromised won't allow things like real DNS names, or they won't because it's too inconvenient for the attacks they push. Don't make the mistake of breaking the standards yourself—like dropping messages on the floor with no errors—in the war against them. Obeying the standards won't eliminate spam and spam attempts, but it's still the best weapon we have against them. If everyone breaks the standards to fight spam, E-mail will soon be so chaotic that there won't even be a chance to fight spam, and that's just what the spammers want.

Don't throw the baby out with the bathwater. If you take nothing else away from this, make sure that your E-mail server never discards a message without alerting either the sender or the recipient. Every message that vanishes is a tiny blow to the life of E-mail itself.

This Page was last updated: Sunday, April 6, 2008 at 3:19:15 PM
Copyright 2008, GCSF, Incorporated. All rights reserved.