On Email and Spam

Ever since I got on the Internet back in 1994, email has been a staple. Everyone on the Internet pretty much has an email address. In fact, in the early years on the Internet, I had something like seven of them at the same time. Of course, that is insanity but, hey, it was a badge of honour to have more email addresses than my buddies at the time. Other protocols and applications have come and some have gone since 1994 but email has stayed relatively unchanged, at least from a user experience.

Email as it exists today can be likened to the postal mail system. To post a letter, you first write the letter. Then you put it into an envelope and put the recipient’s address, your address, and affix a bribe sufficient to convince the post office to deliver the letter. To send an email, on the other hand, you write the message, attach your address and your recipient’s address to it and hand it off to a mail transport agent (MTA) which takes the address information and turns it into a digital envelope. The difference is that no bribe need be affixed to it and the envelope stuffing is done by your mail user agent (MUA).

Now I don’t know about you, but once I’ve received a letter by post, I remove it from the envelope and discard said envelope. Now I have no idea who the letter is from if it isn’t written on the letter itself. Likewise I don’t know if I’m receiving a copy for my information or if I’m the intended recipient if the letter doesn’t say. Fortunately, there is a clever convention used when writing letters. This involves adding the sender’s name and address, the date the letter was written, and, at the very least, the recipient’s name on the first page of the letter. Since this isn’t a tutorial on letter writing, I won’t go into all the details on what can be done within the normal conventions of writing letters.

Email suffers from the same problem as above except that you don’t actually go through the process of discarding the envelope. That is done by the MTA and/or MUA (depending on the exact mechanics of the system being used to retrieve the mail). Thus email also has a convention for encoding useful information like the sender and intended recipients in what are known as headers. The particular headers related to identity and so on are provided by the sender or sender’s MUA. (There are other headers but they are beyond the scope of this discussion.)

It should be obvious that both schemes have exactly the same deficiency. The sender and recipient information, both on the envelope and inside it, is provided solely by the sender. I can easily put my friend’s return address on the envelope of a letter I send by post. I can just as easily do so on an email message. In both cases, should delivery fail, then my friend will receive the returned mail. Likewise, I can put my friend’s return address inside the mail and put mine on the envelope. Then I get the returned message but the recipient may not notice I was the one who sent the message. The only real difference between doing this by post and doing it by email is the lack of the bribe affixed to the envelope.

I’m sure some of you are saying that the postmark can be used to determine if it really was sent by me or my friend. Well, sure, it can be used to give a clue. But it does not tell you anything certain. It only tells you that whoever mailed the letter used a particular post office. It’s certainly legitimate for my friend to mail a letter from a post office in New Brunswick even though he lives in Alberta. After all, he might physically be in New Brunswick at the time. The purpose of the postmark is to make sure the bribes can’t be reused rather than authentication of the source anyway.

Email has a more detailed track but it is no more reliable. Every system the message passes through adds a sort of postmark to the message. (This is done in a header inside the message; your MUA can actually show this information to you.) However, because these postmarks are not authenticated, any one of these records can be forged and/or provided by the sender of the message. Indeed, nothing prevents a malicious MTA from adding additional bogus marks to a message.

This also brings in another potential problem. You have no way of knowing whether the envelope on a message was replaced in the course of the delivery process, at least in the general case. You might be able to identify handwriting or something like that but in the general case, you have no clear way of determining if the envelope was changed. In email, this is even worse because one can change the previously mentioned postmarks and the recipient would be none the wiser.

The preceding discussion also doesn’t mention anything about the possibility of the contents itself being forged with is eminently doable with both post and email. Indeed, people fall for postal forgeries all the time, just as they fall for electronic ones. The problem just seems to be worse on the electronic frontier because the fraudster must bribe the postal service to deliver the message at a not insignificant cost whereas the lack of a bribe in electronic mailing makes it cheap to send many millions of forgeries and/or scams. Thus we come to the modern plague of the email system.

Spamming, as it has become known, is the act of sending a large volume of irrelevant, fraudulent, undesired, or duplicated messages in a given medium. It has been rampant on Usenet for a long time, even before email really caught on. As email caught on, it became a vehicle for spamming as well. Since then, it has become such a problem that it has been estimated that some 70% of all email traffic carried on the Internet is spam and the ratio may actually be much higher on Usenet.

Indeed, I long for the days in 1994 when having an email address in active use did not instantly guarantee that you would receive spam. Even in 1997 and 1998, spam was not a huge problem. It was about then that I started receiving any amount of it regularly. Since then, it has grown steadily, especially since I consolidated my email addresses into a single address in my own domain. Essentially, I haven’t changed email addresses since then and thus my address has made its way onto most of the big lists of email addresses that spammers use.

So rampant has spam become that much bandwidth is consumed with decrying how much bandwidth is consumed by spam. Not to mention the fallout from such discussions ranging from forecasts of the death of the Internet to just the death of email as we know it. A notable example is an article that appeared in The Register on June 1, 2006 entitled “The time has come to ditch email” and it was that example that prompted me to write this.

In the article, Kelly Martin of Security Focus discusses the spam problem after going through a very brief history of email. He deserves recognition for putting into words so well the frustration so many feel with the current email infrastructure. He makes the very valid point that the email system as it stands cannot ever be free of spam or phishing or what have you. He goes on to suggest that by throwing away the existing SMTP based system and replacing it with something totally new, we might be able to solve the problem. He does not, however, have any concrete idea on how to do that – a fact he readily admits and goes on to make some possible suggestions.

His major contention, however, is that the fact that email is still transported over essentially the same protocol (SMTP) that it was a quarter century ago is what causes the spam problem to be rampant. In this, he is essentially right. However, he goes on to suggest that all the time and effort spent trying to fix SMTP with virus scanning and so on is all wasted and that the current widely deployed infrastructure cannot be fixed. He draws parallels to protocols such as telnet, FTP, and so on, that have been supplanted by more modern protocols. In fact, he outright states that the email system cannot be fixed incrementally but must be scrapped outright and wholesale replaced with something new. I wish him luck in his endeavour to make this happen but if the design and deployment of IPv6 is anything to go by, it will take at least a century to effect such a large scale change and by then, by anyone’s measure, it will be too late if the status quo remains unchanged.

My contention with Mr. Martin is that the current email structure need not be wholesale replaced. With the current wide install base, that is simply impractical. Especially since the new system would have to be incompatible with the current system or sheer inertia will prevent most current installations from upgrading. After all, if you run a large, complex network, would you spend all that time and effort upgrading the email system if your existing legacy system will be able to talk to the current email system? Of course not. But, on the other side of the spectrum, if by installing the new system, you prevent yourself from talking with the legacy system, what value is there in changing? In fact, that, in itself, is a significant disincentive. Since wholesale replacement will likely not be practical, that leaves incremental updates.

Granted, incremental updates are harder to orchestrate and design, but they also do not require a wholesale retraining of the entire administration community. After all, it’s taken most of us years, if not decades, to fully understand how things work now. So let’s examine the parts of the current email system that work well enough. They may not be perfect, but in any human endeavour, good enough is usually sufficient, especially when the cost of getting to perfect is huge.

Separation of addressing and delivery mechanism is probably the most important thing the current system gets right. When you send an email to user@domain, you don’t know how it is going to get there. It might travel by SMTP to a mail server somewhere in Texas, then take a UUCP link to Peru, and then get transported across a different type of network altogether (sneaker net, maybe), followed by another SMTP connection on a disconnected network. However, in the end, all you need to know is user@domain. Likewise, the user at the other end likely need not know anything other than your email address to send a reply. Thus people with email addresses need not actually be directly connected to the Internet anyway. So the common misconception that SMTP is email is not true. SMTP is merely one transport mechanism for email. The user@domain syntax can be used over disparate transport mechanisms without detriment to its utility so long as every user@domain uniquely identifies a recipient. Interestingly enough, email addresses are increasingly being used as identifiers for non-email applications such as MSN Messenger or web site logins for just that reason – they are unique.

Next, we have a simple low level transportation mechanism that does not care about the actual contents of what is transported. At least, this is true of the specifications for the commonly used mechanisms. This is actually a good thing as it makes the communication layer easier to debug and get right. As far as the transport layer is concerned, a message is a message and all get treated essentially the same. This is essentially the same as for sending a package by post or courier, although again, there is no bribe required to convince the transporter to actually do the transporting.

Also, the audit trail that is left inside the message as it traverses each link in the forwarding chain is useful. It originated as a means of debugging the email system but it can be used to trace nefarious activity as well. It can also be used as a major hint as to whether the purported sender of the message is, in fact, the sender. However, because this is part of the contents of the message itself, this means the information cannot be relied upon to be accurate.

Now, the major thing that the current infrastructure gets wrong is that there is no way to authenticate that the message arriving at any given mailbox is from a legitimate sender rather than being forged. This is the critical missing piece from the perspective of combatting spam and scams. Of course, knowing the source is not the magic bullet to stopping such things. However, being able to trace the spams and scams will make it possible to hold the perpetrators responsible and it will also make filtering out bogus messages a lot easier. And, it turns out, this problem is not unsolvable.

Efforts are currently underway to define a method of authenticating the source of an email. Such things as SPF attempt to define where on the network email claiming to be from a particular domain is expected to originate. As long as all users of email in a particular domain send mail from their home mail servers, that is not a problem. This does have some issues with the classic method of email forwarding but that is not totally unsolvable either. In fact, the low tech brute force solution of recreating the envelope and resending the message is the way it should have been done originally. (Of course, the relaying system would need to keep some sort of state to determine what to do with bounce messages so they get routed back to the correct sender.)

Additionally, if each message carried along with it some sort of cryptographic hash or other bit of information that could identify the source unambiguously, it would be much easier to identify the source of a message. Additionally, it would be easier to filter out bogus messages as the unique identifier would not match correctly. This problem is a much harder one to solve because the current solutions require user intervention to do it. Such things as PGP certainly work for this purpose. However, for any solution to be nearly universally deployed, it would require a somewhat transparent process handled by the MUA and there must be some means of sharing the necessary information to verify the authenticity of such unique identifiers. However, all these problems must be faced whether the email infrastructure is replaced wholesale or not and the identifier can be added to the existing RFC2821/RFC2822 email infrastructure without preventing systems which do not understand the new specification from receiving mail (so long as the messages do not get encrypted or encoded in some weird manner).

To summarize, the current email system is broken but it can be fixed without tossing everything we have today. The problem with email is not the transport layer but the endpoints in the communication. After all, you wouldn’t expect the post office to authenticate the exact individual that sent a letter to you to make sure he or she is who he or she says he or she is, would you? That seems to be what people expect from the transport layer of the email system. Rather, with postal mail, you look for context in the package and other clues that indicate the authenticity of the message. The same should be done with email, although much of it can be automated in the MUA.

Of course, in the end, nothing can be done in the email infrastructure to correct for the inherent limitations and general contrariness of the meat creatures that use it. There is where the ultimate flaw in email lies; between the keyboard and the chair. At best we can hope to provide sufficient tools to the meat creatures to allow them to deal with their own limitations and contrariness.

As a post script, I feel I should mention that spamming occurs in other media than the Internet, too. It happens by post and fax although it costs more to send post and fax spam than email. It happens on the streets in the form of billboards. It happens all over. It is just that on the Internet, the cost of doing it is so much lower than other media.

William Astle
June 2, 2006 CE… UTC

Back to Ravings about Current Events index