Nice body, shame about the header
If you read my words about fixing SMTP last month you might want to take a look at an draft RFC document I came across regarding technical issues in dealing with spam.
As well as being a useful summation of the issues involved, it’s also framed (as you might expect of an RFC) in fairly uncontroversial language, which is unusual for a subject for which writers are usually so emotive.
Anyway, one of the things I proposed was rejecting mail without a full complement of completely valid headers. By this I’m referring to a minimal complement of standard mail headers, conforming to their BNF
definitions. (Perhaps going a bit further, treating shoulds as musts in the case of things like Message-ID:)
But when it comes to non-standard mail headers… well that’s, ahem, a different matter. Maybe 90% of the legitimate mail I receive contains non-standard mail headers. I know that 100% of the mail I send contains at least one non-standard header. And I wouldn’t want to encourage the blocking of my mail, would I?
And even in the term “standard” I’m being loose in my definition. I’m referring to any header fully defined in a numbered RFC document, regardless of the status of the RFC itself.
While I moan about the non-standard status of the Precedence: header, I understand why its use is discouraged. For a start, what is it supposed to do? The name itself isn’t that instructive. I believe the original (and apparently undefined?) intention was to tag mail so that bulk mail-outs were given a lower transmission priority than normal mail. Then a hacker (
probably Sendmail author Eric Allman) wrote a piece of auto-responder code that checked for the non-existence of this header before sending out auto-responder messages. It soon became the convention of mailing lists managers that this header should be included to suppress these junk replies. And thus Precedence: becomes the de facto standard header to say “I’m a mailing list don’t activate an autoresponder”, the original transmission meaning (I assume) long since lost.
These days if your “out-of-office” auto-responder code needs a simple check to see if any given mail came via a mailing list it should check for something lie the existence of the RFC-defined List-ID: header. Of course, in practice, there’s a good chance your mailing-list software or auto-responder predate RFC2919 (March 2001). Welcome to the Internet, if you don’t like it file a detailed bug report.
But Precedence: and other non-defined headers were forged in those pioneering days of email. When it was still a experimental. Things are different in the multi-billion dollar Internet of today. After all, we’ve had twenty years to get our shit together.
Yet, oddly it seems there’s still no single standard reference location for standard email headers. Occasionally you’ll get a updated document noting common headers but usually if there’s a header you need, you’ll have to go hunting in the RFC pile until you find it.
When software designers need a custom header many will respectfully use the “X-Something:” naming convention for non-standard headers to avoid clashes with possible future IETF standards. Others, God bless ‘em, don’t.
There is a draft proposal for an IANA-hosted registry of standard message headers. And by the time that happens there are a few non-standard headers I’d like to see standardised:
User-Agent:
There is no standard header for MUAidentification. And yet, by default most MUAs will attempt to stamp one on every message they send.
Perhaps there’s an air of inevitability about a MUA identifying header
becoming standard, and possibly in the choice being User-Agent:. It’s recommended and defined in the USEFOR draft for news articles and if that became standard it would be odd not to adopt this precedence for mail. It’s already in wide usage in some clients. (My MUA-of-choice for example.)
I don’t even know it this has been proposed as part of any mail RFC. If and when it does it’ll be a while, possibly a couple of years, before it becomes a numbered RFC. It’ll be a few years after that when most people have upgraded to software that uses the new standard. Yep, that’s the breathtaking speed of Internet time.
Archive:
Another USEFOR proposal, this one useful for lists. This is essentially the
Dejagoo convention of “X-No-Archive:” but without using a double negative to indicate a positive.
Priority:
Technically this a already described in RFC2156 but only for use in X.400 gateways (such as Exchange) rather than in SMTP at large. Essentially this is what “Precedence:” was – an attempt to influence transmission speed.
It has three values urgent/normal/non-urgent (non-urgent being less urgent than normal, rather than urgent-but-not-as).
Now given bandwidth has increased at proportionally faster rate than email sizes (at least in my world) you might think that the need to prioritise would already be obsolete. However on machines with fairly hefty mail queues to process the ability to prioritise would be nice - for example on backup MXes after the recovery from an extended primary failure, or on servers processing mailing-lists.
Imagine you have two mailing lists kitten-photos@lists.example.com and
timely-financial-news@lists.example.com. kitten-photos usually has pretty large messages and a large subscriber list. You might want to specifically treat “kitten-photos” as “non-urgent” so that it doesn’t take priority over normal mail on the system.
To take an example from my recent past, suppose the list timely-financial-news has two types of subscribers: those that have paid you to receive these messages, and those that have not. You want to have at least attempted a delivery to everyone who paid before beginning delivery of those that haven’t. It would be nice if all that needed to be done was to mark the paid envelopes with a “Priority: urgent” header. Obviously just having a header won’t make mail servers magically able to do this, but rather the idea is to make it obvious what servers that can should expect.
Importance:
This is another X.400 feature and the “human” equivalent of priority (low/normal/high) – a suggestion of the importance of the mail, but with no influence on message transmission whatsoever.
If you only allowed Priority, programmers would probably overload its meaning. Although at college people using the horrible WhiteMail X.400 agent would select “importance: high” assuming that it would mean the message (usually gatewayed into SMTP) would get there quicker. I imagine they’re the same people who repeatedly press lit elevator buttons.
An obvious objection to the Priority: and Importance: headers is that they’re easy to abuse. I imagine in places where users mark all mail as “high importance” (I don’t imagine normal MUAs would need to support Priority:) it becomes meaningless. A typical PEBAC issue.
If high importance becomes normal for you then so be it. I’m more interested in de-prioritising some of my outgoing email. I believe was one of the factors in the success of SMS was that it seemed more polite to send a message that could be replied to at leisure, rather than a phone call that was immediate, urgent. I’m pretty slack when it comes to responding to email, it would be nice if I didn’t have to feel I had too. It would also be another little check for auto-responders.
And it’s ultimately up to postmasters how their systems send mail. For most email systems that aren’t overloaded the “Priority:” header would have little or no effect. The Importance: header is already in use in many systems, but oddly when spammers add it (based a grep of my spam-bin) most appear to set it to “Normal”.
I guess if something is consistently abused just becomes another useful heuristic. Certainly I would expect one of those “legitimate email marketeer code-of-conducts” to specify that messages would be marked as low-importance should it ever become widely adopted in internet MUAs (if it isn’t already).
Sensitivity:
Sensitivity is another X.400 header and has the possible values ”Personal” / “Private” / “Company-Confidential”. But perhaps it’s not what I’m looking for here. I’d want a standard header that, based on it’s value, an
MTA could infer routing decisions such as ”Don’t send this mail over an unencypted link” or “Don’t include the message body if you need to send a
DSN“.
Yeah, in an ideal world sensitive mail would be encrypted.
Mail-Followup-To:
I’m not often a supporter of djb‘s proposals. I don’t support the idea of Mail-Reply-To: becoming a standard because it’s just a hack to work around header munging – making it standard would just be compounding the error. I think the real solution for the Reply-To-munging problem is for people to use smarter MUAs (for example one that checks for a List-Post: header when a reply is requested).
However Mail-Followup-To adds something new and useful to mail by grafting a Usenet convention on to mailing lists. It provides an automated way of saying:
“This message is going to the sos-announce mailing list. Please send followups to the sos-discuss mailing list.”
“I’m a subscriber to this mailing list. Please don’t send me an extra copy of followups.”
or, as implied, (since I believe the normal Usenet follow-up mode isn’t to also mail the poster) ”I’m not a subscriber to this list, please send me any followups.”
Delivered-To:
Another djb-ism. qmail uses it as a form of loop detection. Mailing list managers have a variety of techniques for loop detection but for the simplest auto responder a simple header will usually suffice. There’s a lot of software out there using loop detection headers such as X-Beenthere, X-Loop, etc. I imagine Delivered-To:, as well as having a good, descriptive name, is the most widely used for this purpose.
