Summary

  • RFC 2152 reserved + to shift from directly readable ASCII into unpadded Modified Base64 over big-endian 16-bit Unicode quantities; the shift ended at a non-Base64 character and could never cross a line break.
  • The format deliberately allowed choices. Set O punctuation could be sent directly despite gateway risk, and any Unicode sequence could be shifted, so successful decoding established a character sequence—not the original UTF-7 bytes, line layout, transit history or authorial meaning.

Consider a mail archive whose subject lines still look reassuring after a gateway migration. Names in Latin letters remain visible. Dates and ordinary punctuation seem intact. A later reader decodes the UTF-7 segments without error. It is tempting to stamp the transfer “preserved.”

That conclusion skips the part of RFC 2152 that made the format operationally interesting. UTF-7 did not offer one uniform envelope. It exposed an ASCII surface, opened and closed a shifted channel inside that surface, and left some punctuation to an encoder’s discretion. The visible part could survive while the less visible state boundary changed. A decoded sentence could be right while its source representation was no longer the one the author or first mail system emitted.

The constraint was seven-bit mail, not Unicode itself

David Goldsmith and Mark Davis published RFC 2152 in May 1997 as an Informational document, replacing RFC 1642. The memo did not claim Internet Standard status. Its problem was concrete: Unicode 2.0 covered writing systems that the seven-bit US-ASCII mail environment could not carry directly, while using UTF-8 with a MIME transfer encoding could impose two successive transformations and heavy expansion for non-ASCII text.

UTF-7 therefore used only US-ASCII octets. Text consisting mostly of the ASCII repertoire could remain directly readable on systems without Unicode support. The authors explicitly limited the bargain: use UTF-7 normally on seven-bit transports such as mail; elsewhere, straight Unicode or UTF-8 was preferable.

This was not a claim that old equipment understood multilingual content. It was a way to let old equipment transport it while preserving a useful human inspection surface.

Direct did not mean uniformly safe

The RFC divided ASCII into sets. Set D held letters, digits and nine punctuation characters chosen for direct encoding. The plus sign and equals sign were omitted. Set O held another collection of punctuation that could optionally be sent directly.

“Optionally” is the decisive word. The RFC warned that many Set O characters were illegal in header fields or might not pass correctly through some gateways. Backslash and tilde were omitted because ASCII variants often redefined them. Appendix A then printed two versions of the same Chinese example: one used optional Set O characters and might fail at certain gateways; the other avoided that risk.

So a plainly visible mark was not automatically the safest representation. One encoder might leave it direct; another could place the same Unicode character inside a shifted sequence. Both could decode to the same text. They exposed different bytes and different gateway risks.

The plus sign opened a stateful corridor

A literal + was special. It announced that following characters from Set B—the RFC 2045 Base64 alphabet without = padding—were to be interpreted as Modified Base64. The sequence ended when a non-Set-B character appeared. A hyphen used as terminator was consumed; +- represented a literal plus sign. If a plus sign was followed immediately by neither a Set-B character nor a hyphen, the stream was ill-formed.

Inside the corridor, the encoder serialized 16-bit Unicode quantities most-significant octet first. A UTF-16 surrogate pair was handled as two 16-bit quantities. Base64 padding bits were added only to reach a Base64 character boundary; at decode time, incomplete trailing bits were discarded, but any discarded non-zero bits made the sequence ill-formed. An odd number of decoded octets was also invalid.

These rules made decoding testable. They did not make the representation self-authenticating. The RFC’s Hi Mom +Jjo-! example can recover a smiling face between ordinary text. That proves the grammar when the input is intact. It does not say which relay handled it, whether an earlier form used different legal direct choices, or whether the reader saw the same line and glyphs as the author.

A line break was a protocol boundary

Carriage return and line feed are not invisible formatting trivia in UTF-7. A shifted sequence always terminates at the end of a line and may not cross a line break. RFC 2152 therefore recommended breaking lines before UTF-7 encoding—or performing both operations together—and using another MIME transfer encoding if the result remained too long.

It also recommended short SMTP-style lines ending in CRLF, with Unicode LINE SEPARATOR and PARAGRAPH SEPARATOR converted for better interoperability with non-Unicode or non-MIME readers. The memo admitted that this preparation was not absolutely necessary for an aware endpoint, but said readability on older systems would suffer.

That distinction matters for archives and gateways. Inserting a fold inside a shifted sequence is not cosmetic. Converting line separators can preserve intended paragraph structure for one environment while changing the original character sequence. Normalizing CRLF can be operationally correct while defeating a claim of byte-for-byte custody.

Decoding collapses evidence

Because direct and shifted forms overlap, decoding is many-to-one. Rule 2 permits any Unicode sequence to be shifted; Rules 1 and 3 allow selected characters to be direct. After both forms become the same Unicode text, a normal decoder need not retain which path was used.

That is not a defect in a transformation format. It is a warning about the question being asked. If the question is “what Unicode characters should the application process?”, a conforming decoder can answer. If the question is “what octets did the sender sign, archive or submit?”, decoded text is insufficient. Preserve the original message bytes and hash them before transformation.

The same discipline applies to line layout and optional punctuation. Record the content-transfer encoding, boundary handling, decoder and error policy. Compare raw input and raw output at each gateway. Then compare decoded code units separately. A green decode result cannot substitute for those ledgers.

Registration, successors and adjacent encodings

IANA still lists UTF-7, MIBenum 1012 and alias csUTF7, citing RFC 2152. That record proves the registered name. It does not show current adoption, product safety or faithful treatment of a particular message.

The similarly named UTF-7-IMAP is a separate, restricted object. IANA says it is for IMAP mailbox names and should never be used outside that context; RFC 3501 defined its modified convention. A MIME UTF-7 body and an IMAP mailbox name therefore require separate evidence and decoders.

RFC 2152 itself left security undiscussed. Later Unicode security guidance explains why different components comparing or transforming text differently can create gatekeeper failures, while warning that its stabilized report is no longer maintained and some recommendations have newer successors. The historical point is narrower: the 1997 memo did not license a security conclusion.

The RFC Editor records two verified corrections. One fixes the character shown for decimal 96 to grave accent; another restores the missing word in the sentence saying a shifted sequence ends at the end of a line. A third proposed correction was rejected. An evidence set should distinguish all three statuses rather than treating an errata count as three amendments.

Heng Lu’s minimum-specification frame is useful here. A common layer should state only the deterministic rules needed for interoperability; future and contextual decisions remain elsewhere. UTF-7’s shift grammar was such a layer. Running-code primacy then asks what the gateway actually emitted, and the reality-layers distinction prevents a readable surface or charset label from being promoted into a preservation result.

The durable lesson is not that UTF-7 failed because ASCII stayed visible. It is that visibility and custody answer different questions. The format solved a seven-bit carriage problem. Preservation still required receipts at the byte, line, decoder and reader layers.

Sources