Summary

  • RFC 9839 defines Unicode Scalars, XML Characters and Unicode Assignables as increasingly restrictive sets that protocol specifications can name precisely.
  • A JSON document can conform to its grammar while carrying escaped controls, an unpaired surrogate or a noncharacter; successful parsing therefore does not establish field-level admissibility or well-formed UTF-8 output.
  • The operating control is a declared subset plus an explicit reject-or-replace rule, tested across every parser, serializer, queue, database, log and consumer that can change the value.

The request reached the service, the JSON library returned an object, and the validation dashboard turned green. Deep inside one string sat an unpaired surrogate. Another field carried a NUL. A third contained a code point reserved never to be used for open interchange. The envelope had passed. The data had not.

That is the boundary RFC 9839 makes visible. Published on the Standards Track in August 2025, its official record offers three reusable character-repertoire subsets for specifications. The plain-text edition and XML source define the same contract. Publication supplies vocabulary and consensus. It does not prove that any deployed API applies the contract consistently.

A code point is not necessarily usable text

Unicode's codespace contains 1,114,112 possible code points. A code point is a number in that space; it is not a guarantee that the number denotes a character suited to protocol interchange. Surrogates exist to assemble supplementary characters in UTF-16 pairs. Standing alone, they have no textual meaning, and UTF-8 forbids encoding them. Legacy controls mostly carry obsolete or environment-dependent semantics. Noncharacters are permanently reserved away from open interchange.

The distinction is easy to lose behind the phrase “supports Unicode.” BCP 18 requires Internet protocols to be able to use UTF-8, but an encoding form answers how admissible values become bytes. It does not decide which values a particular field should accept. The Unicode Standard supplies the repertoire and definitions; its character database annex describes properties. The application still owns the admission decision.

RFC 9839 names three possible boundaries. Unicode Scalars remove the 2,048 surrogate code points but keep legacy controls and noncharacters. XML Characters also remove most C0 controls and two BMP noncharacters, while retaining C1 controls and noncharacters in higher planes. Unicode Assignables remove surrogates, legacy controls and all noncharacters while preserving values already assigned or available for future assignment. The XML 1.0 grammar is therefore a useful historical profile, not a synonym for “all safe text.”

The sets are described using ABNF, so a specification can point to a stable rule instead of saying “valid Unicode” and leaving each implementer to invent its meaning.

The JSON trap is an authority error

RFC 8259 defines JSON's portable syntax, and its status page records the standard. JSON permits escape sequences in strings. RFC 9839 demonstrates a conforming JSON text whose decoded value includes NUL, a C1 control, an unpaired surrogate and a noncharacter. That value cannot be serialized as well-formed UTF-8, yet libraries can disagree about what happens when they parse it.

Escaping changes representation, not content. If a field is restricted to Unicode Assignables, spelling a forbidden value as \uDEAD does not grant admission. The check belongs after escape decoding and before the value becomes authoritative state.

This is an authority error because teams often let the general-purpose parser decide a question it was never designed to answer. A parser can prove that delimiters, member names and escapes fit the container grammar. It cannot know whether a customer identifier may contain controls, whether a signed claim must round-trip exactly, or whether a display field should use a stricter human-facing profile. I-JSON narrows JSON for interoperability, while Net-Unicode and PRECIS address other string classes. They are evidence that “one Unicode switch” is the wrong control model.

Reject and replace are different contracts

RFC 9839 identifies rejection and replacement as reasonable responses to problematic content. Silent deletion is not a neutral third option. Removing an invisible code point can merge two values, change an identifier, defeat an audit comparison or make the stored record differ from what a signer saw.

Replacement with U+FFFD can keep a display pipeline alive and make damage visible. It also destroys exact identity. A system must not replace inside a value used for a digital signature, content hash, deduplication key, legal record or access-control comparison and then pretend the original survived. For those fields, rejection or quarantined raw evidence is usually the honest result.

Rejection has its own cost. A public endpoint that performs expensive diagnostics on arbitrary invalid strings can become an availability surface. The protocol decision therefore needs bounded error messages, rate limits and a quarantine path. Maintaining Robust Protocols is relevant precisely because robustness is deliberate handling, not indiscriminate acceptance.

The receipt must cross every runtime boundary

The same escaped input should be exercised through every supported runtime and path: edge gateway, application parser, schema validator, message broker, database driver, persistence layer, log formatter, export job and downstream consumer. Record four outcomes separately: container parse failure, subset violation, deliberate replacement and later serialization failure.

A single unit test in the preferred language is weak evidence. RFC 9839 notes that surrogate behavior can differ between programming languages and even between APIs in one language. Differential tests should compare the decoded scalar sequence, decision, error class and stored representation. Canary vectors should cover paired and unpaired surrogates, C0 and C1 controls, every noncharacter pattern, private-use characters, useful controls and newly assigned code points.

CBOR defaults to Unicode scalar values for text strings. That removes surrogates but not every problematic value. A migration from JSON to CBOR can therefore change one admission boundary without completing the policy. Format choice and field policy remain separate decisions.

Narrow scope matters

Unicode Assignables is not a universal identity profile. It does not solve canonical equivalence, mixed scripts, bidi display, confusable characters or contextual validity. Unicode Security Considerations surveys broader deception risks, while Unicode Source Code Handling focuses on programming-language source. The W3C's character-encoding concepts help distinguish characters, code points, code units and bytes. A human-facing username may need PRECIS or another application-specific rule even after it passes Unicode Assignables.

That narrowness is a strength. The central specification need not govern every language or user interface. It can state the minimum invariant: this field admits this named repertoire, at this decoded boundary, with this error behavior. Operators retain the localized decision about stricter profiles, quarantine, telemetry and migration.

This follows the discipline in Heng Lu's minimum-specification argument. Coordination should define the irreducible common rule, not expand into an all-purpose authority. Running code then supplies the evidence: not a slide saying “UTF-8 everywhere,” but identical outcomes across the actual components that receive, persist and reproduce the value.

What the standard does not prove

RFC 9839 does not report adoption, implementation conformance or incident frequency. It does not establish that one subset is right for every field. It does not show that named libraries agree, that databases preserve the same scalars, or that logs display them safely. This report tested no product, runtime or production path.

The justified conclusion is narrower and more useful. Container validity is not content authority. A green parser result is evidence about an envelope. Protocol admission begins only when the decoded value is measured against a declared repertoire and the system can show what happened at every boundary afterward.

Sources