Summary

  • NNTP overview replaced repeated per-article header retrieval with tab-separated summary rows whose common core and optional tail had a published order.
  • LIST OVERVIEW.FMT could advertise a field only when the overview database consistently recorded either its value or its absence across the relevant articles.
  • During a schema change, the server had to withdraw a field before it stopped storing it and wait to add a new field until backfill or expiry restored consistency.

The meaning between two tabs

The smallest object in the overview protocol is an empty cell. On the wire it can be no more than two adjacent tab characters. Yet the client must decide whether that emptiness is information.

If the server has promised that a column is present for every article, a blank says the corresponding header or metadata item is absent. If the server has indexed the field only for newer articles, the same blank might mean “not captured.” Those are opposite conclusions. The first is evidence about the article. The second is evidence about an incomplete database.

RFC 3977 made the distinction explicit. An overview database is consistent for a field when it records the content or absence of that field for all applicable articles. If it contains the field for some articles but fails to record it for others that had it, the database is inconsistent. The schema advertised by LIST OVERVIEW.FMT must not claim an inconsistent or unstored field.

Before the table, one header at a time

The original RFC 977 gave a reader HEAD. Select an article by Message-ID or server-local number and receive its headers. The command was precise, but learning the subjects, authors, dates and thread references of a large group meant repeating a per-article operation.

The overview work changed the unit of retrieval. RFC 2980 documented XOVER, which returned one compact row for every extant article in a requested number range. The row began with the article number, then subject, author, date, Message-ID, references, byte count and line count. Optional fields could follow.

The saving came from precomputation and batching. The reader did not need every full header block merely to draw a list of conversations. But compact rows remove labels. Once values are reduced to positions, both parties must agree exactly what every position means.

A fixed core and a declared tail

RFC 3977 standardized the command as OVER. Its first eight fields are fixed: article number or zero, Subject, From, Date, Message-ID, References, :bytes and :lines. Tabs separate the fields. Missing internal fields preserve their position with adjacent tabs, while trailing empty fields may be omitted.

Values are also normalized to protect the grammar. Folded header lines are unfolded, and tabs within values become spaces. Otherwise content could masquerade as a delimiter and shift every later column.

Extension did not require guessing. LIST OVERVIEW.FMT returned one field description per line and in the order used by OVER. Its first seven descriptions corresponded to fields two through eight. Later entries named additional metadata or used the :full suffix to say that a returned header retained its header name. The core stayed stable while the tail could grow under an explicit contract.

The schema was a claim about history

The crucial innovation was not merely publishing column names. It was limiting those names to what the database could support consistently at that moment.

Suppose an operator decides to start indexing an additional header today. New articles can populate the column immediately; old articles cannot, unless the server rebuilds their overview records. Advertising the column at once would make blank cells ambiguous. A missing value in an old row might mean that the article lacked the header or that nobody had backfilled it.

RFC 3977 therefore makes schema migration a two-stage act. Before a field stops being stored, it must be removed from LIST OVERVIEW.FMT. A new field should enter the advertised format only when the database is consistent again—after reconstruction, or after older articles have aged out so that all remaining rows were created under the new rule.

The format can even change within one session. A client that treats yesterday's schema as permanent can misread today's row. Capability and format discovery are observations of current server state, not eternal documentation.

Withholding a column preserved the truth of a blank

This rule appears conservative: a server may possess some values yet decline to expose the field as a column. But withholding incomplete knowledge protects the semantic value of every empty slot. Once the field is advertised, RFC 3977 allows the client to infer that an empty cell means the article did not contain that header or metadata item.

The protocol thus separated data availability from evidentiary authority. Having partial values was not enough to publish a column contract. The server first had to know that the same capture rule covered the relevant collection.

That is why omission from LIST OVERVIEW.FMT is not equivalent to omission from an article. The former says the index makes no consistent promise about that field. The latter becomes knowable only after the promise exists.

Metadata was not an authored header

Overview also distinguished server-computed metadata from article content. :bytes and :lines begin with a colon and must not be trusted from similarly named headers supplied inside an article. The server computes them.

Even then, authority remains bounded. RFC 3977 specifies how :bytes should count the article but records historical variations and tells clients not to rely on perfect accuracy. :lines has a stricter body-line definition. A field can have clear provenance and still carry measurement limits.

Range output is bounded too. Existing rows are sorted by article number, but removed articles should not be returned and holes can remain. OVER summarizes what the server can report; it does not recreate expired bodies or prove that every number exists.

The IANA NNTP Parameters registry records OVER as the standardized overview-support capability. That establishes an interoperable signal, not current deployment or a performance guarantee.

Speed required a narrower promise

The overview database made newsreading practical by moving repeated parsing into a shared server-side index. Yet the fastest table would have been dangerous if every blank mixed article absence with indexing failure.

NNTP solved that problem by making the schema confess its limits. During incomplete migration, the server advertised less. After consistency returned, it could promise more. A blank field became trustworthy not because emptiness explained itself, but because the surrounding column had earned the right to make an absence claim.

Sources