Summary

  • RFC 9628 offers two contracts for describing VP9 inter-picture dependency: flexible mode puts up to three relative references on predicted pictures, while non-flexible mode declares a recurring Picture Group in Scalability Structure data.
  • Picture IDs, layer indices, P_DIFF, SS, frame-boundary bits and the RTP marker can describe the sender’s structure. They do not prove that every fragment arrived, the receiver retained every reference, hidden codec state was safe or a visible picture was produced.
  • A defensible VP9 receipt joins the negotiated payload context, the active mode-and-structure epoch, packet coverage, reference closure, bitstream validation, decoder output and application rendering.

The forwarding log had everything an engineer might want to admire. Every packet carried a Picture ID. The temporal and spatial layer labels were present. The middlebox could trace the declared references backwards. Its graph closed cleanly.

The decoder still froze.

That is not a paradox in RFC 9628. The standard describes how VP9 video is carried in RTP and how temporal and spatial scalability can be exposed. It gives endpoints and selective forwarders a compact account of structure. It does not turn that account into testimony about every packet, every retained reference, every internal VP9 state transition or the final display.

The distinction matters because the header is persuasive. A graph looks like the system itself. In operation, it is only one declared layer of reality.

One codec, two ways to disclose the past

VP9 may keep as many as eight reference frames, and a new frame may use up to three. For transport, RFC 9628 simplifies the problem into spatial and temporal relationships that a receiver can reason about. The D bit says whether a spatial-layer frame depends on the immediately lower spatial layer in the same picture. Inter-picture references are disclosed in one of two modes.

Flexible mode is local and explicit. When a predicted picture is carried with F=1, at least one and at most three P_DIFF values identify earlier Picture IDs. A current PID of 112 with P_DIFF=3 points to PID 109, calculated modulo the active 7- or 15-bit PID space. Because the references travel with the picture, the encoder can vary the temporal hierarchy over time.

Non-flexible mode is compact and historical. A Scalability Structure block predeclares a recurring Picture Group. The key picture anchors the first position. Later Picture IDs advance through the declared pattern modulo N_G. The receiver obtains a picture’s temporal identity, switching point and relative references from its position in that pattern rather than from a new reference list on every predicted picture.

These modes are not interchangeable encodings of a stateless label. They create different evidence obligations. In flexible mode, an auditor needs the current picture and its explicit references. In non-flexible mode, a later packet cannot be interpreted correctly without the governing SS epoch, its key-picture anchor and the allowed point at which the structure changed.

A Picture ID is useful because it is not a global identity

Picture ID is a running index, not a durable content address. The sender may use seven or fifteen bits. The value starts at a random point, wraps at the end of its field and may change width during the session. Moving from seven to fifteen bits zero-extends the value; moving back truncates it.

Every frame belonging to one picture shares the PID, including the different spatial representations of that instant. A picture with VP9 show_frame=0 still gets a different PID from a later displayed picture. The identifier therefore follows transport pictures, not only images visible to a person.

Nor is continuity guaranteed at the receiver. A selective forwarder may lawfully discard pictures under the declared scalability structure. The standard explicitly warns that received Picture IDs may not be contiguous. A gap can mean loss, a forwarding decision or an observation boundary. A smooth sequence can wrap or conceal missing fragments inside one picture.

The correct audit key is consequently larger: session, SSRC, payload mapping, PID width and value, RTP timestamp, layer, mode epoch and observation point. Removing those coordinates turns a precise identifier into an ambiguous number.

The recurring graph needs an epoch

Non-flexible mode buys bandwidth by making history shared. Its Picture Group describes a fixed recurring dependency pattern. TL0PICIDX separately tracks temporal-layer-zero pictures and tells a higher temporal layer which base picture supports it. That index also wraps, after 255.

The economy works only if sender, middlebox and receiver agree on the same structure. RFC 9628 therefore limits when the mode and SS data can change. The F bit changes only on the first packet of a key picture. In a fixed scalable stream, SS data should appear on the first packet of each keyframe, and a new structure may begin only at the previous group’s first-picture boundary.

A packet capture that begins halfway through an epoch may show immaculate PIDs and layer IDs while lacking the declaration that gives them meaning. A restarted forwarder can possess current packets but not the structure that maps them. A logging system can retain the SS block but lose the key-picture boundary that anchors phase. None of these is repaired by guessing the most common pattern.

For non-flexible operation, the receipt must preserve the structure version, the packet that introduced it, the key-picture anchor, the Picture Group length and the observed mapping from PID to group position. Otherwise “reference graph reconstructed” means “reference graph assumed.”

Frame complete and picture complete are different claims

VP9 frames may be fragmented across RTP packets. B marks the first packet of a frame and E the last. The RTP marker identifies the final packet of the highest spatial-layer frame, which is the end of the picture. If a middlebox removes higher spatial layers, it must move that marker to the final packet of the layer it actually forwards.

These flags solve boundary discovery. They do not supply missing bytes. Seeing E=1 does not show that every RTP sequence number after B arrived. Seeing the marker does not show that all lower spatial frames required by D=1 were complete. A middlebox can rewrite the picture boundary correctly while forwarding a stream whose required reference was lost earlier.

There is no finer-grained access mechanism for a partial VP9 frame in this format. Operationally, the receiver needs packet coverage from B through E, a coherent timestamp and spatial ordering, and the correct final marker for its selected output. Boundary metadata and coverage evidence must remain separate.

The visible graph omits part of codec state

Reference pictures are not the only state carried across VP9 frames. The codec also uses probability tables for entropy and tree coding. RFC 9628 calls out error_resilient_mode because it resets this additional inter-frame state.

In a scalable stream, an encoder must not leave a later frame dependent on state from a frame that may legitimately be removed. The base spatial layer therefore needs error-resilient handling where such removal is possible; higher layers can relax it only under the specified inter-layer conditions.

This is the decisive limit of header-only confidence. P_DIFF and SS can describe the permitted reference-picture graph perfectly while the encoded frame still relies on codec state that was not safely reset after a drop. Conversely, a robust decoder may conceal some damage without validating the sender’s description.

A reference-closure checker is necessary. It is not a decoder. The receipt must add bitstream validation and actual decoder output before making a decodability claim.

Layer labels describe a choice surface

TID and SID let an endpoint or middlebox identify temporal and spatial layers. D reports dependency on the immediately lower spatial layer of the same picture. U announces a switching-up point: after this picture, higher temporal layers will not depend on older pictures above the current temporal layer. Z says upper spatial layers do not depend on the current spatial frame, allowing it to be discarded for a receiver targeting higher layers.

Each field authorizes a narrower inference than dashboards often assign to it. U=1 does not show that the switching picture arrived. Z=1 does not show that discarding was desirable or that the retained path met quality goals. SID=2 is not an absolute measure of resolution or experience. D=0 removes one same-picture dependency but says nothing about inter-picture references or packet completeness.

The VP9 bitstream specification defines what the coded payload means. The RTP descriptor defines how enough structure can be exposed around it. Neither lets a forwarding policy grade its own outcome from labels alone.

Feedback acknowledges a different event

RFC 9628 also connects VP9 to several feedback mechanisms. A receiver can use Reference Picture Selection Indication to name a correctly decoded golden or altref picture, or to prefer a known reference after loss. Full Intra Request asks for a full state refresh. Layer Refresh Request, defined separately in RFC 9627, can request a narrower temporal or spatial transition.

RPSI is stronger evidence than a parsed PID because the receiver asserts correct decoding of a reference. It is still not a universal display receipt. It does not say every spatial frame in the picture was shown, that the application selected it for playout, or that the viewer received the intended improvement before the deadline.

For LRR, RFC 9628 recommends including layer indices and reference fields so a decoder or selective forwarder can walk backwards until it reaches already decodable material. That recommendation explains why the graph matters. It does not collapse a request, a structural response, packet delivery and a rendered upgrade into one transaction.

Capability signalling does not close the execution loop

The SDP mapping binds a dynamic payload type to VP9 at 90 kHz. profile-id is the symmetrical media-format configuration; when absent, Profile 0 is inferred. max-fr and max-fs are declarations of receiver capability.

Those declarations set expectations, not measurements. RFC 9628 says a sender should stay within the advertised limits, while recognizing cases such as pre-encoded media or selective forwarding where an exact fit may not exist. An accepted offer therefore does not prove that a later frame matched the receiver, and a payload-type number means nothing without the negotiated mapping and profile epoch.

The complete receipt begins before the first VP9 descriptor. It records offer, answer, payload type, profile, capability claims, SSRC and any later renegotiation. Only then can packet metadata be interpreted in the right contract.

Sources