Summary

  • RFC 3016 mapped MPEG-4 Audio and Visual directly into RTP, but its most consequential rules governed failure domains: grouping several recoverable units saved overhead while allowing one lost packet to erase all of them.
  • RTP sequence numbers, timestamps and marker bits described transport structure. They did not prove that configuration survived, a decoder produced media, or a listener or viewer received it.

The alarm arrived as a contradiction. The encoder reported no fault. The negotiated profile matched. RTP sequence numbers advanced, timestamps looked plausible and the marker bit appeared where the final fragment should be. Yet the video froze for longer than one missing datagram seemed to justify.

The problem was not hidden inside the codec. It had been decided one layer earlier, when the packetizer placed several logical video packets inside one network packet. The network lost one unit. The application lost several.

RFC 3016, published in November 2000, is usually described by its title: RTP Payload Format for MPEG-4 Audio/Visual Streams. It told implementations how to carry MPEG-4 Audio and MPEG-4 Visual without using the synchronization and stream-management functions of MPEG-4 Systems. H.323 could use H.245; SIP and RTSP could use MIME and SDP. The elementary streams could then travel alongside non-MPEG-4 codecs through the common machinery of RTP.

That architectural simplification moved responsibility rather than removing it. Codec syntax, session description, packet boundaries and receiver memory became separate custody domains. The packetizer had to decide where a network loss would land.

A packet boundary was a failure boundary

MPEG-4 Visual already contained error-resilience tools. RFC 3016 therefore did not add the kind of media-specific RTP header used by some earlier video payload formats. It mapped the Visual bitstream directly, byte aligned, and preserved its syntax elements.

Direct did not mean arbitrary. Configuration and Group-of-VideoObjectPlane material had to begin the RTP payload or follow the syntactically higher header. If headers were present, the payload had to begin with the highest one. A header could not be divided between RTP packets.

Those rules were about recovery, not aesthetic neatness. A receiver encountering a well-placed start code could locate a random-access point or resume from a codec-defined boundary. A header split across two datagrams became hostage to both. Lose either packet and the receiver might lose the description needed to interpret bytes that still arrived.

The specification recommended putting one video packet in one RTP packet and keeping the result below the Path MTU. On a lossy path, the mapping allowed another video packet to remain decodable through its Header Extension Code even when the packet containing the VOP header disappeared. The network's missing unit and the codec's damaged unit were kept close to the same size.

But the rule was deliberately not absolute. At a low bitrate, RTP and IP headers could be expensive beside tiny video packets. Several could be concatenated into one RTP packet to save overhead. The bargain was explicit: one RTP loss would now discard every video packet inside it.

That is a deeper operational fact than “packet loss occurred.” The same loss counter can describe very different damage. One missing datagram might remove one independently recoverable region, several regions bundled for economy, or a header on which later material depends. The counter records the carrier unit. It does not reconstruct the dependency graph inside that unit.

Two packets could carry unequal risks

RFC 3016 included a prohibited arrangement that exposes the problem elegantly. Two logical video packets could be distributed across two RTP packets in more than one way. In the sound arrangement, losing the second RTP packet removed only the second video packet. In the bad arrangement, the first logical packet straddled the boundary and a header for the second sat behind its remainder. Losing the second RTP packet then destroyed both.

The number of RTP packets was identical. The byte count could be similar. The codec family, profile and clock were unchanged. Only placement differed, yet the recovery outcome changed.

This is why packetization is not a clerical stage between encoding and transport. It is an allocation decision. It decides whether overhead is paid repeatedly to preserve isolation, or amortized by enlarging the loss lot. It decides whether a receiver can begin with a recognizable header, and whether a surviving fragment has an independent interpretation.

The same reasoning governed arbitrary fragmentation. If video packets were disabled by the coder, a VOP could be split at arbitrary byte positions. Fixed-size chunks might be tolerable on a path assumed to be error free. On an error-prone path, the RFC warned that the result offered poor loss resilience. The stream remained standards-conformant; the deployment assumption became the fragile part.

A marker marked structure, not success

For MPEG-4 Visual, RFC 3016 assigned the RTP marker bit to the last or only packet of a VOP. When several VOPs shared one packet, the marker was also set. The RTP timestamp described the earliest VOP in that payload, while timing for later VOPs came from their own headers.

This was enough to guide depacketization. It was not a delivery receipt. A sender could set the marker perfectly before the packet was dropped. A receiver could observe the marker after an earlier fragment had vanished. A depacketizer could close a unit that the decoder then rejected. A decoder could succeed while the presentation process remained stalled or muted.

RTP itself keeps the distinction. A sequence number can reveal a gap and help restore transmitted order. A timestamp represents a sampling instant and supports synchronization and jitter calculation. Actual presentation happens later at the receiver. A marker's meaning comes from the payload profile; it is not a universal “frame worked” bit.

An operations system that promotes those fields into a green media-status light is inventing an extra contract.

Audio carried a map that could go missing

The audio side used LATM, the Low-overhead MPEG-4 Audio Transport Multiplex. A complete audioMuxElement, or a fragment of one, went directly into the RTP payload. Its first byte began at the first payload position. One element per RTP packet was recommended, with fragmentation available when the element would exceed Path MTU.

The subtle failure was configuration memory. In in-band mode, useSameStreamMux could tell the receiver to reuse the previous frame's StreamMuxConfig. That saved repeated configuration bytes. If the previous frame had been lost, however, the current frame might be undecodable even though it arrived intact. RFC 3016 therefore recommended repeating StreamMuxConfig according to network conditions.

The missing packet did more than remove its own audio. It could remove the map needed for later audio. Repetition shortened that dependency; omission lengthened it. Out-of-band configuration shifted custody to signaling, such as SDP's config parameter, but did not abolish the need to prove that the receiver obtained the correct value for the current stream.

This is another reason raw loss counts are incomplete. The packet that disappeared may have carried ordinary media, a resynchronization point or the configuration on which a run of later packets depended.

Capability was not configuration

RFC 3016 registered video/MP4V-ES and audio/MP4A-LATM and mapped their parameters into SDP. For Visual, profile-level-id could describe a codec capability combination. The config value described the configuration of the corresponding bitstream and was explicitly not a capability declaration.

The difference is operationally sharp. A decoder may support a Profile and Level yet lack the exact current configuration. Two peers may advertise compatible tool sets while disagreeing over a stream parameter. A well-formed SDP description can be authentic and still be stale, misrouted or inconsistent with the bytes that arrive.

Session description tells a participant how media is supposed to be represented and where it is supposed to go. It does not watch the path, retain a lost in-band configuration or certify a displayed picture.

Recovery mechanisms started after the boundary choice

RFC 3016 allowed Generic Forward Error Correction and redundant audio to improve resilience. Those mechanisms matter, but they answer a later question. FEC can protect packets only after the packetizer has decided what each protected packet contains. If one source packet contains three logical recovery units, recovering that packet restores all three; failing to recover it loses all three. The packetization decision set the stake.

This keeps the history distinct from RFC 2733. That standard explained how parity packets might reconstruct missing RTP media and what overhead and delay the scheme incurred. RFC 3016 exposed the upstream choice: how much media and configuration would become the subject of one loss or one FEC equation.

Likewise, H.263+'s RTP format copied picture-header information into an added payload header so a receiver might decode after the original header was lost. RFC 3016 chose to rely on MPEG-4 Visual's own error-resilience syntax. Different codec tools produced different packetization contracts; the common RTP header did not make them interchangeable.

The historical specification was later corrected

RFC 6416 obsoleted RFC 3016 in 2011. The revision corrected misalignment with the 3GPP Packet-switched Streaming Service, particularly for MPEG-4 Audio. It mandated a newer LATM version that was binary incompatible with the one referenced by RFC 3016, revised StreamMuxConfig and clarified SBR, Parametric Stereo, rate, channel and scalable-layer signaling.

That correction warns against treating an RFC number as a timeless decoder identity. Some implementations described as RFC 3016 had already used the newer LATM form and might align with RFC 6416 while not strictly matching RFC 3016. A label could hide a binary boundary.

The later RFC nevertheless retained the Visual lesson. One video packet per RTP packet protected loss isolation; grouping saved headers at the cost of a larger loss; syntactic placement changed what surviving bytes could mean.

Encryption did not shrink the loss lot

RFC 3016 inherited RTP security considerations and noted that compressed media could be encrypted without conflict. The restricted payload carried audio and video, not the active MPEG-J and script content possible in the complete MPEG-4 System.

Confidentiality, integrity and source authentication protect important questions. They do not answer every question. An authentic encrypted packet can still exceed Path MTU, arrive after the configuration it needs has vanished, or bundle several recovery units behind one sequence number. Cryptography can prove who protected the lot. It does not decide whether the lot was wisely sized.

The Internet history here lives in that allocation. A payload format was not simply a wrapper around a codec. It turned codec boundaries into network failure domains, balanced header economy against independent recovery, and exposed a dependency between lost configuration and surviving media. The codec could remain valid. The packetizer chose what one loss could destroy.

Sources