RFC 3124: A Send Grant Was Not a Packet Queue

Summary

RFC 3124 let applications request transmission opportunities from a host-wide Congestion Manager, but the request carried no application buffer and the callback granted only a bounded right to send, normally up to the path MTU.
A grant was not a queued packet, an enforced reservation or a delivery receipt. The application selected and possibly repacketized content, reported actual bytes emitted with cm_notify, and supplied receiver feedback separately with cm_update.

One host had learned too many congestion lessons

At the turn of the century, an Internet host could run several flows to the same destination while each application learned congestion conditions alone. A browser opened parallel transfers. Audio and video programs adapted their own rates. New sessions repeatedly began with little knowledge even when another connection on the machine had recently measured the same path. The result was not merely duplicated work. Independent control loops could compete, start badly and expose applications to inconsistent views of one bottleneck.

RFC 2140 had already described sharing selected TCP control-block state among connections to the same host. RFC 3124, published on the Standards Track in June 2001, moved the boundary further. Its Congestion Manager, or CM, was an end-system module that grouped related flows into a macroflow and maintained congestion state for them. Applications could use an asynchronous interface to request permission to send, or a synchronous interface to ask for rate and congestion estimates while performing their own scheduling.

The change was architectural. Instead of every application privately deciding both what to send and when, a common service could arbitrate the scarce transmission opportunity. Yet RFC 3124 did not turn that common service into a universal packet queue. That distinction is the heart of the design.

The request deliberately contained no data

An application called cm_request when it had data available. The call did not hand a buffer to the Congestion Manager. The CM therefore did not store the application's packet, inspect its payload or promise that a particular message would leave later. It recorded demand for an opportunity.

When the congestion controller and scheduler allowed transmission, the CM invoked cmapp_send. The callback authorized the application to send up to a specified maximum, normally one path maximum transmission unit. At that moment the application could choose which material mattered most. A streaming program might select a fresher frame, reduce fidelity or repacketize information to fit the grant. A stale frame did not need to sit in a generic operating-system queue merely because it had been the first item submitted.

That late choice explains why the separation was useful. Congestion control could remain shared while application semantics stayed with the application. The scheduler knew that capacity was scarce. The application knew whether an old packet, a key frame or a smaller substitute was valuable. Neither needed to impersonate the other.

It also explains why the callback was not a receipt. The CM had made room in its control model. No packet had necessarily been constructed, passed to IP or accepted by an interface. Treating the callback as evidence of transmission would erase the deliberate gap that enabled adaptation.

Permission expired and unused permission returned

The send opportunity was time-bounded. RFC 3124 said it expired after the maximum of a round-trip time and an implementation-defined threshold. An application receiving an expired callback was not entitled to use it. Congestion state changes with time; a grant derived from an older window cannot be preserved as durable credit.

After using a callback, the application invoked cm_notify with the number of bytes actually submitted at the IP output path. If it could not send, it was expected to report zero. That zero returned the unused opportunity so another request could proceed. The protocol thus had a small but important accounting loop: request, grant, actual-use notification.

RFC 3124 also required robustness when the loop broke. An application might crash or fail to return a zero notification. The CM could not freeze forever waiting for perfect behavior. Implementations needed a way to recover from missing reports while avoiding an immediate assumption that a callback equaled a send.

The nsent value meant bytes sent by the host, not bytes received by the peer. It updated the sender's outstanding-data accounting. Delivery evidence arrived through another interface, cm_update, which conveyed receiver feedback and distinguished reports such as successful reception, congestion notification, loss and absence of feedback. The distinction made three moments observable: permission, local emission and remote evidence.

A macroflow was a control hypothesis

Flows sharing congestion behavior could be placed in one macroflow. They then used common congestion-control and scheduling state. Applications could influence this grouping because they sometimes knew which flows were likely to share a path or administrative purpose.

But membership did not prove that packets traversed the same bottleneck. Routing could change. Multiple interfaces could exist. A host's grouping rule could be too broad. RFC 3124 presented a mechanism for sharing control information, not a cryptographic statement about path identity. Operators evaluating a macroflow therefore needed to retain the grouping key, route observations, interface choice and time of each decision.

The earlier RFC 2140 and the later RFC 9040 help locate the idea historically. Both concern reuse of transport knowledge across connections, but RFC 3124's specific contribution was an application-facing request/grant/notify protocol and a shared scheduler. It let non-TCP applications participate without forcing their payloads into TCP's control block. That makes this chapter different from the history of cached TCP metrics alone.

“Well behaved” was a contract, not a guardrail

RFC 3124 called an application well behaved when it sent only with CM consent and accurately reported all transmitted data. The phrase described compliance with the interface. It did not mean the Congestion Manager could stop every process on the host from transmitting.

The document was explicit that the CM did not enforce congestion behavior on all applications and did not protect the network from a compromised host. A nonparticipating or malicious program could bypass the interface. Host policing or router enforcement would be separate machinery, outside the specification.

This boundary prevents a common historical overclaim. A host running a CM implementation would not, for that reason alone, become congestion-safe. Evidence of the module's presence would need to be joined with evidence that a named application used it, that its calls matched its sends, that competing traffic did not bypass it and that the relevant packets followed the measured path.

Optional adoption mattered too. Applications were not universally required to use the CM. If they did use it, the RFC defined how they should behave. Standards status established the protocol contract; it did not establish deployment, enforcement or market uptake.

Queries could expose uncertainty without fabricating capacity

The synchronous interface served applications that wanted CM estimates but retained their own transmission loop. Through cm_query, an application could obtain rate, round-trip-time and congestion information. Invalid estimates were represented by negative values. Zero was not substituted merely to make the result look numeric.

That choice preserved uncertainty as data. An unknown estimate is different from a measured capacity of zero. Confusing the two could cause an application either to stop unnecessarily or to regard missing measurement as a stable limit. The same discipline applies to historical evidence: an absent deployment receipt cannot be converted into failure, and a standards document cannot be converted into adoption.

RFC 2914's congestion-control principles, RFC 2581's TCP behavior, RFC 2861's treatment of an idle congestion window and the experimental ECN work in RFC 2481 supplied the surrounding technical vocabulary. RFC 1191 supplied the path-MTU context; RFC 5681 and RFC 8085 later restated transport and UDP responsibilities. They illuminate the problem the CM addressed. None proves that a particular operating system or application used RFC 3124.

The evidence chain was longer than one callback

A defensible operational record would start with an application request and macroflow identity. It would retain the scheduler's state, the grant size and timestamp, its expiry calculation, the application's selected payload, the cm_notify count at IP output, route and interface evidence, and later cm_update feedback. Delivery or user-visible quality would require still more observation.

Each item answers a different question. A request proves demand was declared. A callback proves the scheduler offered a bounded opportunity. A positive notification proves the application reported bytes at the sender's IP path. Receiver feedback may support a claim about arrival or congestion. None can silently stand in for all the others.

This is why “send grant” is the most useful phrase for RFC 3124. It avoids the false concreteness of a packet already queued and the false finality of a successful transfer. The manager governed timing and shared congestion state; the application retained content choice; the host network stack performed emission; the path and receiver produced the outcome.

The RFC's durable lesson was not that one manager could command an entire machine. It was that a shared control surface becomes trustworthy only when its permissions, uses and outcomes remain separately observable.

A Send Grant Was Not a Packet Queue: RFC 3124

Why it matters

Summary

One host had learned too many congestion lessons

The request deliberately contained no data

Permission expired and unused permission returned

A macroflow was a control hypothesis

“Well behaved” was a contract, not a guardrail

Queries could expose uncertainty without fabricating capacity

The evidence chain was longer than one callback

Sources

Deeper Profile Context

Strategic Circle

Leadership Alliance

Strategic Circle Briefing

Leadership Alliance Briefing