Summary

  • Equal-cost multipath gives a router several acceptable next hops; it does not certify that their MTU, latency, ordering or multicast role is interchangeable.
  • RFC 2991 compared ways to keep a flow on one path and limit reassignment when a next hop changes. Each algorithm priced disruption against computation differently.

A traceroute can show a path. It cannot, by itself, tell you which paths the router considered or how it assigned other flows. That gap matters because a routing table’s “equal” is a statement about route cost, not a promise that packets will experience equal conditions.

In November 2000, Dave Thaler and Christian Hopps published RFC 2991, an Informational account of multipath forwarding. OSPF and IS-IS explicitly allowed equal-cost multipath (ECMP); some router implementations also used it with RIP and other protocols. When multiple next hops were valid for one destination, the forwarding engine still had to choose one for each packet.

The naïve answer—take turns, or choose randomly—could make a single conversation cross paths with different maximum transmission units and delays. Path MTU discovery would then face a packet-by-packet moving target. Different arrival times could reorder a TCP stream, create buffering pressure and make later packets look like loss to the receiver. Ping and traceroute might observe different branches and offer a misleading picture of the route. Multicast raised a stronger constraint: the protocols discussed built a single tree toward a source, core or rendezvous point, relying on one upstream next hop to prevent loops and duplicates.

“Spread the packets” was not a neutral forwarding policy.

RFC 2991 therefore treated the unit of assignment as a flow: whatever traffic granularity the router kept state for, if any. That was not necessarily the five-tuple microflow defined in RFC 2474. A router might key only on destination, or on source, destination and protocol. Ports could be unavailable in non-initial fragments; using transport fields could also fragment cache reuse for path information such as MTU. The boundary was an implementation decision with operational consequences, not a universal definition hidden in the word “flow.”

Once packets belonging to a flow stayed together, two requirements competed. The selection had to be cheap enough to run, and adding or losing a next hop should disturb as few active flows as possible. ECMP increases the number of routes that can directly affect forwarding, so route membership churn can expose traffic to reordering or loss even when the preferred metric has not changed. The design question became: when the candidate set changes, how much of the existing allocation must move?

Modulo-N hashing was inexpensive. Hash the flow and take the result modulo the number of next hops. But when that number changes, RFC 2991 says (N-1)/N of the flows change paths. Hash-threshold divides the hash space into next-hop regions; changing region boundaries affects flows near them, though the RFC estimates that between one quarter and one half of flows move when a member is added or removed. RFC 2992 analyzes that algorithm’s disruption. Highest Random Weight (HRW) hashes a flow with each candidate next hop and chooses the largest result. It limits a single-member change to about 1/N of flows, at roughly N times the modulo-N computation.

Those figures describe algorithmic models, not measurements from deployed routers. Nor does “one quarter” mean one quarter of bytes, packets, customers or service impact: the metric is the fraction of flows reassigned under the stated assumptions. A few large flows can carry a very different share of traffic than a large population of short ones.

State placement changes where the cost is paid. If a router already keeps per-flow state, it can select a next hop when creating the state rather than recomputing the choice for every packet. RFC 2991 recommends HRW for stateful unicast and for multicast forwarding, where source/group state is maintained; where unicast forwarding has no per-flow state and must calculate on packet arrival, it recommends hash-threshold when CPU is more precious than stability. The recommendation is conditional on architecture, not a single universal winner.

The later record preserves the tradeoff, not proof of universal adoption. In 2011, RFC 6438 described ECMP and link aggregation as balancing path share, avoiding per-flow reordering and keeping links busy—goals that can conflict. That is useful evidence that the engineering bargain endured. It does not show that every router followed RFC 2991 or used its exact algorithms.

The historical point is narrower, and more useful, than “load balancing is hard.” A routing metric classifies candidates. A local selector assigns flows. The physical paths supply their actual MTU and delay; transport endpoints react to what arrives. None of those layers can stand in for another. RFC 2991 made the hidden policy legible: stability is not free, but neither is remapping. An equal-cost label cannot decide which cost a network should bear.

Sources