Summary

  • BGP multipath admits several qualifying routes into local forwarding, but it does not guarantee equal bandwidth, physical diversity or an even byte split.
  • The control chain has distinct authorities: BGP decides eligibility, recursive resolution and hardware decide group membership, and the hash plus the flow population decide actual use.
  • A credible change record follows candidates into the RIB and FIB, then measures per-member packets, bytes, queues and failures. A route-table screenshot proves only one link in that chain.

The arithmetic that the route table never promised

Imagine a bounded test, not a reported incident. Two routes to the same prefix pass an implementation's multipath checks. Both next hops appear in the forwarding table. The change ticket calls the result “two-way load balancing” and forecasts half the traffic on each exit.

Then one long-lived, high-volume flow lands on member A. Its queue grows while member B remains comparatively quiet. The routes are still present. BGP sessions are still established. The ECMP group is not necessarily defective. What failed was the inference that two installed paths implied two equal halves of carried bytes.

That distinction is structural. The BGP decision process in RFC 4271 evaluates eligible routes, identifies a selected route, installs it in the Loc-RIB and determines the immediate next hop. Common multipath implementations extend the local outcome by retaining additional qualifying paths. They do not create a new inter-domain promise that every peer will advertise, install or use the same set. RFC 4271, section 9.1.2

Even the word “equal” needs a subject. Paths may be equal under configured multipath criteria while differing in next-hop topology, failure domain, capacity or commercial meaning. Cisco documents a set of attributes that must match for its multipath procedure and still designates one best path for advertisement.

FRRouting places its multipath check after earlier selection criteria and offers multipath-relax, which can treat equal AS-path lengths as sufficient where exact paths would otherwise differ. Arista documents a different default for that relaxation in the cited EOS context. These are implementation contracts, not a universal definition of equivalent risk. Cisco best-path documentation, FRRouting BGP documentation, Arista EOS BGP documentation

Three decisions, not one

The first decision is admission. Which received routes are feasible, policy-compliant and sufficiently equivalent to coexist? A configured maximum-paths value only caps the answer. It cannot prove that all candidates resolved, reached the FIB or fit the hardware group's limits.

The second decision is construction. Each admitted BGP route must resolve to a usable forwarding next hop. Several control-plane paths can collapse onto shared infrastructure, or one can disappear because recursion fails. The RIB may therefore describe an intention that the programmed ASIC does not reproduce. A reliable audit records the forwarding group identifier and its actual members rather than treating a route-table line as a proxy.

The third decision happens for every flow. ECMP commonly maps a flow key into a member through a hash. RFC 2991 explains why keeping packets in one flow on one path helps avoid reordering; RFC 2992 analyses one hash-threshold method and its behavior when the set of next hops changes. Uniformly spreading hash values means roughly uniform placement of flow identifiers under suitable assumptions. It does not make all flows the same size. RFC 2991, RFC 2992

This is the crucial operational separation:

  1. route equality concerns control-plane eligibility;
  2. member equality concerns programmed forwarding choices;
  3. traffic equality concerns the observed population of packets and bytes.

A success at one layer is necessary evidence for the next, never a substitute for it.

Entropy has a topology

The hash can use only fields visible to the forwarding node and supported by its platform. IPv6 offers the flow label as an additional source of entropy, particularly when transport headers cannot be inspected. RFC 6438 also explains why tunnels, fragments and opaque payloads complicate the choice. A fixed outer tunnel tuple can make many inner conversations look like one flow unless the equipment can use deeper fields or a well-formed flow label. RFC 6438

Encryption does not automatically break ECMP, and a five-tuple is not universally available. The accurate question is narrower: which fields did this device use for this address family and encapsulation, with which seed and granularity? Vendor defaults, hardware generations and configuration modes differ. Arista's documentation, for example, exposes seed controls and resilient ECMP behavior partly to address polarization and member changes. That is evidence of a local implementation surface, not a default to project onto every router. Arista IPv4 forwarding documentation

Flow composition matters just as much. RFC 7424 describes hashing as a many-to-one mapping from flows onto component links and calls out the imbalance created by large flows. Ten thousand small connections can average well; one backup, replication stream or high-rate customer transfer can dominate a member for hours. The correct denominator may be packets, bytes, queue time, drops or service latency, depending on the claim being tested. RFC 7424

Ordinary ECMP also does not know that one path has ten times another's usable capacity unless a separate weighted or adaptive mechanism supplies that distinction. Equal-cost membership can be entirely correct and operationally unsuitable for unequal links. Calling that “bad hashing” hides a policy error.

Visibility is not use

ADD-PATH makes the boundary clearer. RFC 7911 lets a speaker advertise several paths for the same prefix without implicitly replacing the previous one. A receiver thereby gains visibility. It is not obliged to place every advertised path in an ECMP group, and the extension says nothing about the receiver's per-flow distribution. Conversely, a router can install local multipath without advertising all of those paths to an ordinary peer. RFC 7911

PIC is different again. It prepares or programs repair state to shorten the interruption after a failure. Multipath concerns members used concurrently in steady state. The same next hop may participate in both designs, but a fast standby and a traffic-bearing member are not interchangeable evidence.

Junos documentation illustrates another implementation boundary: BGP multipath selection and a forwarding-table load-balancing policy are separate parts of the documented workflow. Its historical “per-packet” label commonly describes a per-flow hash rather than unconstrained packet spraying. The exact forwarding mode must be read from the platform contract and verified on the device. Junos BGP load-balancing documentation

Member changes are traffic changes

Removing a failed member is not merely subtracting a line from a group. Some flows must move. Depending on the algorithm and timing, packets can be reordered or lost; the surviving member can become congested. Restoration can trigger a second redistribution event. Resilient or consistent hashing aims to limit unnecessary remapping, but it does not freeze every flow or remove the need to observe the transition.

This makes a maintenance test more demanding than “both links came back.” It should capture the group before failure, the set of flows that moved, the queues and drops during convergence, and the group after restoration. Where ECMP occurs at several layers, correlated hashing can create polarization: a choice at one device feeds a restricted input into the next. Changing one seed may alter the symptom without proving that shared fate or capacity mismatch has been addressed.

Follow a candidate until it becomes a packet

Evidence begins with the intended scope: prefixes, address families, peer class, path cap, owner and rollback condition. Preserve the Adj-RIB-In alternatives and the exact criteria that made them eligible. Identify the designated best route as well as each additional path. If multipath-relax is involved, record what equality was relaxed; do not translate “same length” into “same trust.”

Next, trace recursive resolution and inventory failure domains. Two next-hop addresses are not independent evidence when both traverse one line card, tunnel, fabric plane or remote circuit. Inspect the FIB or hardware-group object and confirm every intended member is programmed. Platform counters that show failed programming or a reduced group are more probative than a successful configuration commit.

Then measure behavior over a meaningful interval. Collect per-member packets, bytes, drops and queue occupancy. Use flow telemetry to find large contributors without collecting unnecessary payload. Run a controlled population of many flows rather than one convenient ping, from the ingress locations and address families that matter. Repeat through member removal and restoration. A packet probe proves reachability for its own key; it does not prove distribution for every customer.

Rollback deserves the same chain. Reverting the BGP knob is not completion. Confirm the old candidate set, RIB outcome, FIB group and packet behavior have returned—or document the new steady state if the change exposed a pre-existing mismatch.