Summary

  • A firewall HA upgrade left Router B with zero BGP prefixes for 121 seconds, even though the sessions reported Established and traffic still went toward the firewall.
  • The speaker’s packet captures and lab reproduction traced the delay to an IPv6 peer-context parsing error and a TCP reset; the IPv6 End-of-RIB wait also held back IPv4 route installation.

“Established” looked reassuring. The route table had a less diplomatic answer: zero prefixes for 121 seconds.

That was the failure Marian Rychtecký presented at NANOG 98. During a high-availability upgrade of a firewall pair, BGP sessions returned to Established, but Router B received no prefixes. Traffic was still sent toward the firewall. The firewall had not installed the expected routes, so packets were blackholed for almost two minutes.

The number that mattered was 121 seconds. End-of-RIB arrived close to a 120-second reconnect timer, but the timer was a clue, not the whole cause. Rychtecký’s troubleshooting found that IPv4 recovery was waiting on a specific IPv6 BGP session. Graceful Restart treated the negotiated peer state as unfinished until the End-of-RIB marker arrived. One address family’s stuck session therefore held up routes in the other.

Packet captures revealed a TCP reset on the new session. In a lab reproduction, an expected IPv6 peer address, 2001:db8:1::2, was stored as 2001:db8:1:. The operating system could not place the connection in the correct VRF and BGP context, so it returned a reset. The old TCP connection still appeared active during part of the failover, leaving the new one to collide with stale state.

The presentation’s point is sharper than “Graceful Restart can fail.” A green BGP session state did not mean the router had routes; a populated control-plane status did not mean forwarding had recovered. The firewall upgrade was the trigger. The malformed peer context and cross-family wait were the mechanism. The empty RIB and blackholed packets were the impact.

The speaker reports that a shorter connection timer made the reconnection and End-of-RIB exchange finish faster in this case. That is a reported workaround, not a universal setting: changing a timer can shorten one wait while creating a different failure window elsewhere. The durable lesson is to test the full chain that moves from TCP reconnection, through per-family End-of-RIB, to route installation and packet delivery.

The NANOG slides omit device models, firmware versions and customer traffic-loss measurements; those records would show whether this was a single implementation defect or a pattern across deployments.

Sources