Summary

  • The traditional BGP HoldTimer measures qualifying messages received from a peer. It can remain satisfied while the local speaker is unable to send a withdrawal, UPDATE or KEEPALIVE in the reverse direction.
  • RFC 9687 adds a locally configured SendHoldTimer. If no BGP message is successfully sent for the configured interval, the local speaker records the reason, releases BGP resources, drops TCP and returns the session to Idle.
  • Safe deployment depends on evidence and restraint: distinguish message generation from transmit progress, set the timer above the negotiated HoldTime, canary the exact failure, and prevent repeated resets from amplifying peer overload.

Consider an illustrative failure scenario. At 03:17, a transit edge learns that a destination has failed and generates a withdrawal. Its neighbour continues sending KEEPALIVEs every thirty seconds. The ordinary HoldTimer restarts on each arrival, so the BGP finite-state machine remains Established. The local process has done the intellectually correct thing: the route is gone from its own selected state and a withdrawal exists for export.

But the bytes do not move. The neighbour is not draining its TCP receive stream. The local send queue grows, retransmission continues, and the advertised view across the boundary freezes at the last successfully delivered state. Traffic keeps following a path that the local AS has already rejected. A green session has become a source of stale authority.

This is the narrow failure RFC 9687 addresses. It does not ask whether packets can cross the link, whether TCP has closed, or whether the peer can still speak. It asks whether this BGP speaker has made outbound application progress within a bounded time. If not, the side that possesses the evidence is allowed to end the relationship.

One TCP connection can carry two different truths

TCP supplies a bidirectional byte stream, but progress in one direction does not guarantee progress in the other. The canonical RFC 9687 example is a peer whose receive window becomes zero. That peer may still write and transmit BGP messages toward the local speaker while refusing more bytes in the reverse stream. An overwhelmed process, stuck reader or queue failure can produce the same operational asymmetry without a clean transport failure.

RFC 4271's HoldTimer is a receive-side instrument. When the local speaker receives an UPDATE or KEEPALIVE, the timer is restarted according to the state-machine rules. This is useful evidence that something from the neighbour is arriving. It is not evidence that the neighbour has accepted the local speaker's most recent routing truth.

Several weaker signals are often mistaken for that proof. A route process can generate an UPDATE without placing its bytes on the wire. The kernel can accept bytes into a send buffer without advancing them to the remote host. TCP can remain Established while the remote application does not read. A NOTIFICATION can be constructed without being deliverable. Each observation has value, but none should be promoted to a claim it cannot support.

The asymmetry matters because routing messages are not interchangeable heartbeats. A blocked KEEPALIVE may eventually expose the fault, but a blocked withdrawal can preserve a path that should no longer exist. A replacement route can remain local while a remote AS continues forwarding according to an obsolete advertisement. Convergence is therefore constrained by the last successful outbound message, not by the freshness of inbound traffic.

RFC 9687 gives the blocked sender a clock

RFC 9687 updates the BGP finite-state machine with SendHoldTime, SendHoldTimer and event 29, SendHoldTimer_Expires. SendHoldTime is a local, per-peer policy value. It is not negotiated as a new capability and does not require the remote system to understand the mechanism.

When the speaker enters Established after receiving the qualifying KEEPALIVE in OpenConfirm, it starts SendHoldTimer if the configured value is non-zero. Each successfully sent BGP message restarts the timer, unless SendHoldTime or the negotiated HoldTime is zero. Leaving Established stops the timer as the implementation releases the session's resources.

The constraints carry policy meaning. A non-zero SendHoldTime must be greater than the negotiated HoldTime. If the negotiated HoldTime is zero, SendHoldTimer is stopped even if a non-zero local value was configured. The send-side detector must not become a faster parallel HoldTimer that contradicts the timing relationship accepted for the session.

RFC 9687 recommends enabling the mechanism by default and proposes the greater of eight minutes or twice the negotiated HoldTime as the default SendHoldTime. That is guidance, not a universal proof that eight minutes is correct for every peer. A route server, a heavily loaded reflector, a constrained customer router and an internal core session expose different queue dynamics and reset costs.

On expiry, the local speaker must log Send Hold Timer Expired, release BGP resources, drop the TCP connection, update retry state and transition to Idle. It may try to send a NOTIFICATION, but only if that attempt does not delay teardown. This exception respects the evidence: the failure being diagnosed may make the notice itself impossible to deliver.

IANA assigns error code 8, subcode 0, to Send Hold Timer Expired, with no additional data. The local log and state history therefore remain essential. A remote error counter cannot be the sole record of an event whose defining property is that the remote side may not receive it.

HoldTimer, SendHoldTimer and BFD answer different questions

The ordinary HoldTimer asks: when did a qualifying BGP message last arrive? SendHoldTimer asks: when did this speaker last succeed in sending any BGP message? BFD asks whether a forwarding path between systems satisfies a rapid continuity test. TCP retransmission and keepalive mechanisms ask yet other transport questions.

Those instruments can disagree without any of them being defective. BFD may remain healthy because the forwarding path works while a remote BGP application is wedged. HoldTimer may remain healthy because inbound KEEPALIVEs continue. TCP may remain Established because neither endpoint has closed. SendHoldTimer may nevertheless expire because the local BGP stream has made no outbound progress.

The disagreement is the diagnosis. Collapsing it into one red-or-green peer state destroys information. An incident record should preserve at least four separate propositions: inbound BGP receipt, local outbound BGP transmit progress, remote route processing, and data-plane forwarding. Only the first two are directly addressed by the paired BGP timers; the latter two need route and packet evidence.

This boundary prevents overclaiming. SendHoldTimer does not identify every BGP zombie, every stale route or every remote application failure. It detects a sustained local inability to transmit BGP messages under the implementation's semantics. Nor does its expiry prove a forwarding failure. It provides a bounded reason to discard the current session state and force a new exchange.

Closing the session is corrective but not free

A reset invalidates or withdraws routes learned over the session. It may trigger best-path changes, FIB updates, alternate advertisements and a full-table exchange after reconnection. On a high-fan-out reflector or route server, one corrective teardown can create a measurable convergence wave.

An overly short SendHoldTime can convert temporary scheduler delay, control-plane backpressure or a benign zero-window interval into an avoidable reset. If the peer reconnects while the underlying reader remains overloaded, repeated establishment and full-table transfer can add precisely the pressure that caused the blockage. The health mechanism then becomes an oscillator.

An overly long value carries the opposite risk: stale routing authority survives while withdrawals and replacements wait behind the blocked stream. The decision is a balance between false teardown cost and stale-state exposure, but it is not an arbitrary slider. The negotiated HoldTime sets a normative lower relationship; measured queue behaviour, maintenance pauses, update bursts, platform scheduling and route scale set the operational margin.

Peer classes deserve explicit values. A customer edge with a modest table and tight recovery objective need not inherit the same delay as a route reflector carrying millions of paths. A route server may require additional consideration for participant fan-out and reconnect storms. A constrained device may need more margin but also creates greater stale-state risk when overwhelmed. Defaults are a starting hypothesis, not evidence.

If the implementation offers peer oscillation damping after expiry, it can limit rapid retry. It cannot repair a starved receiver, undersized control plane or pathological route churn. Damping should remain subordinate to diagnosis and capacity correction, because suppressing reconnection also extends the period without fresh route exchange.

A canary must preserve the forwarding path

The useful test does not simply unplug the interface. That would prove ordinary loss detection and might trigger BFD or HoldTimer before the send-side mechanism is exercised. A valid canary keeps the transport path and inbound BGP stream alive while deliberately preventing the remote BGP receiver from draining outbound data.

BIRD 3.3.0 documents disable rx for testing and warns against production use. Such a mechanism can create the needed asymmetry in an isolated lab, but it is not an operational setting to leave enabled. The test should use a disposable peer or controlled route domain, bounded announcements, and an explicit recovery action.

The proof chain starts before the fault. Record the negotiated HoldTime, effective SendHoldTime, timer state, peer class, software build and baseline queue behaviour. During the fault, prove that inbound KEEPALIVEs continue, the local last-successful-send timestamp stops advancing, the socket's send queue or zero-window evidence persists, and at least one controlled UPDATE or withdrawal is blocked.

At expiry, capture the precise local reason, event time, optional NOTIFICATION attempt, TCP close and FSM transition. After removing the fault, prove that the session re-establishes without a loop, the controlled route state is corrected, forwarding follows the restored view, and queue pressure returns to baseline. A configuration screenshot alone proves none of these transitions.

FRRouting's current support list names RFC 9687, and BIRD exposes a concrete send hold time control. Those implementation documents justify testing the deployed build; they do not replace it. Packaging, backports, default changes and local configuration can separate a production binary from a documentation claim.

Evidence ownership is part of session ownership

The organization that may reset a business-critical interconnection must also retain the evidence that makes the reset defensible. Useful records include the last generated message, last successful BGP transmission, kernel queue age and depth, TCP receive-window observations, retransmission state, inbound message timestamps, timer start and restart history, and the final teardown reason.

Route evidence must identify what could not be delivered. Which withdrawals or replacements were trapped? Which previously advertised paths remained active at the peer? How long did the stale view survive? A route collector may help, but its observation point and delay must be stated. Packet tests are still needed because a repaired BGP exchange does not by itself prove forwarding recovery.

This is also a practical data-sovereignty question. Socket telemetry may live in the operating system, timer events in the routing daemon, peer errors in an external portal and business impact in a separate NOC. If no accountable operator can join those records, the right to reset becomes either timid or arbitrary. Authority without evidence is unsafe; evidence without decision rights is inert.