Summary

  • TCP-AO can authenticate resets while both endpoints retain valid connection state, but a rebooted endpoint may have lost the traffic keys and connection context required to authenticate a reset for the old session.
  • Rejecting that unverifiable reset preserves TCP-AO's security boundary, yet leaves the surviving endpoint with stale state until keepalives, application recovery or carefully preserved key material clears the mismatch.

The asymmetry created by a reboot

Imagine two routers joined by a long-lived TCP connection protected with the TCP Authentication Option. Before anything fails, the arrangement is straightforward. Each endpoint has TCP state for the connection and the cryptographic material needed to validate protected segments. A reset sent within that shared context can carry a valid authenticator and be accepted.

Then one router reboots.

The returning router may remember that it is configured to use TCP-AO with its peer. That is not the same as remembering the departed connection. The old connection had particular endpoint values, initial sequence numbers and derived traffic keys. RFC 5925 makes those connection values part of the context used by the key-derivation function. If the reboot erased them, the restarted endpoint cannot reconstruct a valid authenticator merely from the packet that arrives next.

Ordinary TCP would often answer traffic for a nonexistent connection with an RST. That response helps the sender discard stale state quickly. Under TCP-AO, however, the restarted endpoint may not know the initial-sequence-number pair needed for the old connection's traffic key. RFC 5925 says that a reset in this situation should be sent without authentication. If the surviving peer requires TCP-AO, it cannot validate the segment and drops it.

The result is an awkward but deliberate asymmetry. One side says, in effect, “there is no connection here.” The other side still has a connection, but it has no authenticated reason to believe the message.

Why accepting the reset would undo the protection

It is tempting to treat the unauthenticated reset as a special recovery signal. That shortcut would reopen the attack TCP-AO is meant to close. An attacker able to forge the relevant addressing and sequence information could send the same kind of unauthenticated RST and tear down a protected session.

TCP-AO therefore does not distinguish a benign post-reboot reset from a malicious forged reset by intention. It asks whether the segment can be authenticated in the connection's established security context. Once the restarted endpoint has lost that context, its honest reset and an attacker's fabricated reset meet the receiver at the same boundary: neither carries acceptable proof.

This is not a failure of the authenticator. It is the consequence of applying authentication consistently. A protocol cannot make a stateless exception for a message whose security meaning depends on state and still claim that the exception is authenticated.

The stale connection is now a recovery problem

Dropping the reset protects the session from forged teardown, but it prevents the fast cleanup ordinary TCP would have provided. The peer that did not reboot can continue to hold the obsolete connection until its own liveness machinery concludes that the path is gone.

RFC 5925 treats that accumulation as an operational concern. Implementations need a way to detect excessive TCP connection state and clear it when necessary to protect memory. The RFC also recommends that TCP-AO connections use TCP keepalives. An application protocol can provide a similar finite bound; BGP, for example, has its own keepalive exchange.

The important distinction is what those mechanisms prove. A keepalive does not authenticate the rebooted peer's connectionless reset. Instead, it lets the surviving endpoint discover, on its own timeline and within its existing state machine, that the connection no longer makes progress. The security boundary remains intact while cleanup moves from immediate peer notification to liveness-driven expiry.

Sources