Summary

  • RFC 5386 let IKEv2 verify a signature against a public key supplied by the peer without claiming that an outside authority had bound the key to a named person, host or organization.
  • Its decisive non-downgrade rule searched ordinary PAD entries first. A peer that matched one and failed its required authentication had to be rejected; only a peer matching none could enter the later BTNS path.
  • A wildcard BTNS entry had to be logically last, its Child SA identities could not overlap stronger entries, and only SPD rules explicitly marked BTNS_OK could carry the resulting traffic.

A failed identity was not a request for weaker treatment

Consider a gateway with one rule for a known partner and a later rule offering encrypted transport to otherwise unknown peers. An attacker claims the partner's identity but cannot produce the credential that the partner rule requires. A permissive implementation might continue searching, reach the anonymous rule and accept the same exchange with fewer guarantees. Every individual rule could look reasonable while the combined evaluation created a downgrade.

RFC 5386 closed that route. The Peer Authorization Database is ordered. Non-BTNS entries are evaluated before BTNS entries. A peer using a signature method may be considered for BTNS only when it matches no non-BTNS PAD entry. If it matches a stronger entry and then fails to authenticate properly, the proposal must be rejected.

The distinction is between “not known here” and “known claim disproved.” The first can be eligible for a deliberately anonymous service. The second is evidence that a named authentication path did not succeed. Turning that negative result into anonymous admission would let the weakest policy override the strongest precisely when the strongest detected trouble.

This is why the wildcard had to be last. Its position was not an optimization hint for a table search. It assigned authority among policy branches. The wildcard could receive traffic left outside the named relationships; it could not act as an exception handler for them.

The signature authenticated a key-shaped identity

BTNS did not remove cryptography from IKE. A peer sent a bare public key in a CERT payload and produced an AUTH signature with the corresponding private key. Verification could establish that the party constructing the exchange controlled that private key. RFC 5386 called the peer “authenticated” in that bounded sense.

What had not appeared was an external statement that the key belonged to a particular company, administrator, host name or address. RFC 5386 therefore introduced a local PUBLICKEY identity type whose value was the key itself. The type was not transmitted as a new IKE identifier. After the ordinary PAD search found no match, the implementation could coerce its local view of the peer to PUBLICKEY and search the BTNS entries.

That sequence matters more than vocabulary. Signature verification is a real receipt: key K produced a valid response bound to this IKE exchange. It is not a receipt that organization O authorized K, that address A belongs to O, or that an application principal may perform operation P. The PAD can choose to authorize the key-shaped identity for a bounded purpose without pretending those absent bindings exist.

Later standards make the distinction easier to see. RFC 7670 restored algorithm-independent raw-public-key support to IKEv2 using SubjectPublicKeyInfo and says deployments need out-of-band key validation when they want confidence in authenticity. RFC 7619 created an explicit NULL Authentication method and ID_NULL, preserving the IKE exchange binding while declining to certify a peer identity. These mechanisms are not proof of RFC 5386 deployment. They show that the standards lineage continued to separate a protected exchange from an externally grounded identity.

One wildcard, two searches, three vetoes

RFC 5386 permitted at most one wildcard BTNS PAD entry. It had to follow every non-BTNS entry. The text described a simple implementation: search once with the peer-asserted identity; if no ordinary entry matches, convert the local identity to PUBLICKEY and search again for BTNS entries.

That first search creates the first veto. A matching known identity owns the decision. Failure ends the attempt.

The second search creates a narrower grant. A specific public key or any public key may match a BTNS entry, but only after the stronger namespace declined jurisdiction. A missing match still ends in rejection.

A further PAD check applies when the peer proposes Child SA traffic selectors. The wildcard's allowed identities must not overlap identities governed by other PAD entries. In the RFC's gateway example, an unknown BTNS peer can obtain the anonymous service lane, but it cannot claim the address of a known host or a network reserved for an authenticated security gateway. This is the second veto: anonymous peer admission does not confer authority over arbitrary traffic identities.

The Security Policy Database supplies the third. BTNS traffic matches only SPD entries whose BTNS_OK flag is set. The fact that an implementation supports the mode does not silently enable it for every protected flow. Operators have to name the traffic class for which lack of network-layer identity is acceptable.

These controls answer different questions. PAD order asks which authentication regime has authority over the peer claim. Child SA constraints ask which traffic identities the admitted peer may assert. BTNS_OK asks which local traffic policy accepts that authentication regime. Flattening them into ike_success=true destroys the boundaries RFC 5386 added.

Encryption remained real; identity remained missing

RFC 5387 described the property of Stand-Alone BTNS as continuity of association. If the initial SA was not subverted, traffic inside it could receive IPsec integrity, anti-replay protection and confidentiality just as traffic inside another SA did. The same unauthenticated source remained on the association during its lifetime.

Continuity is useful. It can make an off-path reset or injection attack harder and protect a long-lived transfer after setup. It is also deliberately weaker than identity. “The entity on connection 23” is not “Bill Smith,” the holder of an approved account, or the owner of a network prefix. The first association can still be intercepted by an active man in the middle.

That bounded claim prevents two opposite errors. BTNS should not be dismissed as “no security”; an established, uncompromised SA can provide real packet protection. It should not be marketed as ordinary authenticated IPsec; the origin claim has been reduced to continuity with a key presented inside the exchange.

The audit needs both statements. Record that ESP integrity and confidentiality were applied under a particular SA. Record separately that the peer identity was unauthenticated or key-shaped. Neither field should overwrite the other.

Higher-layer authentication arrived later

Channel-Bound BTNS proposed a way to recover a stronger end-to-end claim. A higher-layer protocol authenticates its principal and binds that authentication to the IPsec channel. Strong channel binding can detect the case in which an attacker created two separate SAs and relayed traffic between them.

The timing is crucial. In ordinary authenticated IKE, an active MITM should cause IKE authentication to fail. In Channel-Bound BTNS, the IKE exchange can succeed, SAs can be installed and resources can be consumed before the higher-layer authentication discovers the intermediary. RFC 5387 therefore warned against combinations that expose reusable passwords or attackable password-derived material during that later exchange.

Connection latching addresses another boundary. A higher-layer flow may outlive one SA and pass through rekeying. The application needs evidence that successive SAs belong to the same channel. Without that inter-session binding, an on-path attacker may step into the rekey window. RFC 5386 mentions latching but does not define it; a BTNS deployment cannot claim continuity across SAs merely because each SA was internally valid.

The joined receipt is therefore explicit: IKE exchange, key fingerprint, SA pair, latch transition, higher-layer channel-binding value, principal authentication and application authorization. A later higher-layer success can strengthen the claim. It does not retroactively turn the earlier anonymous IKE identity into a certificate-backed name.

Better than nothing meant a lower bound, not a fallback ladder

RFC 5386 explicitly did not define opportunistic fallback to cleartext when a peer lacked IKEv2. RFC 5387 said BTNS should substitute for no protection, not for stronger protection. Those two boundaries matter operationally.

An implementation that tries strong authentication, accepts failure, tries BTNS, accepts failure and then sends cleartext has built a downgrade ladder that the specification did not authorize. A safe decision graph does not rank outcomes only by whether communication eventually happened. It preserves the policy branch and the reason every other branch stopped.

RFC 7619 later articulated a comparable rule for NULL Authentication: where authenticated IKE is possible for a traffic-selector range, unauthenticated IKE should not be allowed to displace it. It also warned that anonymous peers must not use selector narrowing to divert traffic intended for another host. The later mechanism changed protocol details while retaining the governance principle—weak modes require explicit, non-overlapping authority.

The negative receipt is therefore valuable. “Rejected because known peer authentication failed” demonstrates that the stronger boundary worked. If dashboards retain only successful tunnels, they erase the event most capable of proving the wildcard did not swallow an authentication failure.

The policy table is executable governance

RFC 5386 exposes a general problem in control planes: a rule set is not only its members. Order, match semantics and terminal behavior determine who exercises authority. Two installations can contain the same named and wildcard entries and still implement different security if one continues after failed authentication or permits overlapping selectors.

A review should capture the PAD generation, logical order, match result, authentication method, terminal disposition and selector constraints—not merely a screenshot of the entries. Tests should present three peers: an unknown key that is allowed into the anonymous lane, a valid known identity that uses the stronger lane, and an attacker who claims the known identity but fails its authentication. The third must be rejected rather than reclassified.

The same test should propose a Child SA for a range reserved to a known peer. Admission to BTNS must not make that selector legal. Finally, the test should target an SPD rule without BTNS_OK; the anonymous peer must not inherit access simply because another traffic class permits BTNS.

These are small tests with institutional consequences. They prove that a local exception remains subordinate to the relationship it was designed not to replace.

Evidence boundary

The source record establishes standards text, historical status, processing rules and stated threat models. It does not establish that a current product implements BTNS, that a public deployment uses it, that a specific tunnel was attacked, or that current implementations preserve the two-pass semantics. RFC 4306 was superseded, raw-key encoding evolved, and later unauthenticated IKE mechanisms received their own specifications. Any deployment claim needs product, configuration and runtime evidence of its own.

Lu Heng's reality-layer discipline is useful here because the RFC already divides the world into receipts. The asserted ID is one object. The verified signature is another. The ordered PAD decision, selector authorization, SPD rule, installed SA, protected packet, higher-layer principal and application effect are still others. The architecture becomes safer when each is allowed to mean only what produced it.