Summary

  • A BGP confederation presents one externally visible AS while preserving multiple Member-AS policy domains inside. AS_CONFED_SEQUENCE and AS_CONFED_SET support internal loop detection and history, then must be removed before a route is advertised outside the confederation.
  • The removal is a compatibility contract, not proof that the internal path is irrelevant. LOCAL_PREF, NEXT_HOP, MED, candidate visibility, Member-AS identity and route-reflector design can change reachability while the external AS_PATH, public collectors and peer dashboards remain stable.
  • Leadership should permit topology compression only when the operator keeps a pre-sanitisation evidence ledger, owns every membership transition, tests RIB/FIB and packets at each boundary, and can reverse a Member-AS migration without relying on the public route to diagnose it.

At 01:40, an operator begins the last stage of an internal reorganisation. Border router Delta is moving from Member-AS 65021 to 65031. Both belong to confederation AS 64500. The external design is intentionally unchanged: transit providers and customers should continue to see one AS, one peering identity and the same public AS_PATH.

The change record says the new configuration is active. Delta's external peers are green. A route collector still shows 64500 as the first hop. The origin prefix remains valid under the same public origin authorization. The executive dashboard therefore labels the migration successful.

The internal adjacency tells another story. Delta now opens as Member-AS 65031. Its neighbor Echo still classifies the link as a confederation peer in 65021. On one implementation the adjacency never forms because the two speakers disagree about whether and how the link belongs to the confederation. On a surviving alternate path, a route arrives with a different internal sequence, retains a next hop that the new Member-AS cannot resolve and encounters a different MED comparison set. The session to the customer is still alive, but the selected route is not installed.

This incident is synthetic. Every mechanism is real. The external AS_PATH has not failed to reveal the cause; the protocol deliberately removed the internal Member-AS history before export. A confederation is valuable because outside networks do not need to understand its internal topology. That same compression becomes dangerous when the operator mistakes a clean public statement for proof of clean internal operation.

One identifier, several operating domains

RFC 5065 defines three identities that must remain distinct. The AS Confederation Identifier is the globally visible number representing the whole confederation to outsiders. A Member-AS is an internal autonomous system inside that confederation. Its Member-AS Number identifies it to other members.

The confederation identifier is used in transactions with peers outside the confederation: the OPEN identity and the ordinary AS_SEQUENCE presented to the Internet. The Member-AS number is used between participating internal domains. These are external BGP sessions in packet and finite-state-machine terms, yet the routes they exchange receive important internal treatment for selection and policy.

This is not merely a large AS renamed into smaller boxes. Each Member-AS can establish its own full mesh or route-reflection design, own a portion of policy and maintain a different IGP. The confederation reduces the need for every router across the entire organisation to participate in one full IBGP mesh. It also gives the operator AS_PATH-like internal policy boundaries without asking every external peer to track them.

The public identifier therefore performs a thin coordination function. It tells outsiders which AS is speaking and which AS hop the route crossed. It does not say which internal team admitted the route, which Member-AS changed preference, which next hop remained reachable or which control plane programmed the packet path. One public identity can be operationally coherent, but the number alone does not make it so.

The path is preserved, then deliberately erased

Inside the confederation, BGP extends AS_PATH with two segment types. AS_CONFED_SEQUENCE is an ordered list of Member-AS domains traversed by an UPDATE. AS_CONFED_SET is an unordered set used with aggregated information. When a speaker advertises a learned route to another Member-AS, it prepends its own Member-AS number to the confederation sequence, creating the segment when none exists.

That internal sequence is not decorative metadata. A speaker that finds its own Member-AS number in an incoming confederation segment treats the route as a loop, just as an ordinary external speaker rejects a path containing its own AS. The confederation identifier has its own loop boundary: if it appears in a received path, a member treats it as its local AS identity.

At a true external boundary, the rule changes. Before advertising the route, the member must remove every AS_CONFED_SEQUENCE and AS_CONFED_SET segment. It then prepends the confederation identifier as ordinary AS_SEQUENCE information. RFC 6793 also excludes confederation segments from AS4_PATH, preventing the four-octet compatibility attribute from becoming a side channel for internal history.

An outside collector consequently receives a truthful but compressed statement. The route crossed AS 64500. It does not receive the sequence 65021 65017 65009, because the extension exists partly to prevent internal restructuring from becoming a global policy event. The external peer does not need confederation support; only every participating member does.

Compression is not falsification. The mistake is asking the compressed record to answer an internal question. A public AS_PATH can prove neither Member-AS traversal nor internal policy custody because those facts were removed by design. If the network fails to archive them before the border, no external observatory can reconstruct them later.

Membership is executable configuration

A topology diagram may show two boxes inside the same confederation. BGP relies on something stricter: both neighbors must agree through configuration on their identities and relationship. Juniper's current documentation states the operational result plainly—adjacencies do not form when two neighbors disagree about whether the adjacency falls within a particular confederation.

RFC 5065 adds further consistency rules. A route received from a different member should begin with AS_CONFED_SEQUENCE. Confederation segments must never arrive from a neighbor outside the local confederation. Speakers that participate must understand the new segment types. A mismatched speaker or malformed path is not made safe by the fact that the external number is correct.

Current error handling narrows some failures. RFC 7606 treats a malformed AS_PATH as withdrawn rather than automatically destroying the whole BGP session. That limits collateral damage, but it also weakens familiar health signals. The peer can remain Established while selected NLRI disappear. On an internal session, different routers can retain different paths, creating unreachability, suboptimal forwarding or even long-lived loops.

The control is therefore not “the BGP session stayed up.” It is a chain: the two configured identities agree; the OPEN exchange proves the expected peer; the received AS_PATH contains the correct segment class and member; the policy takes the intended action; the resulting RIB and FIB agree; packet tests reach the intended exit.

EBGP mechanics, internal consequences

Links between different Member-AS domains use EBGP sessions. If operators stop at that label, they import the wrong expectations. RFC 5065 deliberately changes how several attributes and selection rules behave across those links.

For the IBGP-versus-EBGP preference step, a path from another member of the same confederation is treated as internal. CONFED_SEQUENCE and CONFED_SET do not add to AS_PATH length. LOCAL_PREF may cross the member boundary. NEXT_HOP and MED may remain unchanged. These exceptions allow the confederation to behave as one policy system while retaining internal subdivisions.

They also carry hidden dependencies. If every member shares one IGP, an unchanged next hop may be reachable everywhere. If members operate independent IGPs, that assumption can fail. RFC 5065 explicitly notes that policy may need to set NEXT_HOP. A route can therefore look correct in the receiving BGP table but fail recursive resolution or hardware programming.

MED comparison is equally conditional. Normal comparison skips the confederation segments and looks to the first ordinary AS_SEQUENCE. Implementations may expose knobs to compare MEDs across member paths. A change in candidate visibility, member topology or comparison policy can alter the winner without changing the external AS_PATH length at all.

This is why “confed EBGP” is a dangerous shorthand. The wire session crosses AS numbers, but policy and route selection intentionally inherit internal properties. The operator must document the exact semantics rather than reason from the session label.

The topology can oscillate behind a stable public route

RFC 5065 warns that improper configuration can duplicate advertisements, waste resources, cause flaps and delay convergence. It also points to the persistent-oscillation condition described in RFC 3345. Confederations and route reflectors can expose different candidate sets at different decision points; MED can then produce a cycle in which each local choice changes the information available to another router.

The public route may look merely noisy—or completely stable—while internal best paths keep changing. A collector beyond the border sees the confederation identifier after internal segments have been stripped. It may see only whichever path survives export policy. It cannot show the rejected candidate that caused the next internal decision.

Mitigation is not one universal knob. Topology-aware IGP costs can reduce some oscillation conditions. Consistent tie-breaking can help. Duplicate advertisements can be filtered according to the intended redundancy design. Route reflectors may need richer candidate exchange. None of these controls substitutes for observing the actual candidate set at every decision point.

The monitoring question is not simply “did the best path change?” It is “which candidates were eligible at this node in this topology epoch, which MEDs were comparable, which member and reflection paths delivered them, and which rule removed each losing route?” Without that record, a cycle becomes an anecdote about vendor behaviour rather than a reproducible policy failure.

Private Member-AS numbers need a collision register

Operators often use private-use ASNs for Member-AS domains. RFC 6996 currently reserves 64512–65534 and 4200000000–4294967294. Private space avoids consuming globally unique numbers for identities intended to disappear at the public border. It does not make those numbers unique inside every future operating scope.

RFC 5065 does not require Member-AS numbers to be private. Whatever range is chosen, each member must be uniquely identified within the confederation. A merger, backbone interconnect or staged federation can bring two previously separate scopes together. If both contain Member-AS 65021, the collision is no longer cosmetic. Loop detection, AS-path policy, logging and responsibility mapping can all attach two meanings to one number.

The remedy begins before packets meet. Maintain a registry of Member-AS number, owner, geography, IGP domain, route-reflector cluster, supported AFI/SAFI, four-octet capability and planned retirement. Compare that inventory during acquisition and interconnect design. Where collision exists, renumber one side through a bounded migration whose old and new identities can be observed simultaneously at controlled boundaries.

Ordinary private-AS removal is not the same control. Juniper documents that remove-private processing occurs after confederation member segments have already been removed. The confederation boundary strips a standards-defined internal segment class; private-AS filtering removes private values from ordinary external path content under its own rules. Treating one as evidence for the other conceals precisely which boundary acted.

Build the evidence before the sanitisation boundary

The minimum forensic record is per adjacency and per AFI/SAFI. It includes local Member-AS, expected peer Member-AS, confederation identifier, negotiated four-octet capability, raw received and advertised AS_PATH segment types, policy version, LOCAL_PREF, MED, NEXT_HOP, origin, best-path reason and withdrawal action.

That record must span stages. Preserve pre-policy and post-policy Adj-RIB-In at the receiving member, Loc-RIB candidates, pre-policy and post-policy Adj-RIB-Out toward the next member, then the final external Adj-RIB-Out after confederation sanitisation. Hash the topology and policy versions so a path can be associated with the configuration that actually evaluated it.

Control-plane truth remains incomplete. Capture recursive next-hop resolution, installed FIB or ASIC state and packets toward the intended exit. A correct AS_CONFED_SEQUENCE does not prove that the IGP reaches the unchanged next hop. An installed route does not prove that traffic follows the expected member boundary. A public collector is useful for the final external statement but cannot replace these internal witnesses.

Alerts should detect divergence, not only failure. Flag a peer whose configured Member-AS differs from the observed OPEN identity; an external neighbor sending confederation segments; an internal neighbor omitting them; a private Member-AS collision; a treat-as-withdraw event without session loss; a best-path change without external AS_PATH change; and repeated candidate cycles involving the same MED set.

Migrate identity as a protocol change

A Member-AS move is not a label edit. It changes the value prepended to internal paths, the loop-detection identity, policy match conditions, route tags and telemetry dimensions. It may change which sessions are classified as intra-member, inter-member or truly external. It can alter the route-reflector domain and the next-hop assumptions attached to it.

Begin with an old-to-new map covering routers, sessions, policies, communities, monitoring, collectors and incident ownership. Prove that the target Member-AS number is unique in every scope that can connect during the migration. Inventory which devices understand four-octet ASNs and confederation segments, and identify any intermediate system that would transform AS_PATH unexpectedly.

Use a narrow canary. Select one redundant boundary, one AFI/SAFI and a controlled prefix set. Capture exact paths in both directions before the change. Apply the new membership, then test adjacency identity, raw segment sequence, candidate set, best-path reason, recursive next hop, FIB and traffic. Observe enough time to expose MED cycles and delayed route-reflector effects.

Rollback must restore more than the old number. It must restore the old membership list, peer classification, path policy, reflection relationship, telemetry labels and route state. Define what evidence authorizes the rollback and what stale routes must be cleared. If restoration depends on reconstructing internal history from the public AS_PATH, the rollback was never viable.