Cloudflare's 2022 Outage Made BGP Change Staging

Summary

On 21 June 2022, Cloudflare says a network configuration change caused an outage in 19 data centers using its Multi-Colo PoP architecture. Those locations were about four percent of its network but handled a large share of traffic. Cloudflare attributed roughly half of total requests during the event to affected traffic, while noting that user impact varied by location. [1]
The intended change standardized informational BGP communities on site-local prefixes. On the MCP spine routers, a policy-term reordering placed a disabled-prefixes term before terms that should advertise critical site-local routes. The resulting route policy withdrew those prefixes. [1]
The withdrawal removed more than external reachability. Cloudflare says its servers could not communicate normally or reach customer origins, and its Multimog load-balancing layer could not move requests among compute clusters inside affected MCPs. Smaller clusters then received traffic comparable to larger clusters and overloaded. [1]
Engineers also had difficulty reaching the affected locations to undo the change. Backup procedures were required. During restoration, engineers sometimes overwrote one another's reverts, causing the problem to recur sporadically before the final location recovered. [1]
Cloudflare's process already included a change ticket, dry-run, peer review, and staged deployment. The control failure was more specific: none of the early rollout stages exercised an MCP location. The first representative test arrived when the final stage reached all MCP spines. [1]
BGP communities are metadata used by routing policy. They do not prove that an export policy is correctly ordered or that required prefixes remain advertised. RPKI origin validation addresses whether an origin AS is authorized for a prefix; it would not validate this internal term order, canary selection, or rollback procedure. [10][11][12][14][15][16]
Accountability should therefore ask for operational evidence: the exact before-and-after candidate configuration, a machine-checkable required-prefix set, route-policy simulation, an MCP-specific canary, commit-confirm behavior, independent management reachability, one authoritative rollback owner, per-location change records, and external route observations reconciled with router logs.
Public route collectors can help establish externally visible announcements and withdrawals, but they cannot reveal every private site-local route or internal policy decision. Operator records remain necessary. [17][18][19]
The public record does not establish malicious intent, negligence, a regulatory violation, every affected prefix, every customer loss, or whether all stated remediation remains deployed. Those limits should remain explicit.

A reviewed change can still be an untested change

Infrastructure organizations often use process labels as shorthand for control. A change was ticketed. A dry-run completed. Multiple engineers reviewed it. The rollout was staged. Each statement can be true while the operating risk remains untested.

Cloudflare's account of its 21 June 2022 outage makes that distinction unusually clear. The change passed through a change-request ticket, included a dry-run, received peer review, and followed a stepped deployment procedure. The early steps did not cause an outage. The failure appeared when the rollout reached the 19 locations using an architecture Cloudflare called Multi-Colo PoP, or MCP. None of the earlier stages had exercised an MCP location. The first stage that represented the relevant routing topology was also the stage that applied the configuration to all MCP spines. [1]

That is not an argument that tickets, reviews, dry-runs, or staging are useless. It is evidence that each control needs a defined claim. A ticket records authorization and intent. A review records that named people examined some representation of the change. A dry-run records what a tool calculated against a specified model or device state. A staged rollout limits scope only when each stage is representative of a distinct failure domain and small enough to stop before the next one.

The accountability question is therefore not, "Did the process exist?" It is, "What did each process step prove?" In this incident, the early rollout proved that the change did not disrupt locations using the older architecture. It did not prove that the policy would preserve required advertisements on the MCP spines. A green result from a nonrepresentative location could not answer the decisive question.

This gap matters because network configuration is executable operational authority. A route-policy term does not merely describe where traffic should go. Its order determines which routes a router advertises or withholds. Once committed, the configuration changes the reachability graph used by servers, peers, origins, operators, and recovery systems. The difference between a harmless metadata addition and a major outage can be one policy branch evaluated before another.

Cloudflare's report should not be reduced to the slogan that a bad configuration caused an outage. The useful lesson is narrower and more demanding: change controls must be bound to architecture, route invariants, failure domains, and recovery access. A staged procedure that skips the relevant architecture can create the appearance of caution without supplying evidence about the system that will actually fail.

The intended change was informational; the effective change was reachability

Cloudflare said it was standardizing the BGP communities attached to a subset of advertised prefixes, specifically adding informational communities to site-local prefixes. BGP communities are attributes that operators use to label routes and influence routing policy. The standard communities specification defines a way to group destinations so routing decisions can apply to a class of routes rather than one prefix at a time. Large Communities extend the format for modern operational needs. [1][11][12]

The intended additions were not described as an effort to withdraw service routes. Cloudflare characterized them as harmless information on the ordinary routers shown in its report. The consequential difference appeared on the MCP spine routers. A configuration diff reordered terms in an aggregate export policy. A term associated with disabled prefixes moved before terms that advertised site-local routes for services and internal functions. Because policy terms are evaluated sequentially, the earlier match stopped the later advertisement logic from doing its work. [1]

This is the difference between semantic intent and effective behavior. The author of a change may intend to attach metadata. A reviewer may see community additions. A ticket may use a title that says "standardize communities." None of those representations controls the router. The router evaluates the resulting policy in order. If the generated or rendered configuration changes that order, the operating effect is whatever the policy engine executes.

BGP itself supplies the reachability mechanism. Networks announce prefixes to tell peers that addresses are reachable through a path. They withdraw routes when that reachability is no longer available or should no longer be advertised. A withdrawn prefix may disappear from selected paths, making the associated addresses unreachable through that route. The BGP-4 specification defines these announcement and withdrawal semantics; it does not know an operator's business intent for a particular policy term. [10]

That distinction creates an evidence requirement. Change approval should bind to the exact candidate configuration that the router will evaluate, not only to a higher-level request or generated template. The review record should show:

The exact rendered policy before and after the change.
The evaluation order of every affected term.
The set of routes expected to be accepted, rejected, advertised, and withdrawn.
The source of each prefix in that set.
The device roles and architectures on which the policy will run.
A simulation or lab result using representative route inputs.
The management and rollback paths that remain reachable if the policy output is wrong.

Without those bindings, review can verify the request while missing the compiled effect. Infrastructure accountability follows the configuration consumed by the running system.

Site-local prefixes were part of the service path

The term "site-local" can sound secondary, as if the affected routes were housekeeping entries with no customer consequence. Cloudflare's report shows otherwise. The prefixes enabled communication among its machines and allowed servers to reach customer origins. Their withdrawal interrupted those paths. [1]

Cloudflare also linked the prefixes to Multimog, an internal load-balancing system used in MCP locations. Multimog could no longer forward requests among servers in the MCPs because the relevant reachability disappeared. Compute clusters of different sizes then received similar traffic loads. Smaller clusters overloaded. [1]

This mechanism is important because it explains why a subset of route withdrawals could have a much larger service effect than a simple count of missing prefixes suggests. A prefix can support a dependency shared by many services. If it carries origin reachability, internal cluster communication, or a load-balancing path, its withdrawal changes how work is distributed across the system. A service can still receive traffic at an edge while losing the internal routes needed to process or forward that traffic safely.

The event therefore tests how an operator classifies route criticality. A registry of prefixes should not stop at ownership or origin authorization. It should record operational role:

Is the prefix Internet-facing, management-only, site-local, customer-origin-facing, or inter-cluster?
Which routers and policies advertise it?
Which services depend on it?
Which monitoring checks prove it is reachable?
Which locations share the same policy template?
What happens if the route disappears but incoming traffic continues?
Does management access use the same route family or policy?
Which alternative path is expected to carry the dependency?

This record is not sovereign over the network. It does not make a route exist. It creates a testable ledger of expected state. Operators can compare the ledger with the route information base, forwarding information base, external collectors, active probes, and service-level results. A record that is accurate, unique, change-controlled, and tied to observed behavior makes a withdrawal detectable before customers become the monitoring system.

The Multimog effect also shows why capacity needs topology context. Cloudflare said affected MCP locations contained clusters of different sizes. When internal forwarding failed, smaller clusters received traffic comparable to larger clusters. A capacity dashboard that showed total site capacity could therefore overstate usable capacity. The route failure removed the balancing mechanism that made aggregate capacity accessible.

An evidence-based review would ask whether each MCP had a tested degraded mode. Could traffic be drained externally when Multimog lost site-local reachability? Could smaller clusters protect themselves with admission controls? Did the route-policy canary include probes for origin access, inter-cluster forwarding, and management access rather than only BGP session health? The public report identifies the dependency, but it does not publish every protective control.

Nineteen locations exposed a concentration inside a distributed network

Cloudflare said the 19 MCP locations were about four percent of its network yet the outage affected roughly 50 percent of total requests. It also emphasized that impact varied by location: some users could not reach properties using Cloudflare while other locations continued to operate normally. [1]

Those numbers should remain attributed and should not be converted into a count of people, customers, websites, or financial losses. They do support a topology lesson. A network can be geographically distributed while traffic remains concentrated in a small set of high-capacity locations. Counting locations is not the same as measuring the share of traffic, customer dependencies, peering capacity, or origin connectivity carried by them.

The MCP program was intended to improve resilience in the busiest locations. Cloudflare described a Clos-style architecture with an additional routing layer and mesh of spine connections. The architecture allowed parts of an internal network to be enabled or disabled for maintenance or fault handling. Those are legitimate resilience goals. The incident does not prove the architecture was inherently unsound. [1]

It does show that a common policy path can connect locations that are physically separate. If the same rendered route policy reaches every MCP spine in one final rollout step, the policy template becomes a shared failure domain. Physical diversity does not remove that common control dependency.

This is a recurring network-accountability pattern. Operators may have multiple routers, facilities, carriers, and paths, but a single automation template, credential system, route policy, controller, or rollout stage can still create common-mode failure. Diversity must be measured at the control surface that caused the incident.

For MCP deployment, a useful change map would include:

Dimension	Evidence needed
Physical location	Facility and city identifiers for each canary and rollout group
Architecture	Older PoP versus MCP, spine role, software train, hardware family
Policy	Exact rendered policy and term order for each role
Route role	Required site-local, customer-origin, management, and service prefixes
Traffic share	Normal request and bandwidth share by rollout group
Management	Primary and independent access paths
Rollback	Commit-confirm state, timer, owner, and per-device result
Observation	Route, reachability, load-balancing, capacity, and customer checks

The table prevents a low-risk first stage from being mistaken for a representative one. A canary should be small, but it must exercise the feature and architecture being changed. If the only representative site carries too much traffic to use as a canary, that is itself an architectural and operational risk requiring a lab twin, shadow evaluation, a smaller representative domain, or stronger static invariants before production.

Losing management reachability changed the recovery problem

Cloudflare said the route withdrawal made it harder for engineers to reach affected locations and revert the configuration. Backup procedures were used to take control. [1]

This detail belongs at the center of the analysis. A network change is not safely reversible merely because a rollback command exists. The operator must still be able to authenticate, reach the device, determine the correct state, and coordinate the rollback while the network is degraded. If management traffic depends on the routes being changed, the recovery path shares the failure domain.

Independent access does not necessarily require an entirely separate global network for every device. It does require a reasoned design that survives the failure being tested. Options can include console servers on a separately routed management network, out-of-band carrier links, local automation that can execute a timed rollback, device commit-confirm features, protected terminal access, or a control channel whose reachability does not depend on the candidate policy.

Each option has limits. An out-of-band carrier can share a facility or power source. A console server can depend on the same identity provider or DNS service. A commit-confirm timer can be canceled too early or fail to restore all dependent state. Local automation can apply the wrong baseline. Accountability requires exercises that prove the selected path works under the exact loss condition.

The relevant test for this incident is straightforward: withdraw or suppress the route set that provides ordinary site-local and management reachability in a representative MCP environment, then verify that operators can still inspect state and restore the approved configuration. The exercise should measure time to access, time to identify the failed invariant, time to start rollback, time to restore required advertisements, and time to confirm service recovery.

A screenshot of a successful login on a normal day is not enough. The test must remove the dependency that failed. Running-system evidence outranks a diagram that labels one path "out of band."

The public report does not identify every backup procedure Cloudflare used, which is reasonable for security and operational reasons. Sensitive topology and credentials should not be published. An organization can still retain auditable evidence under access controls: exercise timestamps, device scopes, independent attestations, route probes, success criteria, and exceptions.

Rollback authority failed as a coordination control

Cloudflare's timeline says the final reverts were delayed because network engineers walked over one another's changes. Some reverts undid previous reverts, causing the problem to reappear sporadically. [1]

This is not merely a human-error footnote. It identifies a missing serialization property in incident response. During a high-impact outage, multiple engineers may have valid access and strong reasons to act. If the change system does not provide one authoritative desired state, visible ownership, per-device locking, and a shared progress record, parallel work can produce oscillation.

The production lesson is not that only one person should ever respond. Investigation, route observation, customer communication, service testing, and rollback preparation can proceed in parallel. Mutation of the same routing state needs coordination. The system should make it difficult for one responder to reintroduce the configuration another responder has removed.

Useful controls include:

A declared incident commander and network-change owner.
A frozen desired-state commit for the recovery.
Device or policy-domain locks during rollback.
A single orchestrator that records each attempted and completed revert.
Commit-confirm timers that automatically restore the prior state if management confirmation is lost.
Read-only observers who verify route and service recovery without changing configuration.
A rule that emergency manual changes are reconciled into the authoritative configuration before normal automation resumes.
Explicit handoff when ownership moves between teams or regions.

These controls should produce a per-location ledger. For each MCP, the record should show the bad configuration identifier, rollback command or commit, executor, start and end time, device acknowledgment, route-table result, management reachability, service probe, and any later change that touched the same policy. The ledger makes interference visible and supports a reliable declaration that the last site is restored.

Cloudflare's remediation included automated commit-confirm rollback and better stagger enforcement. [1] A commit-confirm mechanism is especially relevant when a route change can remove management access. The router can automatically restore the prior state unless the operator confirms the new state after reachability and invariant checks pass.

Commit-confirm is not magic. The timeout must be long enough for validation but short enough to limit impact. The prior state must itself be safe. Confirmation must not be automated solely from a weak signal. Multi-device changes need coordinated semantics so one router does not revert while another remains on the candidate. The mechanism still needs exercises and evidence.

The process existed; its proof obligations were incomplete

Cloudflare's report explicitly says the change had a ticket, dry-run, stepped rollout, and multiple peer reviewers. [1] That makes the incident useful for organizations that already have mature change-management vocabulary.

A weak postmortem response would add another approval. More signatures can slow delivery without testing the missing condition. The evidence points to four stronger questions.

First, what input did the dry-run use? If it rendered policy only for an older architecture, it could not reveal the MCP term-order effect. A dry-run should include every policy role affected by generation logic, especially roles with different templates or inheritance.

Second, what assertions did the dry-run evaluate? A syntax-valid configuration can still withdraw required prefixes. Validation should compare route outputs, not only parse results. The tool should fail if any required site-local, management, origin-facing, or service prefix changes unexpectedly.

Third, how were rollout groups defined? Geographic ordering alone would not expose an architecture-specific fault if every early geography used the older design. Groups should reflect hardware, software, topology, policy role, control-plane version, traffic share, and management path.

Fourth, what stopped progression? A staged rollout is effective only if it has explicit observation windows and automatic gates. The system needs evidence that the canary retained required routes, origin reachability, internal load balancing, management access, and service success before the next group begins.

Peer review has similar obligations. Reviewers need the rendered diff for all roles, expected route outputs, canary matrix, rollback plan, and evidence that the recovery path is independent. Asking reviewers to infer these from a high-level ticket invites the same gap.

The goal is not bureaucratic perfection. It is to bind process to the failure mechanism. For BGP policy, the operating claim is that specified prefixes remain announced to specified peers while intended attributes change. That claim can be tested mechanically.

Route invariants turn intent into a test

A route invariant is a statement that must remain true before, during, and after a change. Examples include:

Every MCP spine advertises the approved set of site-local prefixes.
Management prefixes remain reachable from designated independent access points.
Customer-origin connectivity probes succeed from each compute cluster.
Internal load balancing can move requests between clusters of different sizes.
No unexpected public prefix is announced.
No protected prefix is withdrawn.
The number and attributes of changed routes match the approved change scope.

The required-prefix set should come from an accurate operational registry linked to service ownership. It should not be an informal list copied into a one-time ticket. Prefix identity, role, origin, expected peers, security metadata, and change history need durable ownership.

Before commit, policy simulation can evaluate the candidate against representative routes and peer contexts. After a canary commit, the system can compare the router's route information base and advertised-route view with the invariant. Active probes can test service paths. External collectors can supply a separate view for public advertisements. [17][18][19]

The comparison must distinguish expected changes from unexpected ones. If the purpose is to add communities, route presence should remain stable while attributes change within the approved set. A withdrawal of required site-local prefixes is then an immediate gate failure, not a symptom waiting for global HTTP errors.

Route invariants also improve incident communication. Instead of saying "the network is recovering," operators can report that required advertisements are restored in a defined number of locations, management reachability has returned, internal forwarding passes, and customer request success is within a stated range. Each claim has evidence and a boundary.

There is a risk of false confidence. The invariant set can be incomplete. A route can be present but forward incorrectly. A control-plane view can differ from the data plane. That is why route checks should be paired with forwarding and service probes. The test suite should be updated when an incident reveals a dependency that was not represented.

BGP communities did not cause the outage by themselves

It would be inaccurate to summarize the incident as "BGP communities broke Cloudflare." Communities are metadata attached to routes. Operators use them for policy, tagging, traffic engineering, and operational signaling. RFC 1997 and RFC 8092 define community formats; they do not dictate Cloudflare's policy order. [11][12]

The intended community additions were part of the change context. The outage mechanism was the effective order of policy terms on MCP spines and the resulting withdrawal of required prefixes. [1] The distinction matters because remediation should address the failed control, not stigmatize a standard mechanism.

Communities can improve accountability when their meaning is documented, unique, and consistently applied. They can identify route classes, intended handling, maintenance state, geography, or customer relationship. But labels are only useful if policy evaluation preserves the required result. A community saying "site local" does not keep a route advertised when an earlier term rejects it.

The same principle applies to change names and comments. Human-readable labels support review, but running policy determines reachability. An operator should be able to trace from a community definition to the route set, policy branches, expected advertisements, and observed results. If the trace ends at documentation, the operational evidence is incomplete.

RPKI is important and not the direct repair

RPKI provides a way to bind IP address resources to authorized origin ASes. Route Origin Validation allows a router to classify an announcement based on whether the origin and prefix length are consistent with a Route Origin Authorization. RFC 6480 describes the architecture, and RFC 6811 describes origin validation. [15][16]

Those controls address a different question from the June 2022 failure. RPKI can help determine whether Cloudflare's AS was authorized to originate a public prefix. It does not determine whether an internal export-policy term should advertise a site-local prefix, whether the terms are ordered correctly, whether an MCP canary is representative, or whether rollback access is independent.

An authorized route can be accidentally withdrawn. A valid origin does not prove availability. Conversely, an internal site-local route may never be visible in public RPKI or external collectors. Presenting RPKI as a universal fix would obscure the actual control failure.

RPKI still belongs in the evidence model because network-resource authorization, route policy, and operational continuity interact. An operator should maintain accurate number-resource records and security metadata while separately validating policy behavior and service reachability. Registry integrity and running-code evidence are complementary, not substitutes.

This boundary is also useful for public accountability. A postmortem can state whether an incident involved origin authorization, a route leak, a hijack, an internal policy withdrawal, or another routing mechanism. Precise classification avoids treating every BGP-related event as the same failure and supports the correct remediation.

External route observation is valuable but incomplete

RIPE NCC's Routing Information Service collects BGP data from peers, and RIS Live exposes update streams. CAIDA's BGPStream provides tools for working with routing data from multiple collectors. These systems can help researchers and operators observe announcements, withdrawals, path changes, and restoration visible to participating vantage points. [17][18][19]

For a public-prefix incident, external observations can answer important questions:

When did a prefix disappear from selected collectors?
Which peers or regions observed the withdrawal?
When did advertisements return?
Did paths or attributes change after restoration?
Did the event match the operator's public timeline?

The June 2022 postmortem concerns site-local prefixes and internal MCP behavior as well as externally experienced service failure. Public collectors may not see every relevant route. They do not expose private policy terms, router candidate configurations, internal management reachability, or Multimog state. Absence of an external signal does not prove the private system was healthy.

This limitation should not be used to dismiss external evidence. It defines the reconciliation task. Operators can retain router logs, configuration commits, advertised-route snapshots, active-probe results, and service telemetry. Public collectors provide an independent layer. A credible closeout explains where the views agree, where visibility differs, and why.

The evidence should be timestamped with a consistent clock. Routing updates, configuration commits, incident declarations, reverts, reachability probes, and customer errors can otherwise appear out of order. Time synchronization and immutable logs are accountability controls because they preserve the sequence needed to evaluate response.

Impact figures need their own boundaries

Cloudflare said the 19 MCP locations represented approximately four percent of its network but the outage affected 50 percent of total requests. It also published a request-volume graph and egress-bandwidth view. [1]

The figures show concentration of traffic in high-capacity sites. They do not establish that half of all Internet users, half of Cloudflare's customers, or half of all websites were fully unavailable. Request traffic is not a population count. Some locations continued to operate normally, and customer effects depended on geography and service path.

A responsible impact account should separate:

Requests that failed.
Requests that were delayed or retried.
Locations that lost required routes.
Clusters that overloaded.
Customer origins that became unreachable from affected sites.
Services that continued from unaffected locations.
Time to first recovery and time to final revert.
Residual errors after route restoration.

The public report gives strong information about several of these dimensions but not a complete customer-by-customer inventory. The article should not manufacture precision.

For future incidents, an operator can improve disclosure by publishing denominators and uncertainty. If 50 percent of requests were affected, define the measurement interval, success criterion, retry treatment, geography, and whether the figure includes traffic shifted to other sites. Clear metrics let customers compare the statement with their own logs.

Accountability follows control, not brand exposure

Cloudflare controlled the route policy, rollout procedure, MCP grouping, dry-run scope, peer-review materials, management access, automation, and rollback coordination. Its postmortem accepts that the outage was its error and not an attack. [1] Those facts make Cloudflare the central operator in the accountability analysis.

That does not mean every consequence is legally attributable or that every engineer shared equal responsibility. The public record does not establish individual fault or negligence. Governance should map controls to teams and systems:

The network architecture owner defined MCP roles and failure domains.
The policy owner defined export terms and community handling.
The automation owner rendered and distributed configuration.
The change owner selected stages and observation windows.
Reviewers assessed the supplied evidence.
The incident commander coordinated recovery.
Device and platform teams supplied rollback and commit-confirm mechanisms.
Service teams monitored origin reachability, Multimog, cluster load, and customer requests.
Leadership set acceptable traffic concentration and change risk.

Router and software vendors influence available safety mechanisms, but Cloudflare's public report does not attribute the term reordering to a vendor defect. Standards bodies define protocol behavior but do not operate Cloudflare's policy. Customers relied on the service but did not control the internal site-local route configuration.

Customers still have responsibility for their dependency architecture. An organization whose public service depends on one CDN or authoritative DNS path should understand that dependency, test alternate access where feasible, and define communication when the provider is unreachable. That responsibility does not transfer control of Cloudflare's routers to the customer or make every dependency avoidable.

Regulators and large buyers can request evidence without prescribing private topology. They can ask whether the provider uses architecture-representative canaries, route invariants, independent management access, serialized rollback, and exercises. Contracts can define disclosure and recovery evidence. They should avoid demanding sensitive configurations in public when protected review can serve the purpose.

A practical evidence pack

The strongest post-incident closeout is not an assurance that the same exact error cannot recur. It is a bounded pack showing what changed and how the new controls perform.

Control	Retained evidence	Operating test	Important limit
Change authorization	Ticket, approvers, scope, risk classification	Ticket binds to exact rendered candidate configuration	Approval does not prove route behavior
Policy review	Before/after term order for every router role	Review tool identifies changed matches and actions	Reviewer can miss an incomplete route model
Required-prefix invariant	Versioned prefix list with role and owner	Candidate and canary retain every protected advertisement	Present route does not prove forwarding
Architecture canary	MCP role, hardware, software, traffic and management-path record	Canary exercises the same policy path as later groups	One canary may not represent every MCP
Route simulation	Representative input routes and expected outputs	No unexpected withdrawal or advertisement	Simulation can differ from device behavior
Commit-confirm	Timer, prior state, confirmation criteria	Loss of management or invariant failure triggers rollback	Multi-device consistency remains difficult
Independent access	Topology, dependency and exercise record	Operators reach and restore devices after ordinary route loss	Hidden shared power, identity or DNS can remain
Rollback ownership	Incident owner, lock state, per-device ledger	Parallel investigators cannot overwrite the recovery state	Emergency manual work may bypass automation
External observation	RIPE RIS, BGPStream or other collector evidence	Public route changes align with operator timeline	Private routes and all peers are not visible
Service verification	Origin, internal forwarding, cluster load and request probes	Route restoration produces completed service	Synthetic probes may miss customer-specific paths
Remediation durability	Deployment evidence and recurring exercise results	MCP-specific staging and rollback continue to pass	A one-time test does not prove permanent compliance

The pack separates records from operating proof. A prefix registry is useful because it preserves uniqueness, ownership, role, expected peers, security metadata, and change history. It is not the authority that makes packets flow. Policy simulation is useful because it predicts behavior; the canary and live route state still need observation. A public postmortem is useful because it describes the event; it does not prove every later commitment remains active.

Sensitive details can be protected. Exact management addresses, credentials, topology, and router configuration may create security risk if published. Independent auditors, regulators, or customers under confidentiality can review them. Public disclosure can summarize control outcomes, timestamps, scope, and unresolved limits.

The comparison boundaries matter

Cloudflare has experienced other routing and configuration incidents. Treating them as one generic outage would weaken the analysis.

The June 2019 event involved an external route leak. That is not the 2022 MCP staging failure.

The July 2020 Cloudflare outage was a router-rule outage. Its direct mechanism differs from the 2022 site-local prefix withdrawal.

The March 2025 event involved a faulty ROA. The 2022 routes were withdrawn by internal policy; the public record does not describe a faulty ROA as the cause.

Cloudflare's 2025 feature-file failure was not a BGP export-policy event.

The cases share one abstract lesson: small control inputs can acquire global reach. The specific evidence and repairs differ. Network accountability requires naming the protocol, layer, authority, propagation path, and recovery boundary that actually failed.

What the public record does not prove

Cloudflare's postmortem is detailed, but it is a first-party account. It provides the strongest public source for the mechanism and timeline, while leaving important material private.

The public record does not disclose every withdrawn prefix, complete router configurations, the full change ticket, reviewer comments, automation code, private topology, management-access design, or all service dependencies. It does not show the complete route information base or forwarding state for every device.

The public traffic graphs do not identify every affected customer, user, request type, origin, or financial consequence. The article does not infer service credits, lost revenue, regulatory penalties, contractual breach, or customer negligence.

The postmortem says the incident was Cloudflare's error and not an attack. It does not establish malicious intent, concealment, criminal conduct, or individual blame. It does not by itself establish a legal standard of care.

The remediation section records planned and immediate work. It does not prove that every MCP-specific canary, policy redesign, stagger automation, or commit-confirm mechanism remains deployed and effective today. Current documentation can explain architecture and controls, but it should not be retroactively treated as proof of the exact 2022 state.

RIPE RIS and BGPStream can provide independent routing observations, but the available public evidence does not provide a complete collector reconstruction of every site-local withdrawal. External visibility has limits.

These unknowns do not prevent accountability analysis. They define the questions that evidence holders should answer and prevent the article from turning a technical postmortem into unsupported accusation.

Questions for operators, boards, customers, and reviewers

Network operators should ask:

Which route policies differ by architecture or router role?
Does the dry-run render every affected role?
Which prefixes must remain advertised through the change?
Are the required-prefix lists versioned, owned, and linked to service dependencies?
Does a policy simulator evaluate representative route inputs and term order?
Is the first canary small and architecture-representative?
Which observation window and automatic gate stop progression?
Can operators reach devices after the candidate policy removes ordinary management routes?
Does commit-confirm restore a coherent multi-device state?
Who owns mutation during incident rollback?
Can read-only teams observe routes and service without changing configuration?

Boards and risk committees should ask:

What share of traffic depends on the busiest common control domain?
Does geographic distribution hide a shared policy or automation dependency?
Which management, identity, DNS, power, and carrier dependencies are shared by primary and recovery paths?
What evidence demonstrates recurring exercises rather than one-time remediation?
How are customer-impact metrics defined and reconciled with route and service evidence?

Customers should ask:

Which services depend on Cloudflare reachability, DNS, CDN, security, access, or origin paths?
Can critical communication continue if the provider and its status path are impaired?
Are alternate providers or direct-origin paths technically and operationally tested?
What security trade-offs arise from bypass or failover?
Which logs establish the customer's own impact without assuming a universal outage?

Reviewers and auditors should ask:

Did they inspect the exact rendered policy, not only the request?
Did the canary match MCP topology and spine policy?
Were site-local and management routes included in invariants?
Did independent measurements and router logs agree?
Could two engineers overwrite each other's reverts?
Is each remediation claim tied to a current test result and exception record?

These questions do not assume that zero failure is possible. They test whether a known class of routing error is bounded, observable, and recoverable.

Conclusion: a stage is only as good as the system it represents

Cloudflare's June 2022 outage began with a controlled change. The change had a ticket, dry-run, peer review, and multiple rollout stages. It still withdrew critical site-local prefixes when it reached an architecture none of the early stages had exercised. The withdrawal broke server and origin reachability, disabled internal load redistribution, overloaded smaller clusters, and impaired the management path needed for recovery. Concurrent reverts then caused sporadic recurrence. [1]

The incident made BGP change staging an accountability test. A stage is not evidence merely because it comes before another stage. It must represent the relevant topology, policy role, software, route inputs, traffic concentration, and management dependency. Its success criteria must include the routes and services that must remain available.

The repair model is concrete. Bind approval to the rendered configuration. Maintain an accurate ledger of required prefixes and their operational roles. Simulate policy output. Use an MCP-specific canary. Observe announcements, forwarding, origin connectivity, internal load balancing, and management access. Protect rollback with commit-confirm and one authoritative change owner. Reconcile private router records with independent public measurements where visibility permits.

BGP communities, RPKI, route collectors, tickets, and diagrams all contribute evidence. None is a substitute for the operating result. Communities label routes; policy order controls decisions. RPKI validates origin authorization; it does not prove availability or internal policy correctness. Collectors observe selected external views; they do not reveal every private dependency. Tickets record intent; they do not keep a prefix announced.

That reality layer is the lasting lesson. A distributed network is resilient when its running paths, control policies, management access, and rollback mechanisms preserve service under a representative failure. The proof is not the name of the process. It is the route that remained reachable, the canary that stopped the rollout, and the recovery that one team could complete without another team undoing it.

Sources

Cloudflare, "Cloudflare outage on June 21, 2022": https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/
Cloudflare, 2022 Impact Report: https://cf-assets.www.cloudflare.com/slt3lc6tev37/fBTOgkechN3IcaoT3kbA3/c9b74ee483d28c795d3c7891d8d36034/2022_Cloudflare_Impact_Report.pdf
Cloudflare, "Cloudflare Backbone: A Fast Lane on the Busy Internet Highway": https://blog.cloudflare.com/cloudflare-backbone-internet-fast-lane/
Cloudflare, "The backbone behind Cloudflare's Connectivity Cloud": https://blog.cloudflare.com/backbone2024/
Cloudflare, "Load Balancing without Load Balancers": https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/
Cloudflare, CDN Reference Architecture: https://developers.cloudflare.com/reference-architecture/architectures/cdn/
Cloudflare, IP addresses and anycast network: https://developers.cloudflare.com/fundamentals/concepts/cloudflare-ip-addresses/
Cloudflare, Peering Policy: https://www.cloudflare.com/peering-policy/
PeeringDB, Cloudflare network record: https://www.peeringdb.com/net/4224
IETF / RFC Editor, RFC 4271, Border Gateway Protocol 4: https://www.rfc-editor.org/rfc/rfc4271
IETF / RFC Editor, RFC 1997, BGP Communities Attribute: https://www.rfc-editor.org/rfc/rfc1997
IETF / RFC Editor, RFC 8092, BGP Large Communities Attribute: https://www.rfc-editor.org/rfc/rfc8092
IETF / RFC Editor, RFC 8326, Graceful BGP Session Shutdown: https://www.rfc-editor.org/rfc/rfc8326
IETF / RFC Editor, RFC 7454, BGP Operations and Security: https://www.rfc-editor.org/rfc/rfc7454
IETF / RFC Editor, RFC 6811, BGP Prefix Origin Validation: https://www.rfc-editor.org/rfc/rfc6811
IETF / RFC Editor, RFC 6480, RPKI Architecture: https://www.rfc-editor.org/rfc/rfc6480
RIPE NCC, RIS Live Manual: https://ris-live.ripe.net/manual/
RIPE NCC, Routing Information Service: https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris/
CAIDA, BGPStream: https://bgpstream.caida.org/
Cloudflare, "What is BGP?": https://www.cloudflare.com/learning/security/glossary/what-is-bgp/

Cloudflare's 2022 Outage Made BGP Change Staging an Accountability Test