Summary

  • A maintenance command meant to measure global backbone capacity took down every backbone connection instead, and the audit tool built to stop commands like it did not stop it, because of a bug in the audit tool itself.
  • The authoritative DNS servers then behaved exactly as designed — withdrawing their BGP advertisements when they could no longer reach the data centres — which converted an internal connectivity fault into a total loss of external reachability.
  • Recovery ran through locked doors rather than a remote console, because primary and out-of-band access were both gone.

The command had one job: find out how much capacity Facebook's global backbone had to spare. It removed the backbone instead. The company's own account is unusually blunt about both halves of that sentence. During a routine maintenance job, "a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally" (Engineering at Meta).

Read the next sentence twice, because it is the one that carries the accountability: "Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command."

The command was not smuggled in. Nobody is described as having routed around a safeguard. It went through the process, and the process contained a component whose entire purpose was to say no, and that component had a defect, and the no never arrived. Facebook had a control. It did not have the control it believed it had. The public record reviewed for this article does not include the command's text, its scope, or who signed it off, so the gap between what the tool was asked to judge and what it actually judged cannot be measured from outside. What can be measured is the outcome.

The accounting for that outcome gets worse quickly: "This change caused a complete disconnection of our server connections between our data centers and the internet. And that total loss of connection caused a second issue that made things worse."

That second issue was a safety mechanism doing precisely what it was built to do. "To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection." When the backbone vanished, there was no location left that could speak to the data centres, so "the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP advertisements." The summary sentence is nine words long and describes the whole incident: "The end result was that our DNS servers became unreachable even though they were still operational."

Operational and unreachable. The withdrawal was not a misconfiguration, a mistake, or an unforeseen side effect. It ran. The rule it ran on encodes an assumption — that a DNS server which cannot reach the data centres is looking at a local fault, an unhealthy connection in one place — and on 4 October every location was looking at the same catastrophic fact. Each one made a locally correct judgement. The sum of those correct judgements was that Facebook disappeared from the internet's routing table.

Then the recovery had to happen the hard way. "Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems. But this took time, because these facilities are designed with high levels of physical and system security in mind. They're hard to get into, and once you're inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them."

That paragraph describes two good designs pulling in opposite directions. Physical and system security at a data centre exists to stop a stranger from touching a router; on 4 October it also slowed down the people who needed to touch a router. The out-of-band channel — the classic answer to losing control-plane access — shared the fate of the system it was supposed to rescue. A recovery path that depends on the network it recovers is not a recovery path. It is a hope with a runbook attached.

Outside observers could see the shape of all this but not the cause. Cloudflare measured the disappearance and the return: on its 1.1.1.1 resolver, the DNS name facebook.com "stopped being available at around 15:50 UTC and returned at 21:20 UTC", with renewed BGP activity from Facebook's network appearing around 21:00 UTC and peaking at 21:17 UTC (Cloudflare). Meta's account does not state a duration; the roughly five-and-a-half-hour window is Cloudflare's measurement, not the operator's figure. That distinction matters more than it looks. It is the difference between what can be independently established — that the routes went away and came back — and what cannot: whether the check that failed had ever been tested.

Here is the part worth carrying into any organisation that runs change through software. An audit tool is a control, and a control is only as good as the evidence that it works. That evidence is not a design document. It is a test that feeds the tool a command of the kind it exists to block and shows that the command was blocked; a deliberate exercise proving the tool fails closed rather than silently passing when it cannot evaluate something; and a signal loud enough that a tool which stops functioning is itself an incident. None of those three is visible in the material reviewed here.

Facebook's engineers have now told the world, in their own words, that their verification layer carried a defect. That is more candour than most operators manage. It is also the whole problem: the statement came after the outage, from the organisation that owned both the command and the check. An audit tool's internals are invisible to everyone outside the company, which means the assurance it provides is, in practice, a promise. Promises are fine. Verified promises are better.

What would move this case from an unverified control to a repaired one is a record — dated, specific, and checkable by someone who is not the operator — showing that the audit tool's own tests were rewritten after the incident, that the failing path now fails closed, and that the change was exercised rather than merely announced. The material reviewed for this article consists of Meta's 5 October 2021 engineering post and Cloudflare's independent measurement of the disappearance; neither contains the command, the defect report, or a remediation plan with an implementation date.

So the article you have just read is a description of what durability would require, not a report that it has been achieved.