Summary

  • Fastly service versions make a previous configuration easy to reactivate, but the API-visible active version is control-plane state rather than request-level proof that every serving population has converged.
  • Rollback acceptance should record the last response governed by the withdrawn version at each observed POP, the restored version, the exact harmful test and any separate purge or cache-generation state.

The central screen says the rollback succeeded. The previous service version is active again. Error rates fall. Yet a request reaching one distant point of presence still follows the rule that operations meant to withdraw. That opening is a hypothetical acceptance test, not a reported Fastly incident. Its purpose is to distinguish a command from an outcome.

Fastly's configuration model gives operators a valuable control. A service version is a defined configuration instance that can be cloned, locked, activated or deactivated. Only one version is active at a time, and an older inactive version can be reactivated. This creates a clear object to restore. It also creates a tempting completion timestamp: the moment the activation request returns and the dashboard shows the intended number.

Fastly's own introductory tutorial says configuration changes can take a couple of minutes to propagate. That statement is not evidence of a defect; distributed systems need time to distribute state. It does mean that central acknowledgement and global request behaviour are different observations. An operator that defines rollback at the first has not yet measured the second.

A POP is an execution population, not a dot on a map

Fastly describes a point of presence as a cluster of cache servers. With clustering and shielding, a request may encounter more than one site or POP on the way to origin. Fastly exposes the serving location through server.datacenter in VCL and FASTLY_POP in Compute. It also exposes a service-version environment value to edge code. Those identifiers are the basis for a request-level rollback ledger: which version made this decision, in which population, at what time and for which exact test.

A generic health endpoint is insufficient. The canary must exercise the withdrawn behaviour: the path rewrite, access rule, origin choice, header policy, cache key, Compute branch or security decision that caused the rollback. A green response from unrelated code proves only that the network can answer.

The first useful timestamp is the central activation acknowledgement. The decisive timestamp is the last observed response governed by the withdrawn version or rule. Between them lies the rollback convergence interval. Measure its maximum across observed POPs, not its average. A rare stale decision can matter more than a large population of correct low-risk requests.

Logs provide evidence, not omniscience

Fastly supports near-real-time streaming logs and lets customers record the data centre that served a request. That is strong positive evidence: a log can show that a given POP executed a version or returned an outcome. Fastly also states that real-time log delivery is best effort and not guaranteed; records may be dropped when a distributed system prioritises performance and stability.

The implication is subtle. Seeing a withdrawn version after the deadline proves rollback failure. Not seeing it in a best-effort stream does not, by itself, prove global absence. A credible acceptance record combines active probes, request-level version and POP markers, platform telemetry or support evidence, and an explicit list of unobserved populations. Silence must be labelled as a coverage gap, not converted into certainty.

Anycast makes this difficult. A customer cannot necessarily direct a probe through every POP because routing selects a network-near location. The answer is not to pretend full coverage. Use distributed probes, real user traffic, regional test clients and Fastly evidence; report the population denominator that was actually observed. If a high-risk service cannot obtain sufficient coverage, the decision owner must either extend the verification window, constrain traffic, or accept the residual uncertainty.

Configuration and cache are different rollback surfaces

Restoring a service version does not answer every cache question. Fastly's purge documentation describes several mechanisms and clocks. Purge-all can take up to two minutes. URL and surrogate-key purges propagate through a gossip-style mechanism in roughly 150 milliseconds. A cache-key purge issued by edge code is local to the POP unless global scope is selected. A soft purge deliberately keeps content available as stale in some circumstances.

Fastly also documents a race in which a shield can return pre-purge content to an edge, allowing old content to remain cached. For purge-all, it exposes a cache-generation value through VCL and Compute and describes comparing shield and edge generations. This is exactly why rollback evidence cannot be a single version number. The configuration may be restored while cached content, cache keys, dynamic data or origin state remains independently wrong.

The operator must decide whether the harmful outcome lives in executable configuration, cached objects, both, or somewhere outside Fastly. If a purge is required, the recovery plan also has to protect the origin from the refill surge. A technically complete purge that overloads the origin is not business recovery.

What the 2021 outage contributes

Fastly says its global disruption on 8 June 2021 resulted from an undiscovered software bug triggered by a valid customer configuration change. Its public timeline records onset at 09:47 UTC, identification of the triggering configuration at 10:27, recovery beginning at 10:36, the majority of services recovered by 11:00, mitigation at 12:35 and deployment of a permanent fix beginning at 17:25.

The incident shows that a valid configuration can interact with platform software at global scale and that recovery has stages. It does not state that a specific POP continued to run a withdrawn rule, and it should not be used to invent such a claim. The lesson for procurement is narrower: rollback is a capability whose remediation time and observable completion deserve their own acceptance evidence.

A rollback evidence ledger

For every high-impact change, record the service ID, withdrawn version and intended restored version; the central request and response; request-observed version by POP; POP, region, shield path and trace identity; the exact harmful canary; cache generation or purge identity; first restored and last withdrawn responses; population coverage and log gaps; and the person authorised to declare the incident over.

Set the objective before deployment. For example: no response governed by the withdrawn security rule after the allowed convergence window, across the agreed set of observed populations, with zero cache-generation mismatch and no unexplained gaps in the high-risk regions. The numbers must reflect the business consequence rather than a universal promise this article cannot derive from public documentation.

The board-level conclusion is not that Fastly should eliminate propagation time. It is that buyers should not turn an API response into semantic proof. Rollback is complete when the old decision is no longer observable within the agreed coverage and the remaining unknowns are owned—not when the button changes colour.

Sources