Summary

  • Cloudbleed let an HTML transformation read beyond its buffer, placing fragments from unrelated requests into responses and collapsing the expected boundary between customers at a shared edge.
  • Cloudflare could stop new disclosure with feature kill switches and a parser fix, but cached fragments and exposed credentials demanded a separate recovery process whose completeness could never be inferred from the patch alone.

The trigger and the victim were not the same site

On 17 February 2017, Google's Project Zero reported that pages served through Cloudflare sometimes contained fragments that did not belong there. The material included cookies, authentication tokens, POST bodies and other data drawn from process memory. Search engines had cached some of the contaminated responses.

The popular name, Cloudbleed, invited comparison with Heartbleed. The more important distinction was architectural. Heartbleed asked whether one OpenSSL process had returned more heartbeat payload than a requester supplied. Cloudbleed occurred inside a shared edge service that transformed many customers' pages. A malformed page could activate the defect, yet the adjacent bytes copied into its response might have come from an entirely different customer's request.

That makes a list of trigger domains a poor victim ledger. The site whose HTML reached the faulty path and the site whose data occupied nearby memory played different roles. A customer did not need to run malformed HTML or enable the implicated feature for its data to be exposed; its request only had to share the wrong process at the wrong moment.

The security boundary was therefore not merely “the parser must parse this page correctly.” It was “work performed for one tenant must never read or emit another request's memory.” Once that invariant failed, the edge's efficiency—shared workers, shared buffers and close scheduling—became the route by which unrelated trust domains touched.

A dormant pointer error met a new buffering pattern

Cloudflare traced the defect to old Ragel-generated code used by three HTML transformation features: Email Obfuscation, Server-Side Excludes and Automatic HTTPS Rewrites. A missing fhold operation could leave a pointer at the end of a buffer. The caller then treated the parser's result as a valid span and copied past the intended boundary.

The sharp rise in exposure followed deployment of a newer parser called cf-html. That deployment changed buffering and made the legacy path encounter the boundary condition more often. Cloudflare was explicit about the distinction: the defect was not in Ragel itself and not in cf-html itself. A new component changed the conditions around old code and converted a dormant defect into a more frequent disclosure.

This is a difficult class of change risk. Component review asks whether the new code is correct. Boundary review asks what assumptions in the surrounding system the new code invalidates. A safe rollout therefore needs differential tests at buffer boundaries, canaries that inspect response provenance, and crash analysis across both the replaced component and every legacy consumer it wakes.

An exact shutdown sequence

The first recovery authority was local and immediate: remove the feature path that could emit new data. The incident report's detailed UTC timeline records receipt of the bug details at 00:32 on 18 February, Email Obfuscation disabled worldwide at 01:19, Automatic HTTPS Rewrites disabled at 04:24, and the fleet-wide patch implementing the cf-html kill switch at 07:22. Server-Side Excludes lacked its own global kill switch, so engineers had to build and deploy one.

That sequence shows why a kill switch is part of a feature's security design, not an optional operations convenience. The feature without one required code work during an incident. A reversible feature can be sacrificed while unrelated proxying continues; an inseparable feature forces operators to choose between a wider outage and continued exposure.

Cloudflare also reported that customer SSL private keys were not exposed through this path because TLS termination ran in a separate NGINX instance. The boundary was not absolute: the leaking HTTP process did contain a private key used for Cloudflare machine-to-machine encryption and a small number of internal authentication secrets. Process isolation protected one high-consequence class while leaving other secrets reachable; it reduced the blast radius rather than making it empty.

The repair later moved the parser toward Rust, whose memory-safety checks prevent this kind of out-of-bounds access in safe code. That is a stronger local invariant than asking reviewers to find every pointer mistake. It does not eliminate faulty authorization, unsafe foreign code or bad data-flow design, but it removes one broad failure class from the ordinary implementation path.

The rate was an estimate, not a census

Cloudflare's initial incident report placed the highest exposure between 13 and 18 February and estimated roughly one potentially leaking request in 3.3 million, about 0.00003 per cent. Its later reconstruction estimated 1,242,071 trigger requests from 22 September 2016 through 18 February 2017. Of those, 605,037 were estimated before 13 February from fewer than 180 trigger sites, and 637,034 during the final five days from 6,457 trigger sites.

Those numbers are useful only with their evidence limits attached. Detailed request logs were sampled at one per cent and available for 8–18 February. Older detailed logs had already been deleted, so earlier estimates relied on aggregate analytics and crash records. Cloudflare said it found no evidence of malicious exploitation in the records it reviewed. That is an observation inside a finite record, not proof that no attacker ever noticed or used the leak.

Incident arithmetic must not manufacture certainty that the telemetry could not provide. Trigger requests are not unique victims. A rate across all traffic does not express the sensitivity of a particular leaked fragment. A negative search in sampled logs cannot close the historical record.

Containment did not recall the bytes

Disabling the features ended the creation of new contaminated responses. It did not retract responses already delivered, crawler indexes already populated or credentials already copied. Cloudflare reported that search crawlers accounted for more than half of estimated trigger requests, that it purged more than 80,000 unique cached pages, and that its investigation found data associated with 150 customers in third-party caches.

None of those figures proves a complete victim count. Not every purged page necessarily held sensitive material; 150 identified customers are not an upper bound; a cache purge cannot prove the absence of private copies. The correct operational conclusion is that containment and recovery have different endpoints.

A parser fix proves only that the known path no longer creates new leakage under tested conditions. Cache owners must separately remove stored fragments. Customers must decide which sessions, tokens, API secrets and passwords could have crossed the boundary, then rotate or revoke them at every relying system. Completion evidence is rejection of the old authority, not issuance of a replacement string.

A minimum common boundary, local reversible choices

Heng Lu's Running-Code Primacy supplies a disciplined way to describe the proof. An incident is not over because a status page says “resolved.” The running edge must no longer emit cross-request memory; feature state must be observable; old credentials must fail where they once worked; and residual caches must be queried and cleaned by the parties that control them.

Minimum Initial Specification sharpens the architecture. The common contract at a multi-tenant edge should be small and absolute: a response may contain only data authorized for that response. Optional transformations, parser implementations, rollout cadence and feature shutdown remain local choices. They can evolve or be refused without weakening the isolation invariant or requiring every customer to accept the same feature risk.

Cloudbleed did not show that shared infrastructure is impossible. It showed that sharing needs a hard membrane, and that every optional path crossing that membrane needs a local way to disappear before the rest of the service does.

Sources