Summary

  • Publicly reachable memcached UDP services could turn a small request carrying a forged victim address into a response thousands of times larger, making unrelated caches spend their egress capacity on a party that never asked.
  • The durable answer was divided: memcached 1.5.6 disabled UDP by default, source networks could reject forged addresses near their origin, and GitHub retained local detection, BGP diversion and scrubbing controls for traffic that still arrived.

The reply was an unauthorized expenditure

Memcached was designed as a fast cache, usually inside a trusted application network. Its UDP mode did not establish a connection before returning data. That economy became a security defect when an instance was reachable from the public Internet: a packet could claim to come from somebody else's address, and the cache would send the result there.

Cloudflare described the practical sequence. An attacker first stored a large value on an exposed server, then sent a tiny get request with the victim's address forged as the source. In one synthetic trace, 15 bytes produced 134 KB. Cloudflare separately observed a 15-byte request yield 750 KB—51,200 times the input. Those are examples, not a universal multiplier, but they reveal the economic mechanism. The attacker purchased output bandwidth with another operator's cache and charged delivery to the victim.

Reflection therefore required two permissions that nobody intended to grant. The cache had to answer untrusted UDP, and some access path had to export a packet bearing an address the sender did not control. Remove either condition and that particular chain breaks.

Nine minutes at GitHub's edge

GitHub's own incident report provides the event boundary. On 28 February 2018, GitHub.com was unavailable from 17:21 to 17:26 UTC and intermittently unavailable until 17:30. GitHub said confidentiality and integrity were never at risk.

The volume was nevertheless extraordinary: traffic from more than 1,000 autonomous systems and tens of thousands of endpoints peaked at 1.35 Tbps and 126.9 million packets per second. The figures describe GitHub's attack, not every memcached reflector or every campaign using the vector.

At 17:21, GitHub's monitoring detected an abnormal ratio of ingress to egress traffic. When inbound transit exceeded 100 Gbps at one facility, the team chose to move traffic to Akamai. At 17:26 a ChatOps command withdrew GitHub's BGP announcements over transit providers and announced AS36459 exclusively over the Akamai links. Routes reconverged; ACLs at Akamai's border mitigated the flow; GitHub reported full recovery at 17:30.

Four minutes later, Internet-exchange routes were also withdrawn, moving another 40 Gbps. A second spike of about 400 Gbps arrived after 18:00. GitHub said it would investigate automating activation of DDoS providers to reduce recovery time. That was a declared next step, not evidence that the automation already existed.

The chronology is useful because it shows the victim's real control surface. GitHub could not edit tens of thousands of remote caches during the incident. It could observe its own edge, withdraw its own routes, choose a mitigation partner and test whether service recovered.

Two changed lines were more consequential than a warning

Memcached 1.5.6 was released on 27 February, one day before the GitHub incident. Its release note says the bugfix release primarily disabled UDP by default. The underlying commit is small enough to inspect: settings.udpport changed from 11211 to 0, and specifying only a TCP port no longer implicitly enabled UDP on that same port. The tests changed with the behavior.

UDP was not deleted. An operator could restore it explicitly with -U 11211. That distinction matters. The project did not claim a central power to forbid a legitimate local use; it changed the initial condition so that silence, inattention or a fresh installation did not create a public reflector by accident.

Nor did a version label settle the question. A distributor, service unit or local administrator could pass an option that re-enabled UDP. An older binary could be bound privately and firewalled. The security fact is whether the running host answers an untrusted UDP packet on port 11211 and how much it returns—not what a package inventory implies.

This is Running-Code Primacy at unusually high resolution. A release note described an intention. Two code paths changed the default. A test asserted the new behavior. Only deployed sockets and external probes showed whether a particular cache had actually stopped answering.

The forged source belonged to another operator's boundary

Closing reflectors addresses the multiplication. Source-address validation addresses the lie that points replies at a victim.

RFC 2827, better known as BCP 38, recommends that an Internet provider filter traffic from a downstream network so it carries only source prefixes legitimately used by that customer. At an appropriate aggregation point, that prevents an attacker behind the edge from exporting a packet that pretends to be GitHub.

The objective is simple; implementation is not always so. RFC 3704 updates the practice for multihomed and asymmetric networks. It describes static ingress lists, strict reverse-path forwarding, feasible-path methods and looser variants. A strict check can discard legitimate asymmetric traffic if the topology is modeled badly. A loose check may confirm that a route exists while failing to prove that this customer was entitled to originate the address.

Minimum Initial Specification draws the right line. The shared safety rule should be that a customer edge does not export source identities it is not authorized to use. The common layer need not impose one router command on every topology. Each operator keeps the future decision—ACLs, feasible-path RPF, SAVI, automation, exceptions and route data—but must be able to demonstrate the outcome.

Three controls, three proofs

The reflector operator proves that untrusted UDP is closed or bounded by showing listening configuration, firewall state, external probe results and egress behavior. The source network proves anti-spoofing with customer-edge tests and flow evidence. The potential victim proves readiness by detecting anomalies, rehearsing route changes, retaining scrubbing capacity and measuring recovery.

These proofs are not substitutes. Scrubbing saves the victim after remote caches have emitted the traffic. It does not close those caches or stop forged packets at their origin. Disabling UDP prevents one amplification service from replying, but it does not make the source network honest. BCP 38 blocks many forged-source reflection paths, but it does not give GitHub capacity or a tested BGP escape route.

The architecture is strongest when authority stays with the party carrying the consequence. A cache operator decides whether UDP has a valid local use. An access provider validates the sources it admits. GitHub decides when its own telemetry justifies diversion. Akamai filters only because GitHub chose that path. There is no need to turn the mitigation provider, the software project or a standards body into a standing governor of all traffic.

Evidence limits

The public record supports a powerful amplification mechanism, a measured 1.35 Tbps GitHub incident and a contemporaneous upstream default change. It does not show that every cache was reachable, that every reply amplified by 51,200, or that GitHub's packets reproduced Cloudflare's laboratory trace. It does not prove universal BCP 38 deployment, and it does not show that GitHub later implemented every automation it proposed.

The more durable conclusion is about delegated power. A default can quietly authorize a machine to answer strangers; an unfiltered edge can let a sender borrow somebody else's identity; a victim can recover only through controls it has already built. Publication announces a remedy. Running behavior decides who still has permission to spend bandwidth.

Sources