Summary

  • Adding Kinesis front-end servers also increased the communication work of existing servers. The fleet crossed an operating-system thread limit and could no longer finish building usable routing maps.
  • Restarting created competition between rebuilding those maps and serving customers. Thousands of available machines had to return at a controlled rate of a few hundred an hour.

One of AWS's proposed remedies for the Kinesis failure of November 25, 2020 was to use bigger machines so it could have fewer servers. Fewer servers meant fewer conversations for each server to maintain. The cloud had run into a limit on keeping up with its own guest list.

The AWS post-event report identifies a relatively small capacity addition in Northern Virginia as the trigger. The root cause was an operating-system configuration limit on threads. Adding front ends had pushed every server in the fleet past that limit.

That is a useful distinction for anyone buying capacity. A new machine can bring processors to the business and paperwork to every existing machine. Useful capacity is what remains after the paperwork.

The neighbours came with overhead

Kinesis processed sharded streams in back-end cell-clusters. A front-end fleet handled authentication, throttling and the routing of requests to the right shards. Each front end kept a map of membership and shard ownership.

Several sources fed that map, including continuous messages from other front ends. Each server created operating-system threads to communicate with the other members. When new servers arrived, the existing members discovered them and established the necessary threads. Discovery could take up to an hour.

The installation could therefore finish before every existing machine had absorbed its new obligations. Once the thread ceiling was crossed, map construction failed to finish. The machinery behind the entrance could process streams; the entrance could no longer reliably direct requests to it.

Back-end cells had a defined scope of isolation. The growing front-end conversation had its own scope. A cellular component somewhere in an architecture does not put a fence around everything that component needs.

This is the awkward economics of the marginal server. Its contribution includes an increase in the coordination cost of machines already deployed. A fleet can gain hardware while losing usable service if that cost reaches a shared limit first. A chart of installed processors will faithfully count the hardware and miss the reason customers cannot use it.

Recovery had a speed limit of its own

AWS removed the added capacity. The team wanted further testing before raising the operating-system limit. It also had to restart the front ends, which brought a second constraint into view.

Rebuilding maps competed for resources with incoming requests. Bringing servers back too quickly could make them slow, get them judged unhealthy and remove them from the fleet again. An aggressive restart could undo its own progress.

Engineers changed bootstrapping so front ends could read from the authoritative metadata store rather than obtain that information from their neighbours. Even so, a fleet of many thousands could return only a few hundred servers per hour. The first group took traffic at 10:07am Pacific Standard Time. Kinesis returned to normal at 10:23pm, with error rates falling during the recovery.

The machines existed. Their safe rate of admission into useful service was the scarce resource. Finding a fault and removing its trigger did not instantly deliver all the capacity sitting behind it.

Recovery also brings an unusually demanding mix of work: new requests, accumulated work and the reconstruction of the system's own operating information. A normal throughput test starts with much of that information already in place. A buyer needs to know how the service performs while earning its way back to those test conditions.

The supporting work reached the front door

Cognito used Kinesis to collect and analyse API access patterns. Delivery was designed to be best effort, with local buffering to tolerate temporary trouble. The prolonged disruption exposed a latent buffering bug. Backlogged data began blocking webservers, and some users could not authenticate or obtain temporary credentials.

Information about using a service had become an obstacle to using it. A buffer bought time. Its behaviour when that time ran out determined whether the main function could continue. Making a buffer larger postpones that examination; it still has to pass eventually.

CloudWatch struggled to process metrics and logs. Some alarms entered an insufficient-data state, while reactive Auto Scaling policies using those metrics experienced delays. In Lambda, buffering metrics caused memory contention on invocation hosts and increased errors. These were different failure mechanisms downstream of the same event. The resource that failed first in Kinesis was not the resource that failed next everywhere else.

The usual tool for publishing to the Service Health Dashboard also used Cognito. A backup with fewer dependencies worked, but its more manual operation and unfamiliarity caused early posting delays. The emergency exit needed practice as well as a door.

Buying fewer conversations

AWS's announced actions addressed several parts of the problem. Bigger CPU and memory machines would reduce the total front-end count and the threads each needed for its peers. Finer thread alarms, testing a higher limit and faster cold starts would provide more room and a better route back. Separating the cache, partitioning front ends for some large services and accelerating front-end cellularization would change the shared failure boundary.

Those are different goods: headroom, warning, a quicker restart and a smaller affected population. Folding them into a single capacity number makes their purpose harder to price.

AWS's account of static stability describes preparing a system to keep working through dependency impairment. Its discussion of fault tolerance and fault isolation also recognises the cost of spare resources prepared in advance. Idle capacity occupies capital. Capacity that must be activated during a failure can occupy the customer's time instead.

The historical report supplies neither today's front-end limits nor a recent full-fleet restart test. A current test with disclosed conditions would change a present recovery assessment. The purchasing question from 2020 is already sharp enough: how quickly can the system make its installed capacity useful again?