Summary

  • GitHub materially expanded incident qcvjkzcs7j74 on 11 August; the underlying Actions disruption ran from 15:05 UTC on 6 August to 00:14 UTC on 7 August.
  • At the peak, GitHub says 71% of workflow runs suffered infrastructure failures and 75% of the remainder waited more than five minutes.
  • A routine deployment replaced pods, exposed an existing capacity and concurrency weakness, saturated the remaining service and propagated failure across clusters and dependencies.
  • Engineers added capacity, throttled webhook-triggered work and increased backlog processing, then had to stop runners repeatedly acquiring jobs that were already invalid.
  • Some Actions Runner Controller runners required manual recovery, while some push and pull-request events could not be replayed automatically.
  • GitHub has announced stronger deployment safeguards, monitoring, queue resilience and automatic recovery, but the postmortem does not prove those controls are already deployed or effective.

The deployment consumed its own safety margin

The trigger was not an exceptional traffic surge. GitHub says a routine deployment replaced pods in an internal Actions service responsible for turning events into jobs. As capacity came out of rotation, what remained saturated, services crashed and impact cascaded through multiple clusters and downstream systems.

That sequence makes spare capacity part of deployment safety, not merely an efficiency buffer. A rolling change is safe only if the surviving fleet can carry both live demand and the temporary imbalance created by replacement.

Two percentages describe different populations

The peak figures must not be added or flattened into one rate. GitHub reports infrastructure failure for 71% of runs. Of the remaining 29%, three quarters were delayed more than five minutes. The second percentage therefore describes a subset, not all workflows, and it does not mean the rest completed normally or on time.

The disclosure is nevertheless unusually useful: it shows that the customer surface contained both hard failure and prolonged queueing, which require different recovery checks.

Restoring capacity did not clear job semantics

The first recovery stage used more capacity, lower webhook intake and greater backlog-processing throughput. A second fault then became visible. A latent assignment bug offered runners jobs that were no longer valid; the runners retried those acquisitions and could not move on to legitimate work.

This is the difference between draining a queue and proving that every queue item still represents executable work. GitHub deployed a change to stop the repeated invalid acquisition before the accumulated queues could empty.

Recovery left work outside the platform’s automatic reach

Some Actions Runner Controller runners remained offline after a mitigation unintentionally affected them. GitHub rolled that change back, but some runners needed manual recovery. Separately, certain push and pull-request triggers were not processed and could not be replayed automatically.

For customers, the incident therefore ended in two places: the provider’s service state and each repository’s missing or stuck work. Teams had to determine whether to rerun a workflow, repeat a triggering action or recover a runner.

Resolution came before the detailed explanation

GitHub resolved the incident on 7 August; the substantive postmortem appeared through an update on 11 August. That later publication is the news event in this briefing, not a new outage and not an extension of the impact interval.

The distinction matters for incident statistics. Availability belongs to 6–7 August. Root-cause and control disclosure belongs to 11 August. Combining the clocks would overstate service duration and hide disclosure latency.

Promised safeguards are a test plan, not a result

GitHub says it is improving deployment and capacity safeguards, precursor monitoring, queued-work resilience, runner assignment, cascade containment and automatic recovery. Those commitments map closely to the disclosed failure chain.

They remain future controls until releases, rollout evidence or a later incident demonstrate them. The strongest validation would show how much spare capacity a deployment must preserve, how invalid jobs are retired and how stranded runners recover without manual action.

Customers need an independent reconciliation record

A workflow platform can return to operational while some business actions remain absent, duplicated in manual replay or blocked on a runner. Repository operators therefore need their own ledger of expected triggers, workflow run identifiers, terminal states and deployed artefacts.

That ledger turns a provider postmortem into a bounded recovery exercise. It also separates “GitHub Actions is healthy now” from “every change that should have run during the incident has been accounted for”.

Sources