Summary

  • GitHub says its 17 August incident lasted 7 hours 47 minutes, with web and API errors peaking near 20% and archive and raw-content download errors near 50%. Issues, pull requests, Actions, identity services and Copilot were among the affected surfaces.
  • The company traces the failure to saturated Central US load balancers after an Istio sidecar reached its concurrency limit without scaling. Four HAProxy nodes then exhausted flow capacity, and retry behaviour drove Copilot token traffic from a normal 7,000–9,000 requests per second to 70,000–100,000.

GitHub’s primary services were mostly recovering by 16:36 UTC on 17 August. Its hardest recovery problem was still growing elsewhere.

In a detailed update to incident zkxwbgr0cnmx, published after the service had returned, GitHub says the disruption ran from 13:28 to 21:15 UTC. Issues, pull requests, APIs, Actions and Copilot experienced elevated errors or latency. At the peak, the company measured web and API error rates of about 20%, while archive and raw repository-content downloads reached roughly 50% errors.

The incident also reached enterprise control surfaces. GitHub lists SAML and OIDC authentication, SCIM and Team Sync among the affected services. Actions workflows in GitHub Enterprise Cloud with data residency were affected when they depended on public workflow step definitions hosted on GitHub.com. That is a narrower claim than failure of regional data storage: the public record identifies a workflow dependency, not a breach of data-residency commitments.

GitHub attributes the immediate failure to network saturation on load balancers in its Central US data centre during a new traffic peak. An Istio sidecar pod reached its concurrency limit. The autoscaling policy watched the host service but not the sidecar’s limit, so the constrained component did not scale with demand. The failure cascaded until four HAProxy nodes exhausted their flow limits, degrading the gateway authentication path and producing broader latency and authentication failures.

Moving traffic did help. GitHub says some failing requests were shifted from Central US to Northern Virginia and served successfully while engineers debugged the original failure. Most services had recovered by 16:36 UTC; Actions remained degraded until about 18:03.

But regional capacity did not end the incident because retries carried pressure with the traffic. Delayed replies from one internal endpoint triggered a latent retry bug in VS Code. GitHub says the behaviour amplified traffic by about ten times and delayed recovery of the Copilot Token Service. A failed token operation could create multiple additional requests and enter a retry loop. Token traffic rose from a normal 7,000–9,000 requests per second to 70,000–100,000.

The recovery actions show the difference between adding capacity and stopping amplification. GitHub temporarily reduced gateway retry logic, blocked inbound Copilot token requests at load balancers with HTTP 403 responses, and then restored traffic gradually by site. The company says those controls stabilised the token service, which fully recovered by 21:02 UTC. The incident record closed at 21:15.

Scraping traffic made the response more difficult. GitHub says several attacks targeted its codeload endpoints during the event, but it does not quantify how much they contributed to saturation or recovery time. The disclosed initiating chain remains the sidecar limit, the faulty scaling policy, exhausted HAProxy flows and retry amplification.

The product documentation explains why several customer activities crossed the same failure boundary. GitHub’s guide to reusing workflow configurations allows enterprise workflows to call definitions stored in public repositories when policy permits. Its hosted-runner reference lists GitHub.com, the API, Actions endpoints and codeload.github.com among the network destinations needed for runner operations and action downloads.

Identity is similarly central. GitHub documents SAML SSO and SCIM as mechanisms for controlling enterprise access and account lifecycles. Its data-residency overview describes dedicated GHE.com domains and regional storage choices. The incident shows that regional hosting and shared service dependencies are separate questions: where data is stored does not by itself specify every authentication or workflow path needed to use it.

GitHub’s prevention list now includes sidecar-aware autoscaling, an audit of Istio concurrency and scaling limits, review of gateway and client retry policies, a correction for the VS Code behaviour, stronger load-balancer capacity monitoring and improved regional failover safeguards. Those are commitments, not completed controls. The next evidence will be whether GitHub reports their implementation and whether later failures avoid the same authentication and retry pattern.

Customers do not have GitHub’s internal telemetry, but they can separate their own recovery tests. Repository reads, API calls, SSO, team membership changes, workflow-definition downloads, runner starts and Copilot authentication are different paths. Testing each one after a status change gives more information than treating “GitHub is back” as a single result.

GitHub’s public REST API guidance also tells integrations to honour Retry-After and rate-limit reset headers and to increase the delay between repeated attempts. That document addresses customer API behaviour rather than GitHub’s internal incident logic, but the operational principle is the same: retries without a bounded backoff can turn a delayed response into fresh demand.

Source