Summary
- The disruption began on July 24 at 16:04 UTC; GitHub added the detailed root-cause account on July 29 at 02:36:25 UTC.
- Connectivity failed between the spine switches of one compute cage and the aggregation layer in one of GitHub’s three physical data-centre availability zones.
- The remaining paths saturated, producing packet loss after 25% of available network interconnect capacity was lost.
- During the impact window, 10% of Actions jobs failed, another 5% started late, 27% of Issues interactions were slow or timed out, and 4% of both Copilot requests and git pushes were affected.
- GitHub rerouted connections onto fibre reserved for future upgrades, restored enough capacity at 17:07 and all paths by 17:36, and says it is accelerating older cages from 100Gbps to 400Gbps interfaces.
The useful surprise in GitHub’s disclosure is not that many products shared a failure. It is how differently one packet-loss condition appeared at the application edge. A job could fail, begin late or finish normally. An Issue interaction could stall. A push could require verification. A Copilot request could be retried automatically. Authentication mostly remained successful while becoming slower.
Those are not interchangeable symptoms, and they do not leave the same recovery work. The root-cause update therefore gives operators two ledgers to reconcile: the physical capacity GitHub restored and the application actions customers may need to replay, confirm or deliberately leave untouched.
The new event is the disclosure, not a second outage
The underlying incident ran on July 24. GitHub’s status record shows the service sequence that day, including API Requests, Issues, Pages, Actions, Pull Requests and Copilot. The decision-grade change arrived later: the detailed incident update was modified at 02:36:25 UTC on July 29, inside this reporting window.
That distinction matters. Calling this a July 29 outage would move the event by five days and imply a recurrence that the source does not report. What happened on July 29 was evidence publication: GitHub disclosed the topology layer, capacity loss, service percentages, emergency route and longer-term upgrade response.
The new account still stops short of naming the initiating device, maintenance action or other trigger that caused connectivity to disappear. It explains the failure mechanism after path loss—surviving links saturated and packets were dropped—but not why the original links were lost. The gap should remain visible.
One cage failure crossed six service boundaries
GitHub describes a leaf-spine fabric in each compute cage and an aggregation layer connecting cage spines within each availability zone. In one of three physical zones, links between one cage’s spine switches and aggregation lost connectivity. Workloads depending on compute in that cage then encountered packet loss as the remaining network paths filled.
The published magnitude is unusually specific. Ten per cent of Actions jobs failed, while 5% succeeded after delayed starts. Twenty-seven per cent of Issues interactions were slow or timed out. Four per cent of Copilot requests produced errors, although GitHub says most automatically retried. Four per cent of git push operations were affected. Authentication latency increased, but error rates stayed below 1%.
The denominators are different products and actions, so the percentages must not be added into a fictional global failure rate. They show propagation instead: constrained network capacity became queueing, timeout or error behaviour according to each service’s retry policy, state model and latency budget.
Recovery borrowed tomorrow’s capacity
GitHub mitigated the incident by rerouting the affected connections onto fibre paths allocated for future capacity upgrades. Enough capacity to eliminate packet loss was back at 17:07 UTC. Most services showed full recovery by 17:16, and all paths were restored with services healthy at 17:36.
The sequence exposes three distinct recovery thresholds. First came sufficient headroom to stop packet loss. Then most application services recovered. Finally, the complete path set returned. A dashboard that records only the final time loses the evidence that capacity and service recovery were staged.
Using reserved fibre was an effective emergency option, but it also consumed a contingency that had been intended for later growth. Operators depending on GitHub should not assume the temporary route tells them how much spare capacity remained after the incident. GitHub does not publish that figure, the geography of customer impact or per-customer outage durations.
Every percentage leaves a different cleanup queue
Failed Actions jobs require judgment before replay. GitHub’s documentation allows users with write permission to rerun failed jobs, specific jobs or an entire workflow. A rerun uses the original actor’s privileges, commit SHA and Git ref. That preserves execution context, but it does not make a state-changing deployment, payment, release or database migration safe to repeat automatically.
The 5% of jobs that started late pose another question: they may still have completed after a newer run began. Teams need to compare run IDs, deployment environments and external side effects rather than blindly rerun everything that looked delayed. For git pushes, the correct response is to verify whether the remote reference advanced before pushing again. For Issues interactions, a client timeout does not prove that a comment or edit failed to persist.
Copilot’s automatic retry reduces visible error burden but creates a different measurement problem: application success after retry can conceal degraded first-attempt reliability. Authentication below 1% errors is similarly not “unaffected” when latency rises. The RCA supports separate recovery checks, not one all-clear.
Four hundred gigabits is a remediation, not a post-mortem
GitHub says older cages use 100Gbps network interfaces and that it is accelerating planned upgrades to 400Gbps. More bandwidth at each fabric layer should increase tolerance for a path or device loss. The commitment is material because it responds directly to saturation, the mechanism that converted link loss into user impact.
Capacity alone cannot answer every resilience question. The public account does not say when the upgrade will finish, how many cages remain on the older standard, what failure combinations the new design must tolerate or whether route isolation and detection will change. Four times the interface rate is not automatically four times the end-to-end resilience.
The stronger operational measure is headroom under a defined failure scenario. GitHub can eventually demonstrate whether one cage-to-aggregation loss remains below saturation, whether automation detects it before users do, and whether recovery tools remain reachable during congestion.
What customers and GitHub should measure next
Customers can preserve the incident evidence now: failed and delayed workflow IDs, push outcomes, timed-out Issue mutations, authentication latency and any Copilot retry counts available to them. Recovery checks should reconcile state, not merely repeat requests. A runbook should distinguish idempotent reads from writes with external effects.
GitHub’s next useful disclosure would identify the initiating loss of connectivity, quantify post-mitigation headroom, give the scope and timetable of the 400Gbps programme, and state which controls prevent a cage-level path loss from saturating the remainder. It should also clarify whether the fibre used for emergency routing has returned to reserve status.
The July 29 update is valuable because it makes a shared cloud-service dependency physical again. Six service labels did not fail independently. One quarter of interconnect capacity disappeared, the remaining paths filled, and each product translated packet loss into its own user-visible queue.