GitHub CTO apologises to developers after one of the biggest outages; admits: Since April, monthly commits have doubled, this shows pressure on our systems, but it is no excuse for outages
GitHub has apologised for the outage that took its platform down for 7 hours and 47 minutes on August 17, and the explanation is unusually blunt. Nobody pushed bad code. Nobody fumbled a configuration change. The platform simply ran out of room.In a blog post published on August 20, Vlad Fedorov wrote that traffic hit a new peak and a critical infrastructure component in GitHub’s Central US data centre failed to scale with it. Authentication buckled first, and the pressure spread outward through github.com, Actions, APIs, pull requests, issues and Copilot. “If you were trying to ship software that day, we let you down,” the post reads.
What caused the GitHub outage : an autoscaling policy watching the wrong thing
The root cause analysis points at an Istio sidecar reaching its concurrency limit while the autoscaling policy monitored the host service instead. Load balancers in Central US saturated. A latent retry bug in Visual Studio Code then amplified traffic to a single internal endpoint by roughly 10 times, which is what kept the Copilot Token Service down long after everything else came back. Engineers cut gateway retries and began rejecting inbound Copilot token requests with HTTP 403 responses. Scraping attacks on codeload endpoints made recovery messier still.At the worst of it, GitHub’s status page reported error rates near 20 percent for web and API traffic, and around 50 percent for archive and raw repository downloads. Most services recovered by 1636 UTC, Actions by 1803 UTC, and the Copilot Token Service only at 2102 UTC.
Why GitHub’s own growth has become its biggest reliability problem
Fedorov’s numbers explain the strain better than any apology. Monthly commits have climbed from 1.4 billion in April to 2.9 billion now. GitHub has added more than 3 million CPU cores and 120 petabytes of high-speed storage, and installed as much hardware as available power allowed in existing data centres. Azure now carries about 58 percent of platform load, up from 12 percent in May, and handles half of all Git operations.What comes next is a fix list: consistent retry limits, retry budgets and variable timeouts across service-to-service calls, a review of lower-priority CPU and memory alerts, and a new architecture that scales read capacity linearly with the number of readers, rolling out first to the largest monorepos.The timing is awkward. This was GitHub’s second incident in August, after an Actions failure on August 6. Cursor announced an early beta of its Origin Code Hosting product while GitHub was still dark. CloudBees CEO Moritz Plassnig wrote on LinkedIn that GitHub will not be the default choice going forward, and that the ecosystem is heading somewhere more bifurcated.
