Incident with all GitHub services


Incident resolved in 1h9m27s

Resolved

On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: - We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. - We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.

1772568556

Investigating

We're seeing recovery across all services. We're continuing to monitor for full recovery.

1772568382

Investigating

Actions is operating normally.

1772567704

Investigating

Git Operations is operating normally.

1772567680

Investigating

Git Operations is experiencing degraded availability. We are continuing to investigate.

1772566602

Investigating

We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate

1772566418

Investigating

Copilot is operating normally.

1772566310

Investigating

Pull Requests is operating normally.

1772566301

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

1772566130

Investigating

Issues is operating normally.

1772566043

Investigating

Webhooks is operating normally.

1772565921

Investigating

Codespaces is operating normally.

1772565905

Investigating

Webhooks is experiencing degraded performance. We are continuing to investigate.

1772565863

Investigating

Issues is experiencing degraded performance. We are continuing to investigate.

1772565827

Investigating

We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.

1772565436

Investigating

API Requests is operating normally.

1772565325

Investigating

API Requests is experiencing degraded performance. We are continuing to investigate.

1772565269

Investigating

Codespaces is experiencing degraded performance. We are continuing to investigate.

1772565113

Investigating

Pull Requests is experiencing degraded availability. We are continuing to investigate.

1772564751

Investigating

Webhooks is experiencing degraded availability. We are continuing to investigate.

1772564664

Investigating

We're seeing some service degradation across GitHub services. We're currently investigating impact.

1772564616

Investigating

Webhooks is experiencing degraded performance. We are continuing to investigate.

1772564560

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

1772564427

Investigating

API Requests is experiencing degraded availability. We are continuing to investigate.

1772564420

Investigating

We are investigating reports of degraded availability for Actions, Copilot and Issues

1772564389