We're experiencing errors

Resolved

On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.

Investigating

We continue to see signs of recovery.

Investigating

Issues is operating normally.

Investigating

Git Operations is operating normally.

Investigating

API Requests is operating normally.

Investigating

Copilot is operating normally.

Investigating

Packages is operating normally.

Investigating

Actions is operating normally.

Investigating

Packages is experiencing degraded performance. We are continuing to investigate.

Investigating

Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating

Actions is experiencing degraded performance. We are continuing to investigate.

Investigating

We are continuing to investigate degraded performance.

Investigating

Issues is experiencing degraded performance. We are continuing to investigate.

Investigating

We are investigating reports of degraded performance for API Requests and Git Operations