Disruption with some GitHub services

Resolved

On December 4th, 2024 between 18:52 UTC and 19:11 UTC, several GitHub services were degraded with an average error rate of 8%.The incident was caused by a change to a centralized authorization service that contained an unoptimized database query. This led to an increase in overall load on a shared database cluster, resulting in a cascading effect on multiple services and specifically affecting repository access authorization checks. We mitigated the incident after rolling back the change at 19:07 UTC, fully recovering within 4 minutes. While this incident was caught and remedied quickly, we are implementing process improvements around recognizing and reducing risk of changes involving high volume authorization checks. We are investing in broad improvements to our safe rollout process, such as improving early detection mechanisms.

Investigating

Pull Requests is operating normally.

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

Issues is operating normally.

Investigating

API Requests is operating normally.

Investigating

Webhooks is operating normally.

Investigating

We have identified the cause of timeouts impacting users across multiple services. This change was rolled back and we are seeing recovery. We will continue to monitor for complete recovery.

Investigating

Issues is experiencing degraded performance. We are continuing to investigate.

Investigating

API Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating

We are currently investigating this issue.