All GitHub services are experiencing significant disruptions

Resolved

On August 14, 2024 between 23:02 UTC and 23:38 UTC, all GitHub services on GitHub.com were inaccessible for all users. This was due to a configuration change that impacted traffic routing within our database infrastructure, resulting in critical services unexpectedly losing database connectivity. There was no data loss or corruption during this incident. At 22:59 UTC an erroneous configuration change rolled out to all GitHub.com databases that impacted the ability of the database to respond to health check pings from the routing service. As a result, the routing service could not detect healthy databases to route application traffic to. This led to widespread impact on GitHub.com starting at 23:02 UTC. We mitigated the incident by reverting the change and confirming restored connectivity to our databases. At 23:38 UTC, traffic resumed and all services recovered to full health. Out of an abundance of caution, we continued to monitor before resolving the incident at 00:30 UTC on August 15th, 2024. To prevent recurrence we are implementing additional guardrails in our database change management process. We are also prioritizing several repair items such as faster rollback functionality and more resilience to dependency failures. Given the severity of this incident, follow-up items are the highest priority work for teams at this time.

Investigating

We have fully rolled-back the changes to database infrastructure and mitigated the impact. All services are now fully operational.

Investigating

Git Operations is operating normally.

Investigating

Copilot is operating normally.

Investigating

Codespaces, Packages and Pages are operating normally.

Investigating

Webhooks is operating normally.

Investigating

Actions is operating normally.

Investigating

Actions is operating normally.

Investigating

Actions is operating normally.

Investigating

Actions is experiencing degraded performance. We are continuing to investigate.

Investigating

Pull Requests is operating normally.

Investigating

Issues is operating normally.

Investigating

Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating

Git Operations is experiencing degraded performance. We are continuing to investigate.

Investigating

Codespaces is experiencing degraded performance. We are continuing to investigate.

Investigating

Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating

Actions is experiencing degraded performance. We are continuing to investigate.

Investigating

Pages is experiencing degraded performance. We are continuing to investigate.

Investigating

Service health continues to improve, and we are working to stabilize all services. Some services may experience delays in updates and notifications as we work through a backlog of events.

Investigating

Packages is experiencing degraded performance. We are continuing to investigate.

Investigating

Issues is experiencing degraded performance. We are continuing to investigate.

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

The database infrastructure change is being rolled back. We are seeing improvements in service health and are monitoring for full recovery.

Investigating

We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.

Investigating

Codespaces is experiencing degraded availability. We are continuing to investigate.

Investigating

Webhooks is experiencing degraded availability. We are continuing to investigate.

Investigating

Issues is experiencing degraded availability. We are continuing to investigate.

Investigating

Git Operations is experiencing degraded availability. We are continuing to investigate.

Investigating

Packages is experiencing degraded availability. We are continuing to investigate.

Investigating

We are investigating reports of issues with GitHub.com and GitHub API. We will continue to keep users updated on progress towards mitigation.

Investigating

Copilot is experiencing degraded availability. We are continuing to investigate.

Investigating

Pages is experiencing degraded availability. We are continuing to investigate.

Investigating

We are investigating reports of degraded availability for Actions, Pages and Pull Requests