Incident with multiple GitHub services

Resolved

On July 17, 2024, between 16:15:31 UTC and 17:06:53 UTC, various GitHub services were degraded including Login, the GraphQL API, Issues, Pages and Packages. On average, the error rate was 0.3% for requests to github.com and the API, and 3.0% of requests for Packages. This incident was triggered by two unrelated events:- A planned testing event of an internal feature caused heavy loads on our databases, disrupting services across GitHub.- A network configuration change deployed to support capacity expansion in a GitHub data center. We partially resolved the incident by aborting the testing event at 16:17 UTC and fully resolved the incident by rolling back the network configuration changes at 16:49 UTC. We have paused all planned capacity expansion activity within GitHub data centers until we have stabilized the root cause of this incident. In addition, we are reexamining our load testing practices so they can be done in a safer environment and making architectural changes to the feature that caused issues.

Investigating

Git Operations is operating normally.

Investigating

Pages is operating normally.

Investigating

Packages is operating normally.

Investigating

We're continuing to investigate reports of issues with multiple services. We will continue to keep users updated on progress.

Investigating

Packages is experiencing degraded performance. We are continuing to investigate.

Investigating

Issues is operating normally.

Investigating

Pages is experiencing degraded performance. We are continuing to investigate.

Investigating

Issues is experiencing degraded performance. We are continuing to investigate.

Investigating

We are investigating reports of issues with service(s): Git, Actions, Rules and Permissions, SSH authentication, and Authorization.. We will continue to keep users updated on progress towards mitigation.

Investigating

We are investigating reports of degraded performance for Git Operations