Incident with Issues, Git Operations and API Requests

Resolved

On March 3rd 2025 between 04:07 UTC and 09:36 UTC various GitHub services were degraded with an average error rate of 0.03% and peak error rate of 9%. This issue impacted web requests, API requests, and git operations. This incident was triggered because a network node in one of GitHub's datacenter sites partially failed, resulting in silent packet drops for traffic served by that site. At 09:22 UTC, we identified the failing network node, and at 09:36 UTC we addressed the issue by removing the faulty network node from production.In response to this incident, we are improving our monitoring capabilities to identify and respond to similar silent errors more effectively in the future.

Investigating

We have seen recovery across our services and impact is mitigated.

Investigating

Git Operations is operating normally.

Investigating

Webhooks is operating normally.

Investigating

We are investigating intermittent connectivity issues between our backend and databases and will provide further updates as we have them. The current impact is you may see elevated latency while using our services.

Investigating

We are seeing intermittent timeouts across our various services. We are currently investigating and will provide updates.

Investigating

Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating

We are investigating reports of degraded performance for API Requests, Git Operations and Issues