Incident with Git Operations, API Requests and Issues


Incident resolved in 3h5m32s

Resolved

On April 28th, 2025, between 4AM and 11AM UTC, ~0.5% of customers experienced HTTP 500 or 429 responses for raw file access (via the GitHub website and APIs). Additionally, ~0.5% of customers may have seen slow pull request page loads and increased timeouts in the GraphQL API. The incident was caused by queueing in serving systems due to a change in traffic patterns, specifically scraping activity targeting our API. We have adjusted limits and added flow control to systems in response to the changing traffic patterns to improve our ability to prevent future large queueing issues. We’ve additionally updated rate limiting unauthenticated requests to reduce overall load, more details are here: https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/

1745838550

Investigating

We are seeing signs of recovery and continue to monitor latency.

1745836529

Investigating

We continue to investigate impact to Issues and Pull Requests. Customers may see some timeouts as we work towards mitigation.

1745833929

Investigating

We are continuing to investigate impact to Issues and Pull Requests. We will provide more updates as we have them.

1745830683

Investigating

Users may see timeouts when viewing Pull Requests. We are still investigating the issues related to Issues and Pull Requests and will provide further updates as soon as we can

1745828593

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

1745828508

Investigating

Issues API is currently seeing elevated latency. We are investigating the issue and will provide further updates as soon as we have them.

1745827517

Investigating

We are investigating reports of degraded performance for API Requests, Git Operations and Issues

1745827418