Degraded REST API success rates for some customers

Resolved

On September 4, 2025 between 15:30 UTC and 20:00 UTC the REST API endpoints git/refs, git/refs/, and git/matching-refs/ were degraded and returned elevated errors for repositories with reference counts over 22k. On average, the request error rate to these specific endpoints was 0.5%. Overall REST API availability remained 99.9999%. This was due to the introduction of a code change that added latency to reference evaluations and overly affected repositories with many branches, tags, or other references.We mitigated the incident by reverting the new code.We are working to improve performance testing and to reduce our time to detection and mitigation of issues like this one in the future.

Investigating

The deployment has completed and we expect customers who have been impacted to see recovery. We are continuing to monitor.

Investigating

We are in the process of deploying the PR to revert the change that was causing timeouts to this endpoint. We will update again once that deployment is complete.

Investigating

We have identified a deployed change that correlates with the increase in 5XX errors to the GitRefs REST API. This is particularly affecting requests for repos with very large numbers of commits. We are working on rolling back this change which we expect will resolve the issue.

Investigating

API Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

Customers are experiencing 504 responses for some API requests for regarding repo refs/tags. We are investigating.

Investigating

We are currently investigating this issue.