Disruption with Pull Request Ref Updates

Resolved

Between March 27, 2025, 23:45 UTC and March 28, 2025, 01:40 UTC the Pull Requests service was degraded and failed to update refs for repositories with higher traffic activity. This was due to a large repository migration that resulted in a larger than usual number of enqueued jobs; while simultaneously impacting git fileservers where the problematic repository was hosted. This resulted in an increase in queue depth due to retries on failures to perform those jobs causing delays for non-migration sourced jobs.We declared an incident once we confirmed that this issue was not isolated to the problematic migration and other repositories were also failing to process ref updates. We mitigated the issue by stopping the migration and short circuiting the remaining jobs. Additionally, we increased the worker pool of this job to reduce the time required to recover. As a result of this incident, we are revisiting our repository migration process and are working to isolate potentially problematic migration workloads from non-migration workloads.

Investigating

This issue has been mitigated and we are operating normally.

Investigating

We are continuing to monitor for recovery.

Investigating

We believe we have identified the source of the issue and are monitoring for recovery.

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

We are currently investigating this issue.