Actions workflow run and job status updates are experiencing delays

Resolved

On January 15th, between 14:18 UTC and 15:26 UTC, customers experienced delays in status updates for workflow runs and checks. Status updates were delayed by up to 20 minutes, with a median delay of 11 minutes.The issue stemmed from an infrastructure upgrade to our database cluster. The new version introduced resource contention under production load, causing slow query times. We mitigated this by rolling back to the previous stable version. We are working to strengthen our upgrade validation process to catch issues that only manifest under high load. We are also adding new monitors to reduce detection time for similar issues in the future.

Investigating

We are continuing to monitor as the system recovers and expect full recovery within the next 20-30 minutes. Impacted users will see that job status appears queued, though the job itself is actually running.

Investigating

We are seeing signs of recovery and are continuing to monitor as we process the backlog of events.

Investigating

We are investigating reports of degraded performance for Actions