Incident with CodeQL, Webhooks, Notifications, and Slack Integration

Resolved

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

Investigating

All services have fully recovered.

Investigating

CodeQL has fully recovered. We're continuing to work on recovery for the remaining impacted services.

Investigating

Webhooks have fully recovered. Continuing to work on recovery for the other services.

Investigating

Webhooks is operating normally.

Investigating

We've established that most delays are related to a queuing service and are working to scale out. Early signals from the scale-out are showing signs of recovery for some services. We'll provide an update when services are fully recovered.

Investigating

Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating

We're continuing to investigate issues with CodeQL actions workflows. We're additionally seeing delays for notifications, webhooks, and the Slack integration.

Investigating

CodeQL actions are currently experiencing delays, which may result in those actions being stuck in a pending state or having failed due to a timeout.

Investigating

We are investigating reports of degraded performance for CodeQL