Incident with Webhooks


Incident resolved in 6h30m3s

Resolved

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

1772839693

Investigating

Webhooks is operating normally.

1772839690

Investigating

We have deployed a fix and are observing a full recovery. The affected endpoint was the webhook deliveries API (https://docs.github.com/en/rest/repos/webhooks?apiVersion=2022-11-28#list-deliveries-for-a-repository-webhook) and its organization and integration variants. We will continue monitoring to confirm stability.

1772839607

Investigating

We are preparing a new mitigation for the issue affecting the webhook deliveries API (https://docs.github.com/en/rest/repos/webhooks?apiVersion=2022-11-28#list-deliveries-for-a-repository-webhook) and its organization and integration variants. Overall impact remains low, with under 1% of requests failing for a subset of customers.

1772836508

Investigating

The previous mitigation did not resolve the issue. We are investigating further. The affected endpoint is the webhook deliveries API (https://docs.github.com/en/rest/repos/webhooks?apiVersion=2022-11-28#list-deliveries-for-a-repository-webhook) and its organization and integration variants. Overall impact remains low, with under 1% of requests failing for a subset of customers.

1772832895

Investigating

We have deployed a fix for the issue causing some users to experience intermittent failures when accessing the Webhooks API and configuration pages. We are monitoring to confirm full recovery.

1772828305

Investigating

We continue working on mitigations to restore service.

1772825945

Investigating

We continue working on mitigations to restore service.

1772824027

Investigating

We continue working on mitigations to restore service.

1772822353

Investigating

We continue working on mitigations to restore full service.

1772820473

Investigating

Our engineers have identified the root cause and are actively implementing mitigations to restore full service.

1772819004

Investigating

This problem is impacting less than 1% of UI and webhook API calls.

1772817588

Investigating

We are investigating an issue affecting a subset of customers experiencing errors when viewing webhook delivery histories and retrying webhook deliveries. UI and webhook API is impacted. Engineers have identified the cause and are actively working on mitigation.

1772817160

Investigating

We are investigating reports of degraded performance for Webhooks

1772816290