Notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie are not delivered


Incident resolved in 6h5m38s

Resolved

Between approximately 9:42 AM and 10:38 AM ET on April 10, we observed delivery failures for customers in the US1, AP1, and AP2 regions. During this window, notifications to the following integrations were delayed or temporarily undelivered:

• PagerDuty • Slack • Microsoft Teams • Webhooks • Jira • ServiceNow • OpsGenie • VictorOps • BigPanda • Zendesk • Sumologic

All delayed notifications have been replayed with the following exceptions: PagerDuty pages and Slack notifications queued during the outage window were not replayed, as they were no longer actionable at the time of recovery. Webhook notifications will be reprocessed and delivered at a later date.

No monitoring data was lost during this incident. Monitor evaluations continued normally — only the delivery of triggered notifications was affected.

Thank you for your patience.

1775854789

Update

Webhooks and Slack notifications are still catching up.

1775852021

Update

Webhooks and Slack notifications are still catching up.

1775850264

Update

Webhooks and Slack notifications are still catching up.

1775847736

Update

Notifications for ServiceNow and Microsoft Teams are now caught up. Webhooks and Slack notifications are still catching up.

1775845992

Update

Notifications for JIRA are now caught up.

1775842665

Update

Notifications for Opsgenie, VictorOps, BigPanda, Zendesk, and Sumologic are caught up. Webhooks, Microsoft Teams, Slack, JIRA, and ServiceNow are still catching up.

1775841164

Update

We are still catching-up on past notifications that may have been delayed.

1775837754

Update

We are still catching-up on past notifications that may have been delayed.

1775836745

Update

New notifications are now successfully delivered, and we are catching-up on past notifications that may have been delayed

1775835767

Update

New notifications are now successfully delivered, and we are catching-up on past notifications that may have been delayed.

1775835747

Update

We are continuing to work on a fix for this issue.

1775834600

Update

The issue has been identified and a fix is being implemented.

1775833220

Investigating

We are investigating an issue to send notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie.

1775832851