Incident with Actions resulting in degraded performance


Incident resolved in 1h49m41s

Resolved

On January 7th, 2025 between 11:54:00 and 16:39 UTC, degraded performance was observed in Actions, Webhooks, and Issues, caused by an internal Certificate Authority configuration change that disrupted our event infrastructure. The configuration issue was promptly identified and resolved by rolling the change back on impacted hosts and re-issuing certificates.We have identified what services need updates to support the current PKI architecture and are working on implementing those changes to prevent a future recurrence.

1736267963

Investigating

Issues is operating normally.

1736267942

Investigating

Actions is operating normally.

1736267926

Investigating

Webhooks is operating normally.

1736267915

Investigating

Webhooks is experiencing degraded performance. We are continuing to investigate.

1736266186

Investigating

We have identified a configuration issue that we believe is the source of the Action workflow job delays and page latency increases. We are continuing to investigate and mitigate the issue.

1736265577

Investigating

Issues is experiencing degraded performance. We are continuing to investigate.

1736265115

Investigating

Users may see delays with Action workflow jobs in the UI and API responses. Additionally, several endpoints, including some Pull Request pages are experiencing increased latency. We are continuing to investigate the issue.

1736263028

Investigating

We are investigating reports of degraded performance for Actions

1736261382