Elevated Error Rates for Log Queries and Monitors

Resolved

This incident has been resolved.

Update

Fix rollout has now been completed.

Update

The fix rollout is currently ongoing. Once completed we will confirm resolution.

Update

The fix rollout is currently ongoing. Once completed we will confirm resolution.

Update

We have successfully tested a fix for this issue and are currently deploying it to resolve this incident.

Update

We're still working on a fix for historical data impacted by this incident.

Update

We're still working on a fix for historical data impacted by this incident.

Update

We're still working on a fix for historical data impacted by this incident.

Update

We're still working on a fix for historical data impacted by this incident.

Update

We're still working on a fix for historical data impacted by this incident.

Update

We have deployed a fix and we are monitoring the results. We will provide another update once the issue is fully resolved.

At this time, newly ingested data is properly queryable, and monitors targeting Logs sent from 2023-10-03 20:40 UTC onwards are valid.

Queries targeting logs between 2023-10-02 11:40 UTC and 2023-10-03 20:40 UTC may return erroneous data. We are evaluating a fix that will restore query correctness for this time-window.

Update

We have identified the underlying issue and are working on a fix.

Investigating

We are continuing to investigate these issues, and will provide an update as soon as possible.

Investigating

We are actively investigating issues with Log Queries returning unexpected results. As a result of this issue, some users may experience issues querying logs on the web application or API, and with Logs based Monitors and Log-Based Metrics.