Log Search Performance issues on Fly-Metrics.net

Resolved

This incident has been resolved.

Update

A fix has been implemented and we are monitoring the results.

Update

The issue has been identified and a fix is being implemented. You may see elevated search latency for your app logs.

Investigating

We are continuing to investigate this issue.

Investigating

We are continuing to investigate this issue.

Viewing logs via the Fly.io Dashboard live log viewer, flyctl logs, and the fly-log-shipper remain unimpacted by this.

Investigating

We are currently investigating issues with the log search in the fly-metrics.net grafana dashboard. Users may see log queries take longer than usual, or fail with a timeout error. Retries generally succeed.

Viewing logs via the fly.io/dashboard log viewer, flyctl logs, and the fly-log-shipper are not impacted by this.