Missing Logs on fly-metrics.net

Resolved

This incident has been resolved.

Update

We have applied a fix and are now seeing logs flow through to user's fly-metrics dashboards.

We are containing to monitor the dashboard to ensure recovery.

Investigating

We are investigating reports of missing logs in customer's fly-metrics dashboards.

Only log search in the fly-metrics hosted Grafana instance is impacted. Checking logs using flyctl logs, the live logs page in the dashboard, or using a Fly-Log-Shipper app continues to work as expected.