Delayed Metrics

Resolved

This incident has been resolved.

Update

Almost all metrics have caught up aside from a small handful in sin and syd. We're continuing to monitor and expect these to complete in the next few hours.

Update

Backlogged metrics are still being processed. We're bringing extra processing capacity online to speed up the process.

Update

Backlogged metrics are still being processed and ingestion delays persist for some customers

Update

Backlogged metrics are still being processed and ingestion delays persist for some customers, but we’re continuing to see gradual improvement as the backlog continues to drain.

Update

Backlogged metrics data is still being processed, continue to monitor the situation while the system catches up.

Update

Metric ingestion is still delayed for some customers. We’re seeing gradual improvement and continue to monitor the situation while the system catches up.

Update

We are still in the process of increasing the metrics cluster throughput to catch up with metric backlog.

Update

We are continuing to see delayed metrics due to resource contention on a subset of metrics ingestion hosts, and we are working to rebalance ingestion traffic and reduce the backlog.

Update

We are continuing to see delayed metric exports from a number of hosts. Users will see delayed or missing metrics for machines on impacted hosts at this time.

Update

We've identified an issue on multiple hosts causing delayed metric ingestion into our hosted fly-metrics.net dashboards. We are working on a fix.

Investigating

We are investigating issues with customer facing metrics in the fly-metrics.net dashboard. Users may see delayed or missing metrics at this time.