High query latency for us-prod-east-0 hosted datasources

Resolved

This incident has been resolved. No further issues were seen since adjusting the backend configuration on Friday June 20th. (22:22 UTC)

The root cause has been identified as node CPU saturation, causing high latency on ingesters.

Update

We are aware of a few intermittent re-occurrences lasting 10-15 minutes. We've made some adjustments to the backend configuration and continue to monitor the situation.

Investigating

We are continuing to investigate this issue.

Investigating

Starting at 13:23 UTC and resolved at 13:36 UTC, some hosted Prometheus datasources experience high query latency, which could trigger false alarms or cause timeouts in the UI when querying. The root cause is being investigated.