Synthetics MultiHTTP checks failing in Bangalore and Newark probe locations
This incident has been resolved.
Incident History
Hosted services unavailable in us-central2
Engineering has released a fix and as of 10:30 UTC, customers should no longer experience hosted services issues. At this time, we are considering this issue resolved. No further updates.
Datasource API Server Outage
All clusters have been restarted and datasources should be fully operational.
Loki-prod Write and Read Issues
Starting at ~21:11 UTC, Loki had a brief outage. Users may have experienced degraded write path performance for approximately 10 minutes, and 2-3 minutes of total write and read outage. This has since been resolved.
Slow query responses
This incident has been resolved.
Confluent integration issues in all regions
This incident has been resolved.
Service Disruption in EU-West-0
This incident has been resolved.
Experiencing issues with new stack creation
This incident has been resolved.
Hosted Logs prod-ap-northeast-0 region full read path outage.
The incident is fully resolved now, root cause was determined and mitigated. The particular cell affected had a legacy config present, which caused a panic during the regular rollout. The cell was rolled back as a precaution and the mentioned config is now corrected.
Synthetic monitoring scripted and multihttp tests not executing (all public probes)
Between 10:45 UTC and 11:05 UTC we experienced an outage with scripted and multi-http test executions across all public probe locations. These two check types were not executing tests, so you may see gaps in the synthetics metrics and logs in that window. Protocol-level test executions (ping, dns, tcp, http, traceroute) were unaffected. As of 11:05 UTC the incident is resolved and all check types are executing normally.