Rare intermittent write failures in Tempo

Resolved

This incident has been resolved.

Update

Multiple issues were identified with our rollout process that caused instability on the write path. We have pushed out three fixes that drastically reduce the number of dropped writes and make all failed writes retryable. We will continue to closely monitor this issue and make improvements where necessary.

Update

We are continuing to diagnose and improve this situation. Fixes have been rolled out on Monday January 20th, and today, Wednesday January 22nd, to reduce drops during rollouts.

Update

The issue has been identified and a fix is being implemented.

Investigating

We are continuing to investigate this issue.

Investigating

Users may receive internet write failures on collectors stating: "Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp/cloud", "error": "rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway)"

The agent/collector will automatically retry the write attempt and succeed, so there is no loss in data collection.