Elevated Errors and Query Timeouts for D1 Databases and SQLite Durable Objects
Resolved
We have put in place a series of mitigations to prevent a reoccurrence of severe query timeouts for affected databases.
This includes improvements to our watchdog logic, updates to our internal metrics, and the pre-existing / ongoing (short & medium term) improvements to D1 & Durable Objects SQL performance in these cases.
Update
We are continuing to work on a fix for this issue.
Update
After monitoring today, we still see intermittent elevated errors and query timeouts. Multiple, in-parallel code changes are being implemented or rolling out as additional remediation.
Update
A code change was rolled out and we're monitoring throughout NAMER business hours for reduction in errors and timeouts. Additionally in parallel, we are rolling out more changes that should reduce the impact of these small fraction of errors and timeouts.
Update
One implemented fix is being rolled out across Cloudflare's network but has not yet reached locations running Durable Objects or D1. The release will progress over several more hours after which we will monitor throughout 2025-01-27 for the desired reduction in errors and timeouts.
Update
We are continuing to work on a fix for this issue. Observed errors target only a small number of Durable Objects and D1 databases (D1 errors are less than 0.1% of total requests).
Update
Investigation identified the issue is a recurrence of https://www.cloudflarestatus.com/incidents/kzvk0c2s5fy7, whose remediation did not fully address the elevated errors and query timeouts. We are actively implementing additional fixes to address the errors and timeouts.
Investigating
We are continuing to investigate this issue.
Investigating
A small number of users are reporting severe increases in query latency when querying D1 databases and SQLite-backed Durable Objects. This is resulting in timeouts and/or hard errors for those queries.
We are investigating the cause and potential fixes and will provide updates with our findings.