Elevated errors across multiple services

Resolved

This incident has been resolved.

Update

All products have been completely recovered. Our on-call teams continue to monitor the situation.

Update

We are seeing recovery across all of our products, and Logs have resumed. Historical data in Logs and Speed Insights has been backfilled. Our on-call teams continue to monitor the situation regularly.

Update

Traffic has been rerouted from neighboring regions back to IAD1. Historical runtime Logs data is now being backfilled and will resume normal processing shortly.

Update

We are investigating intermittent errors on the Vercel dashboard.

Update

We have identified the root cause of the Logs and Log Drains issue and we are rolling out a fix.

Update

We are investigating errors affecting runtime Logs and Log Drains.

Update

Edge Config issues are now resolved.

Update

Domain Registrations issues are now resolved.

Update

A fix has been implemented for Edge Config, and we are monitoring the result.

Update

A fix has been implemented for Domain Registrations, and we are monitoring the result.

Update

We have identified Domain Registration failures and are working to restore service.

Update

We have identified degraded performance with Edge Config storage and are working to restore service.

Update

New deployments using Routing Middleware or Vercel Functions in the IAD1 region may fail to deploy. Traffic is still successfully being rerouted away from IAD1, which may result in increased latency. We are seeing full recovery of Edge Function and Middleware invocation errors in our IAD1 region. We are investigating an intermittent elevation of Cron Jobs not being executed. Historical data for Speed Insights and Web Analytics is currently being backfilled.

Update

New deployments using Routing Middleware or Vercel Functions in the IAD1 region may fail to deploy. Traffic is still successfully being rerouted away from IAD1, which may result in increased latency. We are seeing full recovery of Edge Function and Middleware invocation errors in our IAD1 region. We are investigating an intermittent elevation of Cron Jobs not being executed. Historical data for Speed Insights and Web Analytics is currently being backfilled.

Update

We are seeing recovery across Edge Functions and Middleware invocation errors in our IAD1 region. Historical data for Speed Insights and Web Analytics is currently being backfilled. Deployment creation for customers using Secure Compute or Static IPs are still impacted. Traffic is still successfully being rerouted away from iad1, which may result in increased latency.

Update

We are investigating an intermittent elevation of Edge Function and Middleware invocation errors in our IAD1 region. Impacted customers may encounter errors with the code INTERNAL_FUNCTION_INVOCATION_FAILED. Historical data for Speed Insights and Web Analytics is currently being backfilled. Deployment creation for customers using Secure Compute or Static IPs are still impacted. Traffic is still successfully being rerouted away from iad1, which may result in increased latency.

Update

Historical data for Speed Insights and Web Analytics is currently being backfilled. Deployment creation for customers not using Secure Compute or Static IPs has been fully restored. Traffic is still successfully being rerouted away from IAD1, which may result in increased latency.

Update

We are currently investigating reports of delayed data for Speed Insights and Web Analytics.

Update

We are currently investigating reports of delayed data for Speed Insights and Web Analytics.

Update

We are seeing recovery across deployment creation using Routing Middleware and Functions in IAD1.

CDN traffic in IAD1 continues to be rerouted as we work toward full recovery. During this period, users may experience increased latency.

Update

We are continuing to work on a fix for this issue.

Update

We are continuing to work on a fix for this issue.

Update

New deployments using Routing Middleware, or Vercel Functions with iad1 region may still be failing. We are continuing to investigate this issue.

Update

We are seeing recovery across services and CDN traffic is healthy.

Update

Multiple services are affected. We have re-routed CDN traffic from iad1 to a healthy region and are working on managing impact by component. Builds have also been rerouted.

Investigating

We are continuing to investigate this issue.

Investigating

We are continuing to investigate this issue.

Investigating

We are continuing to investigate this issue.

Investigating

We are continuing to investigate this issue.

Investigating

We are currently investigating this issue.