GRU Networking Issues
We observed networking issues affecting a subset of our hosts in GRU from 19:37 UTC through 19:59 UTC. This affected platform operations requiring leases, private networking, and internet connectivity to some endpoints.
Incident History
Network outage in AMS region
This incident has been resolved.
State database issues in GRU
This incident has been resolved.
Private Networking issues (6PN)
This incident has been resolved
Partial Sprites outage
This incident has been resolved.
Elevated private networking errors
This incident has been resolved.
Upstash unavailability in FRA region
Impact
From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region.
What happened
During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out.
What we're changing
- We will be adding tooling to return a replica to service safely after maintenance.
- We will be adding safety checks to our upgrade process.
- We will make the upgrade procedure more resilient to errors like this one.
IPv6 Networking Issues in DFW
This incident has been resolved.
Depot builder failures
This incident has been resolved.
Network issues in US West Coast
This incident has been resolved.