Setting secrets and creating apps is degraded

Resolved

This incident has been resolved.

Update

While the secret storage service was in a read-only state, app creation requests queued up, due to the retry logic and insufficient request concurrency limits in our GraphQL API. This prevented our GraphQL API from serving any other requests. We have scaled up the GraphQL API and are continuing to monitor the situation.

Update

A fix has been implemented and we are monitoring the results.

Update

An ongoing data migration in our secret storage service is causing degraded Machines API functionality.