Elevated error rate across multiple models

Resolved

This incident has been resolved.

Update

We have seen error rates broadly return to normal success levels across all models. We saw elevated error rates on requests to multiple models from 7:08am PT / 14:08 UTC through 8:33am PT / 15:33 UTC.

Update

We are continuing to monitor for any further issues.

Update

A fix has been implemented and we are monitoring the results.

Update

The issue has been identified and a fix is being implemented.

Investigating

We are currently investigating this issue.