Incident History

This incident has been resolved.

Performance has returned to normal levels following our mitigation.

A fix has been deployed, we've monitored the results and this incident is now resolved.

Date of Incident: 2025-05-21

Time of Incident (UTC) 16:06:40 - 16:48:10

Service(s) Affected: USA/Global 1Password.com website, Sign in, Sign up, Admin console, SSO (Single Sign On), Command Line Interface (CLI)).

Impact Duration: 41 minutes

Summary

On May 21st, 1Password's web interface, APIs, browser extension, and CLI tools experienced significant latency and errors. These problems stemmed from a code change that triggered a spike in server requests, leading to increased memory usage and system load. As a result, customers were unable to access their vaults or sign in via SSO.

This was not a result of a security incident and customer data was not affected.

Impact on Customers

During the duration of the incident:

• Web interface, Administration: Customers experienced significant delays when accessing the 1Password web interface. Administrators could not access or use any administration tools.
• Single Sign-on (SSO), Multi-factor Authentication (MFA): Users with SSO or MFA enabled could not sign in and received an "An unexpected error occurred" message. Customers may also have been required to re-authenticate to access 1Password once the issue was mitigated.
• Command Line Interface (CLI): CLI users faced increased latency and timeouts when attempting to access our web APIs.
• Browser Extension: Users requiring web interface authentication were unable to unlock their vaults.
• Number of Affected Users (approximate): All users accessing the service in the US/Global (1password.com) region were affected
• Geographic Regions Affected (if applicable): 1password.com (US/Global)

What Happened?

We deployed code changes that increased the number of queries to our Redis clusters. The increase in queries caused a spike in memory usage which in turn caused latency and errors across all endpoints.

• Timeline of Events (UTC):

  • 2025-05-21 15:52 UTC: Deployment started
  • 2025-05-21 15:57 UTC: Deployment complete
  • 2025-05-21 16:00 UTC: Automated monitoring detects increased errors and latency
  • 2025-05-21 16:01 UTC: Automation pages the incident response team
  • 2025-05-21 16:06 UTC: The team activates our incident protocol and begins investigation
  • 2025-05-21 16:21 UTC: The team initiates a rollback to a previous version
  • 2025-05-21 16:23 Code change causing the issue identified
  • 2025-05-21 16:48 UTC: Incident mitigated—rollback completed and we see a significant improvement in error rates and latency. The team continues to monitor the system.
  • 2025-05-21 17:23:11 UTC: Incident resolved

• Root Cause Analysis:

  We released a code change that caused a significant increase in data writes to our session store cluster.

  All operations, even those with a pre-established session depend on the session store for authenticating requests.

  The resulting resource contention led to increased latency and timeouts.

  The unplanned high volume of writes to this specific datastore also caused a portion sessions to be prematurely evicted, requiring customers to re-authenticate earlier than anticipated.

How Was It Resolved?

Our monitoring systems detected the issue and alerted the response team immediately after the release. The team quickly identified the problem and initiated a rollback.

• Resolution Steps: The team identified the problematic code change and reverted to a previous version. As the rollback deployed, server functionality returned to normal.
• Verification of Resolution: Our monitoring systems were closely observed for 2 hours after the rollback to ensure latency and errors were fully resolved.

What We Are Doing to Prevent Future Incidents

• Our team will implement longer testing periods in lower-traffic environments to improve monitoring and issue detection for similarly high-risk changes.
• Our team is working to improve our deployment process to enhance our incremental deployments, which will allow us to detect system issues earlier and contain fallout.

Next Steps and Communication

• Some customers may need to re-authenticate in order to access 1Password

We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.

Sincerely,

The 1Password Team

The 1Password extension is once again available on the Microsoft Edge Add-on store.

We are currently investigating this issue.

Date of Incident: 2025-04-22
Time of Incident (EST): 8:51am - 9:31am
Service(s) Affected: SSO, Web Sign In, Sign Up, Web Interface, CLI
Impact Duration: 40 minutes

Summary

On April 22, 2025 1Password’s web interface and APIs were unavailable for all customers accessing the US region. This was not a result of a security incident and customer data was not affected. The database stopped committing changes to disk, causing queries to fail. As a result, requests to the web interface and to APIs in the region returned with an error.

Impact on Customers

During the duration of the incident:

• Web interface, Administration: Customers were unable to log in to the web interface for 1Password. Users already logged in could not use the web application. Administrators were unable to use the administration tools. Users were presented with the error “upstream connect error or disconnect/reset before headers. reset reason: overflow.”
• Single Sign-on (SSO), Multi-factor Authentication (MFA): Users on accounts with SSO or MFA enabled were unable to sign in and were presented with the error above, or “An unexpected error occurred.”
• Command Line Interface (CLI): CLI users received the above errors when accessing our web APIs.
• Browser Extension: Users who needed to authenticate via the web interface were unable to unlock their vaults.

Scope

• All users accessing services in the US/Global (non-Canada and non-EU) region were affected.

What Happened?

A problem with database infrastructure prevented services from reading or writing data. There were no recent changes to the infrastructure, but additional load from multiple operations contributed to the issue.

• Timeline of Events:

  • 2025-04-22 08:51 AM ET: Database performance began to degrade.
  • 2025-04-22 08:54 AM ET: Automation detected the issue, and circuit broke requests to the database. The web app began returning errors. The system raised alerts.
  • 2025-04-22 08:58 AM ET: Incident team began investigating.
  • 2025-04-22 09:18 AM ET: Issue was identified.
  • 2025-04-22 09:25 AM ET: Database connections were reset to enable recovery.
  • 2025-04-22 09:31 AM ET: Monitors confirmed service was restored and database was operating normally.

• Root Cause Analysis: The database stopped writing data to its underlying data store. We are continuing to investigate the cause of this fault.

  • Update 2025-05-08: Our investigation concluded that a long-running query combined with a routine database migration blocked reads and writes to a table required for all subsequent queries. During the migration, an internal lock occurred that was not visible to our monitoring systems. This lock prevented database operations from proceeding until the initial long-running query was interrupted.

• Contributing Factors:

  • The incident happened during the peak of daily traffic.
  • Two subsystems were running database operations during or just before the incident.

How Was It Resolved?

• Mitigation Steps: We halted non-critical processes to reduce load on the database.
• Resolution Steps: All database connections were routed to a healthy replica. This restored service.
• Verification of Resolution: Monitoring systems were closely observed for 1 hour to ensure error rates returned to normal.

What We Are Doing to Prevent Future Incidents

• Improving monitors: We are updating our monitoring systems to better detect database issues like this before they can impact customers.

• Improve database performance: We are tuning queries and refactoring services to improve performance and reduce load on the database.

• Review database configuration: We are reviewing database size and configuration to optimize performance and enhance monitoring capabilities.

• Updates 2025-05-08

  • We tuned the database table involved to reduce the duration of the problematic query.
  • We updated the application running the query use a read replica instead of the primary database.
  • We modified the application to run the query during off-peak hours.

Next Steps

• No action is needed from customers

We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.

Sincerely,

The 1Password Team

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.