This incident has been resolved.

Date of Incident: 2025-07-22
Time of Incident (UTC): 10:40 - 06:30
Services Affected: Admin Console Activity log and User Details page
Impact Duration: 43 hrs

Summary

A change to our internal data model that removed an unused type definition led to multiple failures in the Reports system.

Impact on Customers

• Events API and Reporting: The activity log widget failed to display activities.
• Admin console: User details page failed to display user details

What Happened?

A software update introduced an API change that caused a mismatch between client and server. The Audit log page in the admin console was still collecting data but was not able to render it.

• Timeline of Events (UTC):

  • 10:40 : Detected by 1Password personnel and root cause identified
  • 10:42: Issue Identified
  • 13:38 Fix created and testing initiated
  • 06:30: Fix released, Service fully restored

• Root Cause Analysis: A User State was removed from the 1Password.com server but not the client. This was a breaking API change.

How Was It Resolved?

• Resolution Steps: The user state in the 1Password client was removed to get the server and client back into parity.
• Verification of Resolution: 1Password engineering tested the changes and validated that full system functionality was restored.

What We Are Doing to Prevent Future Incidents

• User State change process: 1Password is investigating how to catch breaking API changes by implementing additional end-to-end tests in the CI pipeline.
• Audit log architecture change: Engineering is investigating a rework of the audit log page to change how it aggregates data so that it is not reliant on an endpoint. This would mitigate future occurrences that caused this issue.

Next Steps and Communication

• No action is required from our customers at this time.
• If you are still experiencing issues, please contact our support team at support@1password.com.

We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.

Sincerely,

The 1Password Team

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

Our 3rd party provider deployed a fix and we are no longer seeing delays in submission processing. This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

Incident Postmortem - Intermittent 500 error failures for US customers.

Date of Incident: 2025-06-16
Time of Incident (UTC): 14:39 UTC - 17:47 UTC
Service(s) Affected: Web Interface, Sign in, Sign up, Admin console, Item Sync, SSO (Single Sign On), Command Line Interface (CLI)) for US based users.

Impact Duration: 68 minutes

Summary

At 14:39 UTC, users in the US region experienced intermittent errors while accessing 1Password. The issue stemmed from resource constraints within our infrastructure, specifically affecting the networking services. This was resolved by scaling up the affected services.

Impact on Customers

During the duration of the incident:

• Web interface, Admin Console: Customers were able to log in but saw intermittent 500 errors, including “Failed to get Integrations” on the web Interface.
• SSO (Single Sign On), Command Line Interface (CLI), Item Sync: There was degraded performance for authentication and API requests.
• Sign in, Sign up: There were intermittent failures on sign in and sign up for some customers during the incident.
• Number of Affected Customers (approximate): All users accessing the service in the US region were affected.
• Geographic Regions Affected (if applicable): US

What Happened?

The incident began when our internal services started returning errors after deploying the latest version of the 1Password service. As part of the initial investigation, we restarted a supporting network service within our infrastructure, which resulted in an initial recovery of the affected service.

• Timeline of Events (UTC):

  • 2025-06-16 14:39 UTC: Incident Start - Automation detects servers are returning errors
  • 2025-06-16 14:40 UTC: Initial investigation begins
  • 2025-06-16 15:21 UTC: Networking updates are rolled out
  • 2025-06-16 15:23 UTC: Initial service recovery observed
  • 2025-06-16 15:38 UTC: Root cause identified: Networking applications ran out of allocated resources.
  • 2025-06-16 15:42 UTC: Additional capacity added to networking applications
  • 2025-06-17 17:47 UTC: The spike in server errors stopped, and internal monitoring showed that system health had returned to normal.
  • 2025-06-17 17:53 UTC: Incident resolved

• Root Cause Analysis: An internal service that directs network traffic became resource constrained which caused degraded performance of the service. We first stabilized the system by adding more capacity and have since deployed a permanent fix by increasing system resources to prevent a recurrence.

How Was It Resolved?

• Mitigation Steps: As an immediate mitigation, the number of replicas for the deployment was scaled up.
• Resolution Steps: A more permanent fix was later applied by increasing the allocated resources for the networking applications.
• Verification of Resolution: Around 15:25 UTC, we observed that the spike in 500 errors from the server had completely stopped. The team continued monitoring the errors and confirmed at 17:53 pm EST that allocated resource consumption had been stable for a while.

What We Are Doing to Prevent Future Incidents

• Scale existing resources: We have effectively scaled resources and resource limits to address additional load and will implement monitoring to ensure we do not hit critical limits
• Review and expand existing monitors: We will review our critical service monitors to improve alerting and catch future incidents earlier, before they have customer impact.

Next Steps and Communication

• No action is needed from customers

We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.

Sincerely,

The 1Password Team

We have resolved an issue where account administrators are unable to load the "People" view when logging in to their account on 1Password.com, 1Password.ca, and 1Password.eu.