The system has fully recovered. Here's a summary of full customer impact:
- From 8:13am - 8:56am PT and 10:13am PT - 10:46am PT, notifications to smartphone apps may have been delayed such that if a provider or patient had an inbound message, you may have been notified of the updates in a delayed fashion.
- From 8:13am PT - 8:56am PT, inbox actions like archive, assign, and new conversation updates were slow to flow through to the Spruce inbox.
The issue in general was caused because of an unexpected spike in updating of threads across the platform for a brief period of time. The spike in updating of threads caused cascading effects on notifications, inbox update latency, and the replication lag to the secondary read-only replication instance.
We are still discussing ways to prevent a surge in updates such as this to result in such cascading effects, and will update this incident with a resolution when we have an action plan for how to prevent this issue from happening again.
We're so sorry for the inconvenience and hope to continue to work to improve the Spruce platform for patient and providers.
The Spruce Team
Posted May 29, 2020 - 14:48 PDT
Monitoring
We have resolved the issue and have confirmed that inbox actions are responding with expected levels of latency. We are monitoring the issue to ensure that the system is fully functional while also investigating what caused the replication lag in the first place on the read-only secondary instance.
Posted May 29, 2020 - 09:03 PDT
Identified
We have identified the issue and narrowed it down to a lag in replication of data from our primary instance to our read-only replication instance for the database that manages all conversations in the inbox. We are switching over all reads to the primary instance for now while we investigate the replication lag.
Posted May 29, 2020 - 08:51 PDT
Investigating
We are currently investigating an issue where inbox actions such as archive, assign or marking as read are taking longer than usual to complete. This issue has no impact to inbound/outbound calls, sms, fax or secure messages.