Spruce Health Status

Mar 25, 2023

No incidents reported today.

Mar 24, 2023

No incidents reported.

Mar 23, 2023

No incidents reported.

Mar 22, 2023

No incidents reported.

Mar 21, 2023

No incidents reported.

Mar 20, 2023

Resolved - System is fully operational.

Summary: From 12:52 PT to 13:57 PT on March 20 2023, the following actions experienced degraded performance:
- Searching for contacts or conversations: may be giving errors or returning results in a delayed manner or returning stale results
- Creating conversations: maybe return errors or experiencing a lag when searching for contacts
- Sending bulk messages: may be experiencing a lag where stuck in Processing state for a while or just taking longer to complete
- Opening contact lists: may be returning errors or experiencing a lag in returning results
Mar 20, 16:08 PDT

Monitoring - System is mostly recovered and we will continue to monitor to ensure that we do not see degraded performance for any actions.

There are still some stale search results being returned for contact and conversation search, but other than that, the system is fully operational at the moment.
Mar 20, 14:11 PDT

Update - We are continuing to work on a fix for this issue.
Mar 20, 13:22 PDT

Identified - Users may be experiencing degraded performance for the following actions:
- Searching for contacts or conversations: may be giving errors or returning results in a delayed manner or returning stale results
- Creating conversations: maybe return errors or experiencing a lag when searching for contacts
- Sending bulk messages: may be experiencing a lag where stuck in Processing state for a while or just taking longer to complete
- Opening contact lists: may be returning errors or experiencing a lag in returning results

Phone calls, video calls, exchanging messages (fax, secure, email or SMS) is NOT impacted and operating normally.

We have identified the reason for the degraded performance and are actively working on resolving the issue.
Mar 20, 13:22 PDT

Mar 19, 2023

No incidents reported.

Mar 18, 2023

No incidents reported.

Mar 17, 2023

Database upgrades part 2

Completed - We have completed the database updates, and the system is fully operational based on our monitoring.

Thank you for your patience, and if you have any questions, please email us at support@sprucehealth.com or write into your Spruce Support thread in app.
Mar 17, 01:58 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 16, 23:00 PDT

Update - We will be undergoing scheduled maintenance during this time.
Mar 15, 19:09 PDT

Update - IMPORTANT NOTE: We changed the maintenance window from 7pm PT - 11pm PT to 11pm PT - 3am PT.

It looks like editing the maintenance window time did not trigger a notification to subscribers. So the purpose of this message is to trigger the notification to ensure that all subscribers are aware of the change in time.
Mar 15, 14:13 PDT

Update - We will be undergoing scheduled maintenance during this time.
Mar 15, 13:04 PDT

Scheduled - We will be upgrading our family of databases on March 15 and 16 between 11pm PT and 3am PT. We expect brief periods (likely 2-5 minutes) of outages as we upgrade the core databases during this window. We will be starting with non-critical databases, ensuring that all is good, and then moving over to the critical databases during this time. A member of our engineering team will be online and closely monitoring the system to ensure smooth functioning as we upgrade the series of databases.

During this maintenance window, users may be unable to access their inbox for brief 2-5 minute periods. Inbound/outbound calls may also be impacted during this time. We will do our best to keep these windows as short as possible.

While we likely do not need all 4 hours across 2 days for this, we are intentionally blocking off these times to ensure that we have sufficient time to do these upgrades slowly and safely.

We will update this page as we are making progress on the database updates.
Mar 15, 12:48 PDT

Mar 16, 2023

Spruce experiencing issues with contact and conversation search related activities.

Resolved - The system is fully operational as per our active monitoring over the last 10 hours.

Summary:
From 10:15am PT March 15 to 6:50pm PT March 15, the following actions on Spruce experienced degraded performance:
- Searching for conversations and contacts either took long or failed
- Contact filters frequently failed to load when clicked into
- When starting a new conversation, contact suggestions either took long or failed to load, making it challenging to start new conversations
- Bulk actions (messaging, tagging, deleting) took longer than expected to complete, but eventually completed
- Newly created contacts, conversations and messages during this time period were not searchable. The new items eventually became searchable 6:50pm PT onwards
- Successful searches for contacts and conversations may have brought up stale results, where an update to a contact or conversation was not reflected in search results. The updated items were eventually updated in the search to reflect their latest versions

There was no impact to calls, SMS, Fax, Secure Messaging, Email or Fax during this time.

We will post a postmortem to the incident soon.
Mar 16, 15:23 PDT

Update - The redistribution of data in the cluster is still in progress (note that this happens in the background with minimal impact to searching and indexing of new data). We have been closely monitoring the situation throughout the night. We also increased the capacity of the cluster to accommodate for the redistribution of data and to insure that we are in better shape for today. We have ~20% of redistribution remaining that we believe will have a long standing improvement to the overall performance.

The metrics so far are looking healthy with no signs pointing to poor performance or increased error rate.

We will report back here once the redistribution completes or if we see any signs pointing to degraded performance.
Mar 16, 07:58 PDT

Update - Indexing of data has now caught up such that successfully searches for any contacts, conversations and messages will bring up up to date results. We are continuing to work on better distributing the data across the cluster.

We are not experiencing poor performance or intermittent errors currently. This is likely due to the decreased overall traffic in the system given time of day. That being said, we continue to work on reducing the likelihood of this problem continuing into business day tomorrow.
Mar 15, 19:11 PDT

Update - We have identified a potential cause for the intermittent failures with the search cluster. We are going to work towards better distributing the data across the cluster so as to increase overall performance and reduce the error rate.

To recap, due to the errors throughout the day:
- Searching for conversations, messages or contacts may have failed
- Bulk messages may have taken longer to complete than usual
- Newly created contacts, conversations and messages may not have shown up when searching
- Updates to contacts may not have been searchable

We will continue to work through the evening to reindex the data so as to better distribute it across the cluster and keep this incident up to date as we make progress here.

We're really sorry for the inconvenience this is causing to your workflows.
Mar 15, 18:16 PDT

Update - We continue to investigate this issue. Note that some bulk message operations may take long to complete or may get stuck in a particular state given that the bulk message operations also face similar errors when querying contact lists.
Mar 15, 15:31 PDT

Update - We continue to work on the issue here to reduce the intermittent errors while searching for contacts or loading contact lists. Note that there is no impact to phone calls, SMS routing, loading of inbox, exchanging secure messages, or video calling. Bulk messages will continue to send during this time, albeit in a delayed fashion given that bulk messages work off of contact lists.

We will update here as we make progress against the performance issue here.
Mar 15, 13:29 PDT

Update - We are continuing to work on a fix for this issue.
Mar 15, 12:31 PDT

Update - We are continuing to work on a fix for this issue.
Mar 15, 12:27 PDT

Identified - We are investigating an issue with contact and conversation search related activities.
Mar 15, 10:15 PDT

Database upgrades

Completed - We have completed the database updates for the night. Everything went fairly smoothly and the system is full operational based on our monitoring.

We experienced two brief periods of outages:
-12:00 am PT - 12:06am PT: Practice links failed to load. Patient connections during this time were unsuccessful.
- 12:10 am PT - 12:14am PT: Media files in conversation view + media uploads failed to load. Voicemail greetings and greeting prompts continued to work for inbound calls.

We'll do another round of database updates tomorrow night for the ones that we did not get to today.

Thank you for your patience, and if you have any questions, please email us at support@sprucehealth.com or write into your Spruce Support thread in app.
Mar 16, 00:53 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 15, 23:00 PDT

Update - We will be undergoing scheduled maintenance during this time.
Mar 15, 19:09 PDT

Update - IMPORTANT NOTE: We changed the maintenance window from 7pm PT - 11pm PT to 11pm PT - 3am PT.

It looks like editing the maintenance window time did not trigger a notification to subscribers. So the purpose of this message is to trigger the notification to ensure that all subscribers are aware of the change in time.
Mar 15, 14:13 PDT

Update - We will be undergoing scheduled maintenance during this time.
Mar 15, 13:02 PDT

Update - We will be undergoing scheduled maintenance during this time.
Mar 15, 12:47 PDT

Scheduled - We will be upgrading our family of databases on March 15 and 16 between 11pm PT and 3am PT. We expect brief periods (likely 2-5 minutes) of outages as we upgrade the core databases during this window. We will be starting with non-critical databases, ensuring that all is good, and then moving over to the critical databases during this time. A member of our engineering team will be online and closely monitoring the system to ensure smooth functioning as we upgrade the series of databases.

During this maintenance window, users may be unable to access their inbox for brief 2-5 minute periods. Inbound/outbound calls may also be impacted during this time. We will do our best to keep these windows as short as possible.

While we likely do not need all 4 hours across 2 days for this, we are intentionally blocking off these times to ensure that we have sufficient time to do these upgrades slowly and safely.

We will update this page as we are making progress on the database updates.
Mar 15, 12:47 PDT

Mar 15, 2023

Mar 14, 2023

No incidents reported.

Mar 13, 2023

No incidents reported.

Mar 12, 2023

No incidents reported.

Mar 11, 2023

No incidents reported.

Related