Database Migration
Scheduled Maintenance Report for Spruce Health
Completed
The scheduled maintenance has been completed.
Posted Mar 02, 2019 - 02:00 PST
Update
Component #3 has been migrated. The Spruce inbox may have been unavailable to some users between 01:42:41 PST and 01:43:45 PST (~1m 4s). Inbound communications (calls and SMS texts) were not impacted.

And with that, all planned migrations have been completed! Our engineering team is monitoring the system to ensure that Spruce it is fully operational before considering this maintenance complete.
Posted Mar 02, 2019 - 01:53 PST
Update
Beginning migration of component #3.
Posted Mar 02, 2019 - 01:43 PST
Update
Component #2 has been migrated. Inbound communications (calls and SMS texts) may have been unavailable between 01:25:28 PST and 01:26:37 (~1m 9s). If there were any call events that failed during this brief window, we will report them directly to you. We will also deliver any delayed SMS texts directly to you. The Spruce Inbox was not impacted during this time period.

Our engineering team will continue to monitor the system to ensure that Spruce is fully operational before moving on to the next migration.
Posted Mar 02, 2019 - 01:38 PST
Update
Beginning migration of component #2.
Posted Mar 02, 2019 - 01:16 PST
Update
Component #1 has been migrated. The Spruce inbox may have been unavailable to some users between 12:38:50 AM PST and 12:40:39 AM PST (~1m 49s). No inbound communication (calls or SMS texts) were impacted.

Our engineering team will continue to monitor the system to ensure that Spruce is fully operational before moving on to the next migration.
Posted Mar 02, 2019 - 00:50 PST
In progress
The migration process has begun. We expect to migrate three components today. Our engineering team will keep this page up to date as we have updates to share.
Posted Mar 02, 2019 - 00:00 PST
Update
-
Posted Feb 28, 2019 - 02:01 PST
Update
-
Posted Feb 28, 2019 - 02:01 PST
Scheduled
In an effort to improve the overall stability of the platform, we have been working on separating database servers for each of the core components that make up the Spruce platform. This will help ensure that the entire platform does not become unavailable if the database server is having an issue, like it did during the incident on Jan 14 (https://status.sprucehealth.com/incidents/b9nhkvv3hrhk). It will also help us scale the database for each component independent of the others, as well as scale the platform over time.

In preparation for this, we have already migrated three components that have no critical role in powering the Spruce inbox or the inbound communications flow. Now that those migrations have gone smoothly and we’re feeling well prepared, we’d like to focus on the core components that the Spruce inbox and inbound communications depend on.

Our most recent 14-day analysis of communications across our customer base indicates that the 3-5 am EST window is the best time period during which to conduct this migration. So, we will be conducting the migration across two time windows: Thursday, Feb 28, 3-5 am EST, and Sunday, March 3, 3-5 am EST. We will try our best to complete the migrations in a single window on Feb 28 to reduce the potential impact on your business. We will be updating this page on the progress of the migration, so feel free to subscribe to the page for updates. This page will persist after the incident is complete, so you can review them after the fact as well.

During the maintenance windows, the Spruce inbox and inbound communications (calls and SMS texting) may become unavailable for up to one minute at a time no more than four times during each window (we plan to migrate two databases in each window, but if all goes smoothly, we will try to migrate all four databases in a single window). We will update this page with exact periods of unavailability so that you can stay informed.

If any of your patients call during one of the brief outage periods, they will hear a message that informs them that the system is undergoing scheduled maintenance and that asks them to try again in a few minutes, or to dial 911 if it’s an emergency. If any of your patients sends you an SMS text during an outage period, the message will be delivered to your inbox once the system has been restored and we have identified the delayed inbound messages, which we project to take no more than 30 minutes after system restoration. Please note that informing you of the delayed calls and SMS texts will be a manual process, but we will do our absolute best to deliver them to you as soon as is feasible.

If we are unable to complete any of the database migrations, we will abort the migration attempt and continue using the current database server. In this worst-case scenario, the total period of unavailability will be less than five minutes.


I know this is a lot of information, so please let us know if you have any questions at all by emailing us at support@sprucehealth.com, and I’ll be happy to answer them.
Posted Feb 25, 2019 - 12:52 PST