Incident Summary

Spruce experienced a service incident from 6:04 pm to 7:55 pm PST on Wednesday, September 11, during which time customers may have experienced the following impacts:

• Delayed delivery of inbound communication via SMS, fax, phone, and voicemail throughout this time.
• Disrupted access to the Spruce inbox, preventing outbound communication from 6:46 pm to 7:55 pm PST.

There was no interruption to inbound calls, which remained fully functional throughout this time.

After assessing the severity of the incident, we triggered our backup communications system at 7:20pm PST, giving customers who have signed up for this service notifications of inbound communication during the incident.

Analysis and Response

As part of a regular deployment to our Spruce platform, we executed a database modification that ran much longer than anticipated. During this time, writes to the database stalled and then failed, causing delayed delivery of inbound communication. Eventually the buildup of stalls consumed all database resources, causing the Spruce inbox to become inaccessible.

Our engineering team was involved immediately. After assessment, we decided to allow the database modification to complete in order to safeguard data integrity. Upon this decision, the team took steps to accelerate the database modification operation. We also triggered our backup communications system.

Action Items

As a result of this incident, we have identified the following action items:

• Adopt new protocol for executing database operations on large data models with high traffic patterns in order to prevent service disruptions at any hour of the day.
• Work with a database consultant to optimize the databases used across our platform.

In addition, we encourage every customer to subscribe to our backup communications system, which successfully delivered events to customers during this incident, even when the inbox was inaccessible. Please review more information here or contact Spruce Support from the app