Spruce experienced a service incident from 6:04 pm to 7:55 pm PST on Wednesday, September 11, during which time customers may have experienced the following impacts:
There was no interruption to inbound calls, which remained fully functional throughout this time.
After assessing the severity of the incident, we triggered our backup communications system at 7:20pm PST, giving customers who have signed up for this service notifications of inbound communication during the incident.
As part of a regular deployment to our Spruce platform, we executed a database modification that ran much longer than anticipated. During this time, writes to the database stalled and then failed, causing delayed delivery of inbound communication. Eventually the buildup of stalls consumed all database resources, causing the Spruce inbox to become inaccessible.
Our engineering team was involved immediately. After assessment, we decided to allow the database modification to complete in order to safeguard data integrity. Upon this decision, the team took steps to accelerate the database modification operation. We also triggered our backup communications system.
As a result of this incident, we have identified the following action items:
In addition, we encourage every customer to subscribe to our backup communications system, which successfully delivered events to customers during this incident, even when the inbox was inaccessible. Please review more information here or contact Spruce Support from the app