The Spruce platform is fully operational now and all metrics are at a healthy level. Given that we migrated 3 core components last night (see here: https://status.sprucehealth.com/incidents/mkfdtztxbjsj) we will continue to monitor the system with increased sensitivity to ensure that it continues to function smoothly.
The message queue has completely drained and all messages that were backed up have been successfully delivered to inboxes. The overall latency for the platform seems to be at a health rate and we are continuing to monitor the situation to ensure it stays that way.
Posted Feb 28, 2019 - 10:33 PST
Monitoring
We have confirmed that lots of inbound messages were queued up due to the database over-utilization issue which prevented them from getting delivered to customers' Spruce Inboxes. The good news is that we see the message queues draining and messages getting successfully delivered to inboxes. We will keep monitoring the message queues to ensure there is no build up causing messages to arrive in a delayed fashion.
The connectivity issues to the Spruce Inbox that customers may have experienced should also have been resolved as a result of the scaling operation completing.
We're all hands on deck to ensure that Spruce is fully operational as soon as possible.
Posted Feb 28, 2019 - 10:15 PST
Update
The scaling operation just completed. The Spruce Inbox may have been unavailable to some users between 2019-02-28 10:06 PST and 2019-02-28 10:08 PST.
We are monitoring the situation now to see what impact the scale operation had on the overall latency and connectivity issues and will report back.
Posted Feb 28, 2019 - 10:10 PST
Update
Inbound messages, faxes and voicemails may arrive in the inbox in a delayed fashion due to the database over-utilization. The database is still undergoing the scaling operation and will update here as soon as the operation has completed.
Posted Feb 28, 2019 - 10:06 PST
Identified
Some of our customers are experiencing connectivity issues with the Spruce Inbox. Your inbox may be slow to load and messages may be showing up in your inbox in a delayed fashion.
We have identified the issue to related to the database size we are using for one of the components after the migration last night and are working to scale the database size as we speak.
Posted Feb 28, 2019 - 09:57 PST
This incident affected: Web App, Mobile Apps, Fax, and SMS Routing.