As of 12:55pm PT on October 25 2022, we started to see recovery and found the system to return to being fully operational. There was a re-allocation of data across the nodes that store and process the search engine data. The re-allocation seems to have helped the overall recovery.
We will continue to investigate ways in which we can further optimize our search engine database to prevent such spikes in search latency.
Posted Oct 25, 2022 - 12:55 PDT
Update
Note that while we are experiencing this performance issue, new contacts, updates to contacts and messages may not be reflected in search results. This is because there is a backup in indexing of new data into the search engine.
Also note that failures may be intermittent. Contact lists may fail to load but then if you refresh the page it may end up loading. This will also be the experience with conversation and contact search results.
Posted Oct 25, 2022 - 12:17 PDT
Investigating
The performance issues seem to have started again as of 11:02am PT today, despite the infrastructure improvements. We are continuing to investigate to identify an performance bottlenecks.
Posted Oct 25, 2022 - 11:17 PDT
Monitoring
We have made some underlying infrastructure changes that should help alleviate the performance issues that we were experiencing. We will continue to monitor the overall health of the database that backs searches and contact lists during normal business operations today (October 25 2022) to ensure that the issue is fully resolved and the platform is back to being functional.
Posted Oct 25, 2022 - 00:49 PDT
Investigating
We are currently investigating an issue where contact lists and search results fail to load intermittently.
There should be no impact to the Spruce inbox, inbound/outbound calls, video calls or exchanging of messages over SMS, secure messaging, email and fax.