All Systems Operational
Web App ? Operational
90 days ago
99.88 % uptime
Today
Mobile Apps ? Operational
90 days ago
99.79 % uptime
Today
Phone Call Routing ? Operational
90 days ago
100.0 % uptime
Today
Marketing Website ? Operational
90 days ago
100.0 % uptime
Today
Fax ? Operational
90 days ago
100.0 % uptime
Today
SMS Routing Operational
90 days ago
100.0 % uptime
Today
Video Calling ? Operational
90 days ago
99.88 % uptime
Today
Spruce app notifications ? Operational
90 days ago
100.0 % uptime
Today
Integrations ? Operational
90 days ago
100.0 % uptime
Today
API ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
Jan 30, 2023

No incidents reported today.

Jan 29, 2023

No incidents reported.

Jan 28, 2023

No incidents reported.

Jan 27, 2023

No incidents reported.

Jan 26, 2023

No incidents reported.

Jan 25, 2023
Postmortem - Read details
Jan 25, 17:46 PST
Resolved - The Spruce system experienced an outage from 6:45am PT to 9:45am PT on January 25, 2023. During this time:

- The Spruce Inbox failed to load for patients and providers
- Providers could not place voice or video calls to patients, given that the inbox would not load
- Providers could not send SMS or secure messages to patients
- Patients could not send secure messages to practices
- Inbound fax, SMS messages, and email arrived to the Spruce inbox in a delayed fashion
- Inbound calls were operational during this time; however, voicemails arrived to the Spruce inbox in a delayed fashion
- Workflow automation was executed, albeit in a delayed fashion

The outage was caused by CPU exhaustion on one of the core databases. The engineering team believes the CPU exhaustion to have been the result of a frequently run inefficient query that built up over time, and which was optimized as part of the fix for this incident. The engineering team will be proactively and closely monitoring the system in the next days to ensure that there is no sign of database CPU exhaustion, including especially during peak hours of the day.

We know how important it is for Spruce to be fully operational at all times. Working to build and maintain a medical communications system gives us all immense energy on a daily basis and is not a job we take lightly. We're very sorry for the outage and the impact to practices and patients. We will continue to work hard in pursuit of a highly available and reliable service. If you have any questions at all, please don't hesitate to reach us at support@sprucehealth.com.

Jan 25, 17:44 PST
Monitoring - The system should be fully functional at this point. We are continuing to monitor the system. We will post an incident report once we've had a chance to investigate more deeply here.
Jan 25, 10:05 PST
Identified - We are starting to see some recovery and are slowly ramping the system back up to fully serviceable to see the impact on database and CPU in general. We'll keep updating this page as we have more to share.
Jan 25, 09:45 PST
Update - We have made a database optimization for a high frequency lookup. We have intentionally brought down the API layer that clients connect to while we work to ensure that the rest of the system is functional. Once all asynchronous work has been completed, we will turn on the API layer slowly to ensure that we are not seeing any CPU performance issues again.
Jan 25, 09:21 PST
Update - We continue to investigate the issue with no root cause yet unfortunately. We are all hands on deck working to identify the reason for the outage.
Jan 25, 08:52 PST
Update - The backup system for notifying providers of incoming SMS, Fax, call events and voicemails has now been activated. Anyone that has registered contact information for our backup system will now get notified over email. You can read more about the backup system here: https://help.sprucehealth.com/article/424-spruce-backup-system
Jan 25, 08:20 PST
Update - We have not identified the root cause yet. The inbox continues to be down for most. We are actively investigating this issue.
Jan 25, 07:51 PST
Investigating - The Spruce inbox is unable to load at the moment and consequently patients and providers are unable to view/send messages, fax or make calls. Inbound calls should be working. Inbound fax likely delayed.
Jan 25, 07:17 PST
Jan 24, 2023

No incidents reported.

Jan 23, 2023

No incidents reported.

Jan 22, 2023

No incidents reported.

Jan 21, 2023

No incidents reported.

Jan 20, 2023

No incidents reported.

Jan 19, 2023

No incidents reported.

Jan 18, 2023

No incidents reported.

Jan 17, 2023

No incidents reported.

Jan 16, 2023

No incidents reported.