On 9/5/2023 at 8:47am MT, our team detected a system issue in one US environment. Some users in this environment experienced general slowness within the system. The cause of this degraded performance was several servers failed intermittently throughout the day. While the majority of servers remained active and performant, multiple times throughout the day a server would go offline. Affected users experienced slow response times for page loads, search, and general features.
We were able to put in emergency measures to stabilize. Full performance was regained by 1:11pm MT.
We are still actively investigating the root causes to put additional measures into place to prevent the issues from recurring.