This incident has been resolved. We traced this issue back to an errant query that was being amplified by a feedback loop between our mail system and a long running database query - an unexpected side effect triggered increased load on the database and caused requests to timeout and use up resources. We have removed the errant code and mail hooks, as well as implemented auto-scaling on our web processes to handle up to a 4x increase in load.
Posted Apr 03, 2022 - 02:33 UTC
Monitoring
A fix has been implemented and we are monitoring results
Posted Apr 03, 2022 - 01:01 UTC
Identified
We have identified the underlying issue, an errant unindexed query, and are working on a fix.
Posted Apr 02, 2022 - 23:29 UTC
Investigating
We have identified an issue in the past 24 hours that is causing intermittent 503 requests and timeouts on calls against our API. We are coordinating with our service provider in an attempt to limit these errors and increase capacity.