Resolved
We resolved the underlying issue. A distributed cache roll out for ClickHouse caused the latency and error spikes. The roll out was reverted and we see normal API performance as of now.
Identified
We identified the issue and work on a fix momentarily.
Investigating
We scaled our systems but still see elevated errors and latencies. We continue to investigate the underlying issue.
Investigating
We observed that our ClickHouse based queries have elevated latencies and high error rates. We currently scale the database so resolve the situation.