Our error rates are back down and operations behave normally.

Together with the ClickHouse team we developed two fix candidates that are currently being tested. We see a reduction in the total error rate across multiple endpoints and will continue to observe the situation.

The incident is still ongoing and we're working on a resolution with the ClickHouse Cloud team. Overall, we see a total of 1-2% of tracing related List and Get calls fail with a higher impact on queries that span longer timeframes.

We're observing elevated error rates for reads in the application and the API. Prompt management, authentication, and ingestion are not affected. Our team is investigating the situation.