Database unresponsive
Resolved
Dec 13 at 12:53pm CET
Queued trace and project deletions were processed. We're fully back to normal operations.
Affected services
[US] Trace Ingestion
Updated
Dec 13 at 05:24am CET
All events and API functionality have been fully restored and are operating as expected. We've temporarily disabled Trace and Project deletions to prevent performance issues with our Clickhouse database. While users can still initiate deletions through the UI, these requests will be queued and processed once we re-enable the deletion functionality.
Affected services
[US] Trace Ingestion
Updated
Dec 13 at 05:13am CET
We identified the root cause and provided a fix. API latencies and error rates are improving. We will slowly speed up event processing again.
Affected services
[US] Trace Ingestion
Created
Dec 13 at 05:00am CET
Since 04:45 GMT, we observe low responsiveness of our database. Incoming events are not processed and API calls which return tracing data (traces, observations, scores) fail or take longer.
Affected services
[US] Trace Ingestion