Back to overview
Degraded

Database unresponsive

Dec 13 at 05:00am CET
Affected services
[US] Trace Ingestion

Resolved
Dec 13 at 12:53pm CET

Queued trace and project deletions were processed. We're fully back to normal operations.

Updated
Dec 13 at 05:24am CET

All events and API functionality have been fully restored and are operating as expected. We've temporarily disabled Trace and Project deletions to prevent performance issues with our Clickhouse database. While users can still initiate deletions through the UI, these requests will be queued and processed once we re-enable the deletion functionality.

Updated
Dec 13 at 05:13am CET

We identified the root cause and provided a fix. API latencies and error rates are improving. We will slowly speed up event processing again.

Created
Dec 13 at 05:00am CET

Since 04:45 GMT, we observe low responsiveness of our database. Incoming events are not processed and API calls which return tracing data (traces, observations, scores) fail or take longer.