Previous incidents

We have resolved this issue

The queue is processing in time again.

Root cause identified and resolved: The issue was caused by an inefficient database query that was consuming excessive resources on our ClickHouse cluster.

We are now processing all events in time. We identified the root cause of the issue and continue to invest into performance improvements to ensure all data is processed in time.

We scaled our ingestion containers and reduced the amoint of observed timeouts and latencies. We continue to observe the situation. We count 19.95k requested which were timed out or were closed by the client.

We process all events within 60 seconds as of now.

We are fully recovered and process events within 60 seconds.

We increased our processing rate and have ingestion latencies of less than 60 seconds.

All APIs are recovered. All traces that have been accepted prior to the full downtime at 7:06pm GMT have also been processed and are available via APIs and UI.

All traces that have been accepted prior to the full downtime (https://status.langfuse.com/incident/492749) at 7:06pm GMT have also been processed and are available via APIs and UI.

We resolved all issues and are processing events at a high rate. We are now processing with a delay of 3 minutes and less.

The issue is resolved, trace processing is back to regular service levels and all outstanding traces that were accumulated have been processed.

All events are processed in time again.

Events are processed within a minute now.

Queued trace and project deletions were processed. We're fully back to normal operations.