Previous incidents

The Prompts API responds with very low latency again.

All events are processed in time. We continue to scale our systems to serve increasing demand.

We have resolved the incident.

The issue has been resolved.

Issue has been resolved.

The issue has been resolved.

We're back to normal processing times.

Our error rates are back down and operations behave normally.

[Prod US] We scaled our web containers and no longer see elevated error rates across APIs.

We have resolved this issue

The queue is processing in time again.

Root cause identified and resolved: The issue was caused by an inefficient database query that was consuming excessive resources on our ClickHouse cluster.

We are now processing all events in time. We identified the root cause of the issue and continue to invest into performance improvements to ensure all data is processed in time.

We scaled our ingestion containers and reduced the amoint of observed timeouts and latencies. We continue to observe the situation. We count 19.95k requested which were timed out or were closed by the client.

We process all events within 60 seconds as of now.

We are fully recovered and process events within 60 seconds.

We increased our processing rate and have ingestion latencies of less than 60 seconds.

All traces that have been accepted prior to the full downtime (https://status.langfuse.com/incident/492749) at 7:06pm GMT have also been processed and are available via APIs and UI.

All APIs are recovered. All traces that have been accepted prior to the full downtime at 7:06pm GMT have also been processed and are available via APIs and UI.

We resolved all issues and are processing events at a high rate. We are now processing with a delay of 3 minutes and less.

The issue is resolved, trace processing is back to regular service levels and all outstanding traces that were accumulated have been processed.