Previous incidents

We process all events within 60 seconds as of now.

We are fully recovered and process events within 60 seconds.

We increased our processing rate and have ingestion latencies of less than 60 seconds.

All traces that have been accepted prior to the full downtime (https://status.langfuse.com/incident/492749) at 7:06pm GMT have also been processed and are available via APIs and UI.

All APIs are recovered. All traces that have been accepted prior to the full downtime at 7:06pm GMT have also been processed and are available via APIs and UI.

We resolved all issues and are processing events at a high rate. We are now processing with a delay of 3 minutes and less.

The issue is resolved, trace processing is back to regular service levels and all outstanding traces that were accumulated have been processed.

All events are processed in time again.

Events are processed within a minute now.

Queued trace and project deletions were processed. We're fully back to normal operations.

We are fully recovered.

We rolled our back our change. The API behaves as expected now.

We processed all missing events which we lost during the outage.

Today, between 12:54 CET and 13:18 CET, we observed database connection issues in our asynchronous event processors in one of our hosting zones. Due to those issues, we stopped our workers which caused the previously reported delays. Before we scaled down the worker instances, we had a window of approximately 10 minutes in which events that were accepted on the API, were not processed by the worker and dropped with an error. This should affect about 1/3 of events that were send to our EU inst...