Incidents | Langfuse

[US] Delayed Ingestion Processing

Mon, 30 Jun 2025 19:41:00 -0000

We scaled our services and events are processed in time.

[US] Delayed Ingestion Processing

Mon, 30 Jun 2025 17:36:00 -0000

Since 07:36 pm CEST, we have delayed ingestion on the US data region. We currently have a delay of ~3 minutes. We are scaling out systems to allow for higher throughput.

Delayed ingestion processing

Tue, 24 Jun 2025 11:45:00 -0000

All systems are healthy again.

Delayed ingestion processing

Tue, 24 Jun 2025 10:32:00 -0000

Ingestion is currently delayed.

[US] Delayed Ingestion Processing

Tue, 24 Jun 2025 09:22:00 -0000

The issue has been resolved and we're processing events in real time again.

[US] Delayed Ingestion Processing

Tue, 24 Jun 2025 08:32:00 -0000

The delay is back to 5 minutes. We're actively working on a mitigation.

[US] Delayed Ingestion Processing

Tue, 24 Jun 2025 08:09:00 -0000

We still see a fluctuation around wait times, but reduced the average time to approximately 1 minute.

[US] Delayed Ingestion Processing

Tue, 24 Jun 2025 07:21:00 -0000

We are observing a delay around ingestion event processing of approximately 5min. We are investigating the situation.

[US] Ingestion is currently delayed.

Mon, 23 Jun 2025 23:01:00 -0000

All queues are processed in time.

[US] Ingestion is currently delayed.

Mon, 23 Jun 2025 22:30:00 -0000

Currently, event ingestion in the US dats region is delayed by 7 minutes. We are working on scaling our systems.

Delayed ingestion

Mon, 23 Jun 2025 14:13:00 -0000

All systems are healthy again.

Delayed ingestion

Mon, 23 Jun 2025 12:32:00 -0000

Ingestion is currently delayed.

[US] API latencies for prompts API

Tue, 10 Jun 2025 20:18:00 -0000

We have scaled our services to address latency issues. The p90 latency is now reduced from 10 seconds to 30 milliseconds.

[US] API latencies for prompts API

Tue, 10 Jun 2025 19:51:00 -0000

We experience higher than usual Prompt API latencies. We are investigating the root cause.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 09:03:00 -0000

All systems are healthy again and Google Auth issues are resolved.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 09:03:00 -0000

All systems are healthy again and Google Auth issues are resolved.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 09:03:00 -0000

All systems are healthy again and Google Auth issues are resolved.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 09:03:00 -0000

All systems are healthy again and Google Auth issues are resolved.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 08:59:00 -0000

API and tracing performance have recovered. Google Auth issues remain in both US and EU.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 08:59:00 -0000

API and tracing performance have recovered. Google Auth issues remain in both US and EU.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 08:59:00 -0000

API and tracing performance have recovered. Google Auth issues remain in both US and EU.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 08:59:00 -0000

API and tracing performance have recovered. Google Auth issues remain in both US and EU.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 08:10:00 -0000

We're currently observing failing Google Authentications across the EU and US environment and elevated error rates and latencies for API and frontend operations within the US environment. We're investigating the situation.

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 08:10:00 -0000

[EU + US] Failing Google Auth and Degraded API Performance

Tue, 20 May 2025 08:10:00 -0000

Elevated error rates across public APIs and UI

Tue, 06 May 2025 17:31:00 -0000

Issue has been resolved.

Elevated error rates across public APIs and UI

Tue, 06 May 2025 17:00:00 -0000

We are currently experiencing elevated error rates in both the Langfuse Cloud UI and on public API routes affecting traces, observations, and scores. Investigation is ongoing.

Delayed ingestion

Thu, 01 May 2025 15:21:00 -0000

All events are inserted in a minute or less.

Delayed ingestion

Thu, 01 May 2025 14:37:00 -0000

Ingestion is currently delayed in EU. Investigation is ongoing.

Delayed ingestion

Fri, 25 Apr 2025 12:20:00 -0000

Ingestion has recovered. All services work as expected.

Delayed ingestion

Fri, 25 Apr 2025 11:58:00 -0000

Ingestion is currently delayed in EU. Investigation ongoing.

Delayed ingestion

Thu, 24 Apr 2025 13:50:00 -0000

Ingestion is currently delayed in EU. Investigation is ongoing.

Delayed ingestion

Thu, 24 Apr 2025 12:06:00 -0000

Ingestion is currently delayed in EU region. Investigation is ongoing.

Delayed event ingestion

Wed, 23 Apr 2025 09:46:00 -0000

Ingestion delayed for around 10 minutes.

Langfuse interface unavailable for some users [workaround: refresh the page]

Tue, 15 Apr 2025 19:28:00 -0000

Fix deployed and issue resolved.

Langfuse interface unavailable for some users [workaround: refresh the page]

Tue, 15 Apr 2025 19:28:00 -0000

Fix deployed and issue resolved.

Langfuse interface unavailable for some users [workaround: refresh the page]

Tue, 15 Apr 2025 18:54:00 -0000

We have identified the issue and are currently deploying a fix. It will be available in 20-30 minutes across all regions. In the meantime, reloading the page should resolve the issue if you encounter it.

Langfuse interface unavailable for some users [workaround: refresh the page]

Tue, 15 Apr 2025 18:54:00 -0000

Langfuse interface unavailable for some users [workaround: refresh the page]

Tue, 15 Apr 2025 18:33:00 -0000

We got reports that the UI is unavailable for some users, we are investigating the situation.

Langfuse interface unavailable for some users [workaround: refresh the page]

Tue, 15 Apr 2025 18:33:00 -0000

We got reports that the UI is unavailable for some users, we are investigating the situation.

High prompt API latencies

Mon, 24 Mar 2025 22:15:00 -0000

The Prompts API responds with very low latency again.

High prompt API latencies

Mon, 24 Mar 2025 21:07:00 -0000

Since 10:07 pm CET, we observe abnormally high delays on our Prompts API due to delays in the event loop of our Node containers.

[US] Delayed event ingestion

Thu, 20 Mar 2025 22:50:00 -0000

All events are processed in time. We continue to scale our systems to serve increasing demand.

[US] Delayed event ingestion

Thu, 20 Mar 2025 21:04:00 -0000

We're currently observing around 20 min delay in processing newly ingested events for the US environment. We're investigating the situation.

[US] Delayed event ingestion

Thu, 20 Mar 2025 20:54:00 -0000

We're currently observing around 12 min delay in processing newly ingested events for the US environment. We're investigating the situation.

[US] Trace Ingestion Delayed

Wed, 19 Mar 2025 18:00:00 -0000

We have resolved the incident.

[US] Trace Ingestion Delayed

Wed, 19 Mar 2025 17:42:00 -0000

We have scaled workers, wait time reduced to approx. 2mins.

[US] Trace Ingestion Delayed

Wed, 19 Mar 2025 17:30:00 -0000

We're currently observing around 4min delay in processing newly ingested events for the US environment. We're investigating the situation.

[EU] Trace Ingestion Delay

Wed, 19 Mar 2025 11:53:00 -0000

The issue has been resolved.

[EU] Trace Ingestion Delay

Wed, 19 Mar 2025 11:49:00 -0000

We're currently observing around 5min delay in processing newly ingested events for the EU environment. Situation is recovering.

[EU] Trace Ingestion Delay

Wed, 19 Mar 2025 11:34:00 -0000

We're currently observing around 14min delay in processing newly ingested events for the EU environment. We're investigating the situation.

[EU] Trace Ingestion Delay

Wed, 19 Mar 2025 11:29:00 -0000

We're currently observing around 9min delay in processing newly ingested events for the EU environment. We're investigating the situation.

[EU] Trace Ingestion Delay

Wed, 19 Mar 2025 11:24:00 -0000

We're currently observing around 6min delay in processing newly ingested events for the EU environment. We're investigating the situation.

[EU] Delayed Trace Ingestion

Wed, 19 Mar 2025 11:05:00 -0000

Issue has been resolved.

[EU] Delayed Trace Ingestion

Wed, 19 Mar 2025 10:59:00 -0000

We're currently observing around 6min delay in processing newly ingested events for the EU environment. We're investigating the situation.

[EU] Delayed Trace Ingestion

Tue, 18 Mar 2025 16:09:00 -0000

The issue has been resolved.

[EU] Delayed Trace Ingestion

Tue, 18 Mar 2025 15:21:00 -0000

We're currently observing around 7min delay in processing newly ingested events for the EU environment. We're investigating the situation.

[US] Delayed trace ingestion

Tue, 11 Mar 2025 14:17:00 -0000

We're back to normal processing times.

[US] Delayed trace ingestion

Tue, 11 Mar 2025 14:11:00 -0000

We made changes to our processing and reduced the processing delay to approximately 3 minutes.

[US] Delayed trace ingestion

Tue, 11 Mar 2025 13:16:00 -0000

We are currently investigating a delayed processing of newly ingested traces on the US data region. Current delay: 10 minutes

[EU & US] Elevated Error Rates Across Tracing APIs

Tue, 25 Feb 2025 17:39:00 -0000

Our error rates are back down and operations behave normally.

[EU & US] Elevated Error Rates Across Tracing APIs

Tue, 25 Feb 2025 17:39:00 -0000

Our error rates are back down and operations behave normally.

[EU & US] Elevated Error Rates Across Tracing APIs

Tue, 25 Feb 2025 17:26:00 -0000

Together with the ClickHouse team we developed two fix candidates that are currently being tested. We see a reduction in the total error rate across multiple endpoints and will continue to observe the situation.

[EU & US] Elevated Error Rates Across Tracing APIs

Tue, 25 Feb 2025 17:26:00 -0000

[EU & US] Elevated Error Rates Across Tracing APIs

Tue, 25 Feb 2025 10:25:00 -0000

The incident is still ongoing and we're working on a resolution with the ClickHouse Cloud team. Overall, we see a total of 1-2% of tracing related List and Get calls fail with a higher impact on queries that span longer timeframes.

[EU & US] Elevated Error Rates Across Tracing APIs

Tue, 25 Feb 2025 10:25:00 -0000

[EU & US] Elevated Error Rates Across Tracing APIs

Mon, 24 Feb 2025 19:26:00 -0000

We're observing elevated error rates for reads in the application and the API. Prompt management, authentication, and ingestion are not affected. Our team is investigating the situation.

[EU & US] Elevated Error Rates Across Tracing APIs

Mon, 24 Feb 2025 19:26:00 -0000

We're observing elevated error rates for reads in the application and the API. Prompt management, authentication, and ingestion are not affected. Our team is investigating the situation.

[US data region] Elevated error rates across APIs

Thu, 20 Feb 2025 16:21:00 -0000

[Prod US] We scaled our web containers and no longer see elevated error rates across APIs.

[US data region] Elevated error rates across APIs

Thu, 20 Feb 2025 16:21:00 -0000

[Prod US] We scaled our web containers and no longer see elevated error rates across APIs.

[US data region] Elevated error rates across APIs

Thu, 20 Feb 2025 16:21:00 -0000

[Prod US] We scaled our web containers and no longer see elevated error rates across APIs.

[US data region] Elevated error rates across APIs

Thu, 20 Feb 2025 15:33:00 -0000

We see elevated error rates across APIs. This is a small share of requests (<1%), we are actively working on identifying the root cause. Prompt Management and Tracing are robust to these transient errors as they retry/cache requests on errors.

[US data region] Elevated error rates across APIs

Thu, 20 Feb 2025 15:33:00 -0000

[US data region] Elevated error rates across APIs

Thu, 20 Feb 2025 15:33:00 -0000

[US] Tracing ingestion delayed

Fri, 31 Jan 2025 15:06:00 -0000

We have resolved this issue

[US] Tracing ingestion delayed

Fri, 31 Jan 2025 14:33:00 -0000

Due to a drastic surge in trace volume we currently observe a 7 minute delay on the trace processing.

Delayed event ingestion in US data region

Thu, 30 Jan 2025 16:23:00 -0000

The queue is processing in time again.

Delayed event ingestion in US data region

Thu, 30 Jan 2025 15:34:00 -0000

We're currently observing around 10 minutes of delay for event ingestion in the US data region.

Slow UI and failing API endpoints

Thu, 23 Jan 2025 23:31:00 -0000

Root cause identified and resolved: The issue was caused by an inefficient database query that was consuming excessive resources on our ClickHouse cluster.

Slow UI and failing API endpoints

Thu, 23 Jan 2025 23:13:00 -0000

We currently experience a slow UI and partially failing APIs to fetch data from Langfuse. We are working on finding the root cause.

Delayed event ingestion in US data region

Wed, 22 Jan 2025 10:47:00 -0000

We are now processing all events in time. We identified the root cause of the issue and continue to invest into performance improvements to ensure all data is processed in time.

Delayed event ingestion in US data region

Tue, 21 Jan 2025 08:44:00 -0000

We're currently observing around 10 minutes of delay for event ingestion in the US data region.

High latencies and timeouts on ingestion API

Mon, 20 Jan 2025 02:35:00 -0000

We scaled our ingestion containers and reduced the amoint of observed timeouts and latencies. We continue to observe the situation. We count 19.95k requested which were timed out or were closed by the client.

High latencies and timeouts on ingestion API

Mon, 20 Jan 2025 02:21:00 -0000

We are observing high latencies on our ingestion API. We are investigating the issue and will try to resolve the issue.

Delayed event ingestion in US data region

Fri, 17 Jan 2025 18:06:00 -0000

We process all events within 60 seconds as of now.

Delayed event ingestion in US data region

Fri, 17 Jan 2025 17:54:00 -0000

The delay is down to approximately 10 min.

Delayed event ingestion in US data region

Fri, 17 Jan 2025 17:47:00 -0000

The delay is down to approximately 15min.

Delayed event ingestion in US data region

Fri, 17 Jan 2025 17:32:00 -0000

We continue to scale our infrastructure. Delay is at 25 minutes right now. We persist all events which are entering our infrastructure. We will process all events eventually.

Delayed event ingestion in US data region

Fri, 17 Jan 2025 17:04:00 -0000

We're currently observing around 10 minutes of delay for event ingestion in the US data region.

Delayed event ingestion in US data region

Wed, 15 Jan 2025 19:22:00 -0000

We are fully recovered and process events within 60 seconds.

Delayed event ingestion in US data region

Wed, 15 Jan 2025 18:10:00 -0000

We observe delayed event ingestion. Events are visible in the UI and available on GET APIs with a delay of up to 7 minutes. We are investigating the issue and scaling our services to process events faster.

Delayed event processing in EU data region

Mon, 13 Jan 2025 17:44:00 -0000

We increased our processing rate and have ingestion latencies of less than 60 seconds.

Delayed event processing in EU data region

Mon, 13 Jan 2025 17:00:00 -0000

Since 17:00 UTC, we observe delayed event ingestion. Events are visible in the UI and available on GET APIs with a delay of up to 7 minutes. We are investigating the issue and scaling our services to process events faster.

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:17:00 -0000

All APIs are recovered. All traces that have been accepted prior to the full downtime at 7:06pm GMT have also been processed and are available via APIs and UI.

[US] Delayed tracing processing and temporary downtime

Wed, 08 Jan 2025 19:17:00 -0000

All traces that have been accepted prior to the full downtime (https://status.langfuse.com/incident/492749) at 7:06pm GMT have also been processed and are available via APIs and UI.

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:17:00 -0000

All APIs are recovered. All traces that have been accepted prior to the full downtime at 7:06pm GMT have also been processed and are available via APIs and UI.

[US] Delayed tracing processing and temporary downtime

Wed, 08 Jan 2025 19:17:00 -0000

All traces that have been accepted prior to the full downtime (https://status.langfuse.com/incident/492749) at 7:06pm GMT have also been processed and are available via APIs and UI.

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:17:00 -0000

All APIs are recovered. All traces that have been accepted prior to the full downtime at 7:06pm GMT have also been processed and are available via APIs and UI.

[US] Delayed tracing processing and temporary downtime

Wed, 08 Jan 2025 19:17:00 -0000

All traces that have been accepted prior to the full downtime (https://status.langfuse.com/incident/492749) at 7:06pm GMT have also been processed and are available via APIs and UI.

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:13:00 -0000

Langfuse UI and prompts APIs are available again, we are recovering the trace ingestion endpoint.

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:13:00 -0000

Langfuse UI and prompts APIs are available again, we are recovering the trace ingestion endpoint.

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:13:00 -0000

Langfuse UI and prompts APIs are available again, we are recovering the trace ingestion endpoint.

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:06:00 -0000

A configuration change made for to resolve another issue (https://status.langfuse.com/incident/492708) led to downtime across services in the US region. We are working on fixing the issue while recovering all services.

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:06:00 -0000

[US] Downtime across all APIs

Wed, 08 Jan 2025 19:06:00 -0000

[US] Delayed tracing processing and temporary downtime

Wed, 08 Jan 2025 17:43:00 -0000

Our Clickhouse database is currently not able to process incoming events at the desired speed. We observe up to 10 minutes of delayed event ingestion and work on scaling the database and our containers for faster ingestion.

Delayed ingestion processing

Tue, 07 Jan 2025 16:15:00 -0000

We resolved all issues and are processing events at a high rate. We are now processing with a delay of 3 minutes and less.

Delayed ingestion processing

Tue, 07 Jan 2025 03:30:00 -0000

Our Clickhouse database is currently not able to process incoming events at the desired speed. We observe up to 15 minutes of delayed event ingestion and work on scaling the database and our containers for faster ingestion.

Delayed trace ingestion

Thu, 02 Jan 2025 18:14:00 -0000

The issue is resolved, trace processing is back to regular service levels and all outstanding traces that were accumulated have been processed.

Delayed trace ingestion

Thu, 02 Jan 2025 18:06:00 -0000

At the current rates, we expect to be fully caught up with ingested traces in the next 15 minutes.

Delayed trace ingestion

Thu, 02 Jan 2025 18:05:00 -0000

After halting trace deletion operations, we have now disabled deletion across the UI on Langfuse Cloud in order to make the UI more predictable. We will work on improving the performance of deletions before enabling it again.

Delayed trace ingestion

Thu, 02 Jan 2025 17:18:00 -0000

We are halting processing of trace deletions made via the UI as they contribute to this issue. We are further investigating the problem.

Delayed trace ingestion

Thu, 02 Jan 2025 16:52:00 -0000

We have identified the issue and are now catching up with the delayed trace processing.

Delayed trace ingestion

Thu, 02 Jan 2025 16:40:00 -0000

Currently trace ingestion is delayed on Langfuse EU. We are investigating this issue.

[US] Delayed trace ingestion and increased levels of frontend errors

Wed, 18 Dec 2024 11:25:00 -0000

All events are processed in time again.

[US] Delayed trace ingestion and increased levels of frontend errors

Wed, 18 Dec 2024 11:25:00 -0000

All events are processed in time again.

[US] Delayed trace ingestion and increased levels of frontend errors

Wed, 18 Dec 2024 11:13:00 -0000

Wait time is at 15 minutes right now and we see a downwards trend.

[US] Delayed trace ingestion and increased levels of frontend errors

Wed, 18 Dec 2024 11:13:00 -0000

Wait time is at 15 minutes right now and we see a downwards trend.

[US] Delayed trace ingestion and increased levels of frontend errors

Wed, 18 Dec 2024 11:00:00 -0000

We currently observe delayed trace ingestion and elevated errors by the Langfuse UI. We are investigating the issue. No events are lost and they will be processed with delay of roughly 12 minutes.

[US] Delayed trace ingestion and increased levels of frontend errors

Wed, 18 Dec 2024 11:00:00 -0000

We currently observe delayed trace ingestion and elevated errors by the Langfuse UI. We are investigating the issue. No events are lost and they will be processed with delay of roughly 12 minutes.

Delayed event processing

Wed, 18 Dec 2024 05:15:00 -0000

Events are processed within a minute now.

Delayed event processing

Wed, 18 Dec 2024 05:06:00 -0000

Events are still delayed. The delay is at 3 minutes at the moment. We continue observing the situation.

Delayed event processing

Wed, 18 Dec 2024 01:36:00 -0000

In the US data region, we observe delayed event processing since 00:45 UTC. In the worst case a delay (event visible in the UI - event arrived at our ingestion API) can take up to 8 minutes. We are working on scaling the system.

Database unresponsive

Fri, 13 Dec 2024 11:53:00 -0000

Queued trace and project deletions were processed. We're fully back to normal operations.

Database unresponsive

Fri, 13 Dec 2024 04:24:00 -0000

All events and API functionality have been fully restored and are operating as expected. We've temporarily disabled Trace and Project deletions to prevent performance issues with our Clickhouse database. While users can still initiate deletions through the UI, these requests will be queued and processed once we re-enable the deletion functionality.

Database unresponsive

Fri, 13 Dec 2024 04:13:00 -0000

We identified the root cause and provided a fix. API latencies and error rates are improving. We will slowly speed up event processing again.

Database unresponsive

Fri, 13 Dec 2024 04:00:00 -0000

Since 04:45 GMT, we observe low responsiveness of our database. Incoming events are not processed and API calls which return tracing data (traces, observations, scores) fail or take longer.

Delayed event processing and API errors

Thu, 28 Nov 2024 15:33:00 -0000

We are fully recovered.

Delayed event processing and API errors

Thu, 28 Nov 2024 15:21:00 -0000

The UI is working as expected. We are currently processing all delayed events. We will be caught up shortly.

Delayed event processing and API errors

Thu, 28 Nov 2024 15:11:00 -0000

We are currently experiencing very high load on our Clickhouse cluster which causes CPU and Memory issues. We are scaling the database momentarily.

Our scores API (api/public/scores) returned 500 due to performance issues.

Mon, 25 Nov 2024 10:52:00 -0000

We rolled our back our change. The API behaves as expected now.

Our scores API (api/public/scores) returned 500 due to performance issues.

Mon, 25 Nov 2024 10:52:00 -0000

We rolled our back our change. The API behaves as expected now.

Our scores API (api/public/scores) returned 500 due to performance issues.

Mon, 25 Nov 2024 09:30:00 -0000

Due to performance issues on our scores API (api/public/scores), we returned 500 errors in some cases. We changed the behavior of that API over the weekend. We rolled that change back and are working on a fix.

Our scores API (api/public/scores) returned 500 due to performance issues.

Mon, 25 Nov 2024 09:30:00 -0000

Event ingestion delayed on EU instance, affects GET API and UI

Tue, 19 Nov 2024 21:33:00 -0000

We processed all missing events which we lost during the outage.

Event ingestion delayed on EU instance, affects GET API and UI

Tue, 19 Nov 2024 18:25:00 -0000

APIs behave as expected as of now. We are recovering events between (UTC 18:17:00 - UTC 18:25:00) which are not visible in the UI or for the REST API.

Event ingestion delayed on EU instance, affects GET API and UI

Tue, 19 Nov 2024 18:24:00 -0000

Due to a peak in resource usage, event ingestion delayed on EU instance This affects GET API and UI. We are investigating the issue.

Delayed event processing

Tue, 05 Nov 2024 14:54:00 -0000

Today, between 12:54 CET and 13:18 CET, we observed database connection issues in our asynchronous event processors in one of our hosting zones. Due to those issues, we stopped our workers which caused the previously reported delays. Before we scaled down the worker instances, we had a window of approximately 10 minutes in which events that were accepted on the API, were not processed by the worker and dropped with an error. This should affect about 1/3 of events that were send to our EU instance within that timeframe. We added additional error handling to record and store failed events which allows us to replay them in the future instead of dropping them on errors. We apologize for any inconvenience.

Delayed event processing

Tue, 05 Nov 2024 12:38:00 -0000

We found the root cause and provided a fix. Everything should be working as expected now.

Delayed event processing

Tue, 05 Nov 2024 12:00:00 -0000

We identified an issue with our database and observe delayed event processing. We are working on finding the root cause and providing a fix.

UI degraded, some routes error, backend API routes unaffected

Thu, 31 Oct 2024 17:59:00 -0000

We identified the issue and resolved it.

UI degraded, some routes error, backend API routes unaffected

Thu, 31 Oct 2024 17:59:00 -0000

We identified the issue and resolved it.

UI degraded, some routes error, backend API routes unaffected

Thu, 31 Oct 2024 17:49:00 -0000

Some tables in the UI are degraded and do not load, we are working on a fix.

UI degraded, some routes error, backend API routes unaffected

Thu, 31 Oct 2024 17:49:00 -0000

Some tables in the UI are degraded and do not load, we are working on a fix.

Delayed data processing

Mon, 14 Oct 2024 19:42:00 -0000

All historic data is processed.

Delayed data processing

Mon, 14 Oct 2024 19:42:00 -0000

All historic data is processed.

Delayed data processing

Mon, 14 Oct 2024 19:41:00 -0000

We fixed the bug and process events again. Processing will be delayed for EU. The US environment is processing as expected. Events between 09:34 GTM and 09:40 GMT will be processed later.

Delayed data processing

Mon, 14 Oct 2024 19:41:00 -0000

We fixed the bug and process events again. Processing will be delayed for EU. The US environment is processing as expected. Events between 09:34 GTM and 09:40 GMT will be processed later.

Delayed data processing

Mon, 14 Oct 2024 19:40:00 -0000

We fully caught up all queues and process events at the usual latencies. Some events from the time between 09:34 GMT and 09:40 GMT failed to process. We are working on recovering them.

Delayed data processing

Mon, 14 Oct 2024 19:40:00 -0000

We fully caught up all queues and process events at the usual latencies. Some events from the time between 09:34 GMT and 09:40 GMT failed to process. We are working on recovering them.

Delayed data processing

Mon, 14 Oct 2024 19:34:00 -0000

Due to a programming error, we observe delayed event processing

Delayed data processing

Mon, 14 Oct 2024 19:34:00 -0000

Due to a programming error, we observe delayed event processing

[EU] Issues with Ingestion

Fri, 11 Oct 2024 18:33:00 -0000

The incident was resolved by completing the infrastructure upgrade.

[EU] Issues with Ingestion

Fri, 11 Oct 2024 18:26:00 -0000

Our metrics indicate that ingestion calls succeed again. We're monitoring the situation.

[EU] Issues with Ingestion

Fri, 11 Oct 2024 18:24:00 -0000

We're currently observing elevated error rates on our ingestion endpoint. This is caused by an infrastructure update. Our team works on a resolution.

E-Mail Login Not Working

Tue, 08 Oct 2024 09:58:00 -0000

All cloud environments should be working again.

E-Mail Login Not Working

Tue, 08 Oct 2024 09:58:00 -0000

All cloud environments should be working again.

E-Mail Login Not Working

Tue, 08 Oct 2024 09:52:00 -0000

We reverted the critical change and validated the effect in our staging environment. A fix is available via v2.83.1 and is being deployed to the Langfuse Cloud environments.

E-Mail Login Not Working

Tue, 08 Oct 2024 09:52:00 -0000

We reverted the critical change and validated the effect in our staging environment. A fix is available via v2.83.1 and is being deployed to the Langfuse Cloud environments.

E-Mail Login Not Working

Tue, 08 Oct 2024 09:30:00 -0000

We are currently investigating an incident where logins via email-address and password are not working.

E-Mail Login Not Working

Tue, 08 Oct 2024 09:30:00 -0000

We are currently investigating an incident where logins via email-address and password are not working.

Infrastructure outage caused failing APIs, slow event processing, and lost events.

Thu, 19 Sep 2024 20:06:00 -0000

We resolved all infrastructure issues. Users might have experienced failing API calls. Also, we lost some traces and processed them slowly. We make continuous efforts to increase the stability and resilience of Langfuse.

Infrastructure outage caused failing APIs, slow event processing, and lost events.

Thu, 19 Sep 2024 20:06:00 -0000

Infrastructure outage caused failing APIs, slow event processing, and lost events.

Thu, 19 Sep 2024 16:28:00 -0000

We exhausted connections on our database. Hence, our containers are partially not be able to complete their APIs or queue processing.

Infrastructure outage caused failing APIs, slow event processing, and lost events.

Thu, 19 Sep 2024 16:28:00 -0000

We exhausted connections on our database. Hence, our containers are partially not be able to complete their APIs or queue processing.

Infrastructure upgrade

Wed, 11 Sep 2024 07:59:00 -0000

All systems operate as expected again.

Infrastructure upgrade

Wed, 11 Sep 2024 07:59:00 -0000

All systems operate as expected again.

Infrastructure upgrade

Wed, 11 Sep 2024 07:28:00 -0000

We will execute an infrastructure upgrade on our EU data region. During this time, data ingestion via the SDKs will be possible. Fetching Prompts will be degraded. Using the Langfuse UI will not be possible for the migration period.

Infrastructure upgrade

Wed, 11 Sep 2024 07:28:00 -0000

Infrastructure upgrade

Sat, 07 Sep 2024 07:26:00 -0000

Everything has been executed successfully.

Infrastructure upgrade

Sat, 07 Sep 2024 07:26:00 -0000

Everything has been executed successfully.

Infrastructure upgrade

Sat, 07 Sep 2024 07:20:00 -0000

We will execute an infrastructure upgrade on our US data region. During this time, data ingestion via the SDKs will be possible. Fetching Prompts will be degraded. Logging into Langfuse will not be possible for the migration period.

Infrastructure upgrade

Sun, 01 Sep 2024 15:26:00 -0000

Everything has been executed successfully.

Infrastructure upgrade

Sun, 01 Sep 2024 15:26:00 -0000

Everything has been executed successfully.

Infrastructure upgrade

Sun, 01 Sep 2024 15:20:00 -0000

Infrastructure upgrade

Sun, 01 Sep 2024 15:20:00 -0000

API rate limiting bug

Fri, 30 Aug 2024 11:41:00 -0000

Bug was reverted. All APIs are fully functional.

API rate limiting bug

Fri, 30 Aug 2024 11:41:00 -0000

Bug was reverted. All APIs are fully functional.

API rate limiting bug

Fri, 30 Aug 2024 11:41:00 -0000

Bug was reverted. All APIs are fully functional.

API rate limiting bug

Fri, 30 Aug 2024 11:41:00 -0000

Bug was reverted. All APIs are fully functional.

API rate limiting bug

Fri, 30 Aug 2024 10:37:00 -0000

When releasing a new implementation of custom API limits, we realized a bug. Reverted immediately. Users on a paid plan were not authorized during deployment.

API rate limiting bug

Fri, 30 Aug 2024 10:37:00 -0000

When releasing a new implementation of custom API limits, we realized a bug. Reverted immediately. Users on a paid plan were not authorized during deployment.

API rate limiting bug

Fri, 30 Aug 2024 10:37:00 -0000

When releasing a new implementation of custom API limits, we realized a bug. Reverted immediately. Users on a paid plan were not authorized during deployment.

API rate limiting bug

Fri, 30 Aug 2024 10:37:00 -0000

When releasing a new implementation of custom API limits, we realized a bug. Reverted immediately. Users on a paid plan were not authorized during deployment.

API latencies and error rates

Tue, 27 Aug 2024 21:42:00 -0000

All APIs work as expected.

API latencies and error rates

Tue, 27 Aug 2024 21:42:00 -0000

All APIs work as expected.

API latencies and error rates

Tue, 27 Aug 2024 20:50:00 -0000

Our API endpoints partially fail or take long time to respond as we are rate limited by AWS. In our infrastructure we exceeded network traffic and are rate limited. We are upgrading the infrastructure to resolve the problem.

API latencies and error rates

Tue, 27 Aug 2024 20:50:00 -0000

Infrastructure upgrade

Mon, 26 Aug 2024 14:24:00 -0000

All APIs are recovered.

Infrastructure upgrade

Mon, 26 Aug 2024 14:19:00 -0000

We executed an infrastructure upgrade which lead to a downtime on our ingestion API. We use Redis internally and upgraded the Redis instance. During this upgrade, we had a 3 minutes window where we dropped events on our ingestion API. All SDKs are built with a retry on errors and are not throwing exceptions into the application.

API latencies and error rates

Fri, 23 Aug 2024 21:42:00 -0000

All APIs behave as expected after infrastructure updates.

API latencies and error rates

Fri, 23 Aug 2024 21:42:00 -0000

All APIs behave as expected after infrastructure updates.

API latencies and error rates

Fri, 23 Aug 2024 20:50:00 -0000

API latencies and error rates

Fri, 23 Aug 2024 20:50:00 -0000

API latencies and error rates

Thu, 22 Aug 2024 18:20:00 -0000

We upgraded our infrastructure. Everything behaves as expected now.

API latencies and error rates

Thu, 22 Aug 2024 18:20:00 -0000

We upgraded our infrastructure. Everything behaves as expected now.

API latencies and error rates

Thu, 22 Aug 2024 18:00:00 -0000

Our Redis queue is not reachable due to exceeded network traffic. We are upgrading our infrastructure to resolve the issue.

API latencies and error rates

Thu, 22 Aug 2024 18:00:00 -0000

Our Redis queue is not reachable due to exceeded network traffic. We are upgrading our infrastructure to resolve the issue.

APIs degraded

Tue, 09 Jul 2024 18:51:00 -0000

All services are fully restored.

APIs degraded

Tue, 09 Jul 2024 18:51:00 -0000

All services are fully restored.

APIs degraded

Tue, 09 Jul 2024 18:51:00 -0000

All services are fully restored.

APIs degraded

Tue, 09 Jul 2024 18:51:00 -0000

All services are fully restored.

APIs degraded

Tue, 09 Jul 2024 18:51:00 -0000

All services are fully restored.

APIs degraded

Tue, 09 Jul 2024 18:51:00 -0000

All services are fully restored.

APIs degraded

Tue, 09 Jul 2024 18:20:00 -0000

We have identified the issue and are currently deploying a fix.

APIs degraded

Tue, 09 Jul 2024 18:20:00 -0000

We have identified the issue and are currently deploying a fix.

APIs degraded

Tue, 09 Jul 2024 18:20:00 -0000

We have identified the issue and are currently deploying a fix.

APIs degraded

Tue, 09 Jul 2024 18:20:00 -0000

We have identified the issue and are currently deploying a fix.

APIs degraded

Tue, 09 Jul 2024 18:20:00 -0000

We have identified the issue and are currently deploying a fix.

APIs degraded

Tue, 09 Jul 2024 18:20:00 -0000

We have identified the issue and are currently deploying a fix.

APIs degraded