Query performance recovered. System operational.

The processing has been stable for several hours now.

We've completed the catch-up for experiment spans and resolved the problem.

Degraded database read replicas were re-created and now serving the public API path. Public API latency remains stable.

The LLM-as-a-Judge delays on HIPPA data region have now been resolved. We will continue to monitor the situation.

All data is processed in time now.

We resolved the issue by scaling our infrastructure.

The backlog was processed and ingestion delays are below 30 seconds again.

We've adjusted scaling in the system and currently see a delay of below 5min for LLM as a judge executions.

This issue has been resolved.

The system returned to a fully stable state.

The issue has been resolved.

We've mitigated the impact and event processing delays are below 60 seconds and falling continuously.

We scaled our systems and resolved the issue.

We scaled our systems and are able to process data in time. Single projects may still be delayed.

We scaled our systems. APIs should respond fast again.

The backlog is being processed and we're below 60s processing delay.

We scaled our systems and queries should respond in time.

We are now processing all llm as a judge executions in time.

We scaled our systems and resolved the issue.

The ingestion delay it back down and events are processed near real-time.

All data is processed in time now.

The incident has been resolved by a patch. The underlying issue was an unbounded data query data kept fetching data and forced our containers to be out of memory killed.

The issue has been resolved, and the LLM-as-a-Judge Evaluators are now executed again as expected.

We've caught up on all outstanding events and are processing newly received events within our usual timeframes.

For the past 40min, we observed elevated rates of 403 errors across Langfuse Cloud APIs, mainly the prompt API route. As a follow-up to previous CloudFlare incidents (https://status.langfuse.com/incidents/01KAGW71ZPD2WTXCFZQBGV7EHH) we moved away from CloudFlare and route messages directly to our Application LoadBalancers.

As a measure to enhance loading times, we've added a CloudFront distribution in front of our LoadBalancer. The CloudFront distributions went live around 4pm UTC and initially, we saw now unexpected rise in 5xx errors which we were actively monitoring.

Once customer complaints about elevated 403 rates, we reverted the change and restored service for all of them.

The root cause is a behaviour in CloudFront that automatically rejects all GET requests with a non-empty request body with a 403 Forbidden code. Parts of the auto-generated clients in our SDKs add `{}` as a request_body if request_options, e.g. a timeout is configured.

We're currently investigating alternative options to improve first-time load speeds globally for Langfuse Cloud.

We apologize for any inconvenience caused.

We scaled the systems. Everything works as expected again.

The fix was successful, and the prompts endpoint latency is back to normal. We’re no longer seeing issues and will continue to monitor.

Issue resolved.

Context: We experienced 5xx errors on POST /api/public/dataset-items due to a migration sequencing failure.

• Root Cause: A column was dropped before the code deploy finished, while the Prisma ORM was still attempting to write to that deprecated field.

• Status: Resolved; all containers are now running the updated code.

• Prevention: We will strictly enforce "Expand and Contract" patterns for all future schema deletions.