For the past 40min, we observed elevated rates of 403 errors across Langfuse Cloud APIs, mainly the prompt API route. As a follow-up to previous CloudFlare incidents (https://status.langfuse.com/incidents/01KAGW71ZPD2WTXCFZQBGV7EHH) we moved away from CloudFlare and route messages directly to our Application LoadBalancers.
As a measure to enhance loading times, we've added a CloudFront distribution in front of our LoadBalancer. The CloudFront distributions went live around 4pm UTC and initially, we saw now unexpected rise in 5xx errors which we were actively monitoring.
Once customer complaints about elevated 403 rates, we reverted the change and restored service for all of them.
The root cause is a behaviour in CloudFront that automatically rejects all GET requests with a non-empty request body with a 403 Forbidden code. Parts of the auto-generated clients in our SDKs add `{}` as a request_body if request_options, e.g. a timeout is configured.
We're currently investigating alternative options to improve first-time load speeds globally for Langfuse Cloud.
We apologize for any inconvenience caused.
Resolved
For the past 40min, we observed elevated rates of 403 errors across Langfuse Cloud APIs, mainly the prompt API route. As a follow-up to previous CloudFlare incidents (https://status.langfuse.com/incidents/01KAGW71ZPD2WTXCFZQBGV7EHH) we moved away from CloudFlare and route messages directly to our Application LoadBalancers.
As a measure to enhance loading times, we've added a CloudFront distribution in front of our LoadBalancer. The CloudFront distributions went live around 4pm UTC and initially, we saw now unexpected rise in 5xx errors which we were actively monitoring.
Once customer complaints about elevated 403 rates, we reverted the change and restored service for all of them.
The root cause is a behaviour in CloudFront that automatically rejects all GET requests with a non-empty request body with a 403 Forbidden code. Parts of the auto-generated clients in our SDKs add `{}` as a request_body if request_options, e.g. a timeout is configured.
We're currently investigating alternative options to improve first-time load speeds globally for Langfuse Cloud.
We apologize for any inconvenience caused.
Investigating
We're investigating elevated 403 errors across multiple APIs, including the Prompt API.
