Previous incidents

All services are fully restored.

Enabling Open Telemetry-based instrumentation on the API routes hosted on Vercel resulted in Gateway Timeouts (HTTP 504) for a portion of requests from 6:00 PM to 9:00 PM (UTC).

The share of 504 timeouts across different parts of the application was as follows:
- US overall: 2.77%
- US Public API (/api/public*): 2.87%
- US Tracing (/api/public/ingestion): 0.89%
- US Prompt Management (/api/public/prompts): 14.01%

The behavior of Langfuse SDKs when the API was partially unavailable:
- Tracin...

We have identified the root cause (an infrastructure migration script which overused database resources) and fixed it. All APIs are available again.