Previous incidents

August 2024
Aug 30, 2024
1 incident

API rate limiting bug

Degraded

Resolved Aug 30 at 01:41pm CEST

Bug was reverted. All APIs are fully functional.

1 previous update

Aug 27, 2024
1 incident

API latencies and error rates

Degraded

Resolved Aug 27 at 11:42pm CEST

All APIs work as expected.

1 previous update

Aug 26, 2024
1 incident

Infrastructure upgrade

Downtime

Resolved Aug 26 at 04:24pm CEST

All APIs are recovered.

1 previous update

Aug 23, 2024
1 incident

API latencies and error rates

Degraded

Resolved Aug 23 at 11:42pm CEST

All APIs behave as expected after infrastructure updates.

1 previous update

Aug 22, 2024
1 incident

API latencies and error rates

Degraded

Resolved Aug 22 at 08:20pm CEST

We upgraded our infrastructure. Everything behaves as expected now.

1 previous update

July 2024
Jul 09, 2024
1 incident

APIs degraded

Degraded

Resolved Jul 09 at 08:51pm CEST

All services are fully restored.

2 previous updates

Jul 05, 2024
1 incident

Degraded APIs in US data region

Degraded

Resolved Jul 06 at 12:05am CEST

Enabling Open Telemetry-based instrumentation on the API routes hosted on Vercel resulted in Gateway Timeouts (HTTP 504) for a portion of requests from 6:00 PM to 9:00 PM (UTC).

The share of 504 timeouts across different parts of the application was as follows:
- US overall: 2.77%
- US Public API (/api/public*): 2.87%
- US Tracing (/api/public/ingestion): 0.89%
- US Prompt Management (/api/public/prompts): 14.01%

The behavior of Langfuse SDKs when the API was partially unavailable:
- Tracin...

2 previous updates

June 2024
Jun 28, 2024
1 incident

Downtime of EU instance

Downtime

Resolved Jun 28 at 02:48pm CEST

We have identified the root cause (an infrastructure migration script which overused database resources) and fixed it. All APIs are available again.

1 previous update