Previous incidents

September 2024
Sep 19, 2024
1 incident

Infrastructure outage caused failing APIs, slow event processing, and lost ev...

Degraded

Resolved Sep 19 at 10:06pm CEST

We resolved all infrastructure issues. Users might have experienced failing API calls. Also, we lost some traces and processed them slowly. We make continuous efforts to increase the stability and resilience of Langfuse.

1 previous update

Sep 11, 2024
1 incident

Infrastructure upgrade

Degraded

Resolved Sep 11 at 09:59am CEST

All systems operate as expected again.

1 previous update

Sep 07, 2024
1 incident

Infrastructure upgrade

Degraded

Resolved Sep 07 at 09:26am CEST

Everything has been executed successfully.

1 previous update

Sep 01, 2024
1 incident

Infrastructure upgrade

Downtime

Resolved Sep 01 at 05:26pm CEST

Everything has been executed successfully.

1 previous update

August 2024
Aug 30, 2024
1 incident

API rate limiting bug

Degraded

Resolved Aug 30 at 01:41pm CEST

Bug was reverted. All APIs are fully functional.

1 previous update

Aug 27, 2024
1 incident

API latencies and error rates

Degraded

Resolved Aug 27 at 11:42pm CEST

All APIs work as expected.

1 previous update

Aug 26, 2024
1 incident

Infrastructure upgrade

Downtime

Resolved Aug 26 at 04:24pm CEST

All APIs are recovered.

1 previous update

Aug 23, 2024
1 incident

API latencies and error rates

Degraded

Resolved Aug 23 at 11:42pm CEST

All APIs behave as expected after infrastructure updates.

1 previous update

Aug 22, 2024
1 incident

API latencies and error rates

Degraded

Resolved Aug 22 at 08:20pm CEST

We upgraded our infrastructure. Everything behaves as expected now.

1 previous update

July 2024
Jul 09, 2024
1 incident

APIs degraded

Degraded

Resolved Jul 09 at 08:51pm CEST

All services are fully restored.

2 previous updates

Jul 05, 2024
1 incident

Degraded APIs in US data region

Degraded

Resolved Jul 06 at 12:05am CEST

Enabling Open Telemetry-based instrumentation on the API routes hosted on Vercel resulted in Gateway Timeouts (HTTP 504) for a portion of requests from 6:00 PM to 9:00 PM (UTC).

The share of 504 timeouts across different parts of the application was as follows:
- US overall: 2.77%
- US Public API (/api/public*): 2.87%
- US Tracing (/api/public/ingestion): 0.89%
- US Prompt Management (/api/public/prompts): 14.01%

The behavior of Langfuse SDKs when the API was partially unavailable:
- Tracin...

2 previous updates