Previous incidents

December 2024
Dec 18, 2024
2 incidents

[US] Delayed trace ingestion and increased levels of frontend errors

Degraded

Resolved Dec 18 at 12:25pm CET

All events are processed in time again.

2 previous updates

Delayed event processing

Degraded

Resolved Dec 18 at 06:15am CET

Events are processed within a minute now.

2 previous updates

Dec 13, 2024
1 incident

Database unresponsive

Degraded

Resolved Dec 13 at 12:53pm CET

Queued trace and project deletions were processed. We're fully back to normal operations.

3 previous updates

November 2024
Nov 28, 2024
1 incident

Delayed event processing and API errors

Degraded

Resolved Nov 28 at 04:33pm CET

We are fully recovered.

2 previous updates

Nov 25, 2024
1 incident

Our scores API (api/public/scores) returned 500 due to performance issues.

Degraded

Resolved Nov 25 at 11:52am CET

We rolled our back our change. The API behaves as expected now.

1 previous update

Nov 19, 2024
1 incident

Event ingestion delayed on EU instance, affects GET API and UI

Degraded

Resolved Nov 19 at 10:33pm CET

We processed all missing events which we lost during the outage.

2 previous updates

Nov 05, 2024
1 incident

Delayed event processing

Degraded

Resolved Nov 05 at 03:54pm CET

Today, between 12:54 CET and 13:18 CET, we observed database connection issues in our asynchronous event processors in one of our hosting zones. Due to those issues, we stopped our workers which caused the previously reported delays. Before we scaled down the worker instances, we had a window of approximately 10 minutes in which events that were accepted on the API, were not processed by the worker and dropped with an error. This should affect about 1/3 of events that were send to our EU inst...

2 previous updates

October 2024
Oct 31, 2024
1 incident

UI degraded, some routes error, backend API routes unaffected

Degraded

Resolved Oct 31 at 06:59pm CET

We identified the issue and resolved it.

1 previous update

Oct 14, 2024
1 incident

Delayed data processing

Degraded

Resolved Oct 14 at 09:42pm CEST

All historic data is processed.

3 previous updates

Oct 11, 2024
1 incident

[EU] Issues with Ingestion

Downtime

Resolved Oct 11 at 08:33pm CEST

The incident was resolved by completing the infrastructure upgrade.

2 previous updates

Oct 08, 2024
1 incident

E-Mail Login Not Working

Degraded

Resolved Oct 08 at 11:58am CEST

All cloud environments should be working again.

2 previous updates