All events were replayed and usual service levels are restored.

The situation has fully recovered and we identified the root cause as a network congestion within our infrastructure. We're starting to replay events from the 2am to 7am timeframe from earlier to day to restore full service.

We did not see any failure anymore in the past 10 minutes. We continue to observe the situation to ensure all data is processed properly. We will also start to look into replaying the failed data.

We made changes in our infrastructure and thereby reduced the error rate.

We are partially dropping ingestion data before it is written to Clickhouse. We are investigating what the issues are. Roughly 15 percent of events are affected by this.