Resolved
Steady throughput for the last 30 minutes. Delay < 5min.
Monitoring
our autoscaling responded slower than expected, but responded nonetheless. the backlog is draining
Identified
evaluation throughput was overwhelmed by a slow tenant/model workload, with lock-expiry retries multiplying work; the concurrent worker rollout prolonged the incident.
Investigating
Eval execution in EU region is taking longer than usual