Resolved
The queue backlog has been fully processed for legacy trace and dataset targeted evals. There is no longer a delay for eval executions.
Identified
We have identified the root issue and are scaling our infrastructure to work through the evaluation backlog.
Investigating
We currently see a delay of LLM as a judge execution for trace-level evaluations by 30 minutes. We currently look into the issue to make sure we execute evals in time. Observation level evaluations are not impacted as they are on a new code path and more scalable by an order of magnitude. Please migrate to observation level evaluations.