Evaluation & Benchmarks

A commit-pinned census of Inspect Evals shows that executable evaluation code does not automatically support the historical claim attached to its metric. Of 124 mechanically eligible units, 110 stopped before deterministic inference because evidence or semantic grounding was missing.

UrbanGround turns territory-wide 3D geospatial data from Hong Kong into an interactive city sandbox for multimodal agents. The benchmark asks whether local visual understanding can be transformed into reliable, long-horizon navigation and adaptation.

FIRM-Video introduces a checklist-driven reward modeling pipeline that verifies video evidence before assigning scores. Its released dataset and benchmark support finer-grained evaluation across instruction following, world coherence, and perceptual quality.

Video-IFBench shifts video evaluation beyond answer accuracy to the ability to satisfy visual, audio, semantic, formatting, and conditional constraints. Its evaluation of more than 20 recent MLLMs shows that complex video instruction following remains difficult.

AnTrap evaluates Android GUI agents by injecting realistic runtime anomalies into otherwise solvable task trajectories. Its results show broad performance degradation and a clear gap between traps that training can fix and failures rooted in deeper reasoning limits.

VGI-Bench moves video-generation evaluation beyond visual quality to the logic of evolving visual processes. Its results suggest that current systems show pockets of reasoning ability but are still far from reliable.

Google DeepMind is testing a cryptographically protected evaluation setup with external partners. The approach keeps both proprietary model weights and confidential test prompts hidden from the other side.

A new benchmark evaluates coding agents on long-horizon, whole-repository stack migrations rather than isolated bug fixes. Across 520 runs, only 5.4% passed migration, behavioral, and agentic checks.

FrontierChallenge tests whether scientific agents can deliver complete, verifiable workflows rather than merely produce plausible answers. Its results expose a wide gap between partial progress and reliable completion.

ClawProBench moves agent evaluation beyond final answers by inspecting evidence gathering, runtime routing, safety boundaries, and repeatability. Its results show that native-runtime tasks are harder and that one successful run is a poor proxy for reliable performance.

Thinkingbox evaluates agents by checking whether they produce the correct end state in policy-constrained business workflows, rather than merely generating valid replies or tool calls. Its benchmark exposes a wide gap between occasional success and repeatable execution.

GameXpert-Bench evaluates coding agents across the full game development lifecycle, covering generation, bug repair, and multi-turn optimization. The results show that agents can build playable foundations, but still struggle with proactive debugging, runtime verification, and regression control.

MobilePA-Bench introduces an interactive benchmark for mobile planning agents, moving beyond screen manipulation and offline API matching. It evaluates tool use, long-horizon planning, memory, skills, and collaboration under realistic runtime constraints.

LongRCA Bench separates long-horizon agent failure diagnosis into responsible-role attribution and earliest decisive root-step localization. On 1,140 real failed trajectories, its training-free RCTA method improves over baselines while showing that exact causal localization remains difficult.

A new audit shows that expanding a retrieval index can change answers even when the model, prompt, retrieval policy, and generation controls remain fixed. The proposed Snapshot Compatibility Audit separates update-driven answer churn from ordinary sampling variability.

FlavourBench turns open-ended model evaluation into a reproducible portfolio-selection task. Models choose three ingredients from eight, while a versioned culinary system scores every possible combination in advance.

A new study finds a substantial behavioral gap between thinking and non-thinking modes in hybrid-thinking multimodal language models. It introduces PatternEval and PatternRL to measure and reduce response failures beyond task accuracy.

A study from Hume AI introduces a way to measure benchmark optimization in automatic speech recognition. It finds that some high-scoring open-source models may reproduce reference text from public benchmarks even when the audio is contradictory, masked, or ambiguous.

NARU is a benchmark for evaluating whether multimodal models can follow evolving narratives and interpret implicit cultural meaning in extremely long Japanese videos. Its results point to persistent weaknesses in long-range integration and culturally grounded reasoning.

QuoteBench argues that a coding agent’s result depends not only on the command it generates, but also on how that command is serialized, wrapped, and parsed before execution. A matched score can therefore conceal major failures introduced by the interface.

A Greek reasoning study finds that nearly unchanged accuracy can hide major behavioral gains. SFT teaches models to reason in the user's language, while RLVR repairs formatting and channel-control failures.