Evaluation & Benchmarks

A new study finds that large language models may continue to produce visible reasoning even when explicitly instructed to provide an answer only. The authors call this persistence “Thinking Inertia.”

AgentSWE evaluates whether coding agents truly satisfy requirements when building other agents, rather than merely performing well on visible demos. Its results point to a persistent gap between demonstrated progress and reliable delivery.

A study of real JetBrains IDE sessions suggests that coding-agent evaluation is too often reduced to solving a single issue. Real developers ask questions, plan, review, refactor, execute tasks, and switch between these activities within one session.

People routinely infer causes, trajectories, spatial feasibility, social authority, and unwritten rules from a brief visual glimpse. The HSS benchmark turns this largely implicit ability into a measurable test for multimodal models.

A study of 127,833 queries from 6,342 healthcare workers finds that clinical assistants are used mainly for documentation, administration, and knowledge retrieval—not diagnosis. Public benchmarks cover only a limited share of that real-world workload.

Static coding benchmarks can leak into training data and lose their ability to distinguish stronger systems. SWE-Duel lets agents create, hide, and repair bugs for one another during evaluation, allowing task difficulty to emerge from the competitors themselves.

SpireBench uses the full commercial game Slay the Spire to examine whether language-model agents can plan over long horizons. Frontier models approach average human performance, but lose ground to experts as a run continues.

U-Space uses mechanistic interpretability to track when uncertainty appears during generation and what may cause it. Its U-Lens combines interpretable internal signals with predictive entropy to estimate answer reliability from a single generation.

A Johns Hopkins study introduces a trajectory-level way to evaluate epistemic humility in LLM agents facing conflicting knowledge. It finds that strong task accuracy does not necessarily mean an agent can resolve or communicate uncertainty well.

A new study finds that language models can carry small talk and background speech into clinical notes, sometimes misattributing it or using it in reasoning. The results suggest that medical AI needs dedicated tests for resistance to incidental information, not just conventional quality scores.

TestPrism argues that evaluating coding agents against a single reference implementation can significantly overstate test quality. Its multi-implementation benchmark and TestHelix method offer a stricter path toward behavioral test evaluation.

Learn2Play Bench evaluates agents in newly designed text games whose rules are unfamiliar or counterintuitive. Its results suggest that preserving full interaction histories and improving the agent harness can matter as much as changing the underlying model.

A study on HotpotQA and MuSiQue compares spending evaluation budgets on more questions, more search trajectories, or repeated answers. Under similar token budgets, broader question coverage generally produces more stable estimates than repeated runs on a smaller set of questions.

A new study argues that agent evaluation should measure not only what a system can do, but also how efficiently it improves. Its proposed “agent plasticity” metric exposes gaps between final capability, learning efficiency, and generalization.

The Recurrent Looped Transformer, or RLT, combines a parallel causal encoder with a recurrent decoder that feeds each token’s final state into the next step. Experiments suggest this design improves length extrapolation on several algorithmic tasks.

SWE-Game evaluates coding agents across 247 game-development tasks, from implementing mechanics to repairing faults and porting projects between engines. The results show that agents can produce playable prototypes, but still struggle with complete requirements and reliable gameplay logic.

UltraText Bench evaluates dense visual text generation across English and Chinese, multiple regions, and three difficulty levels. Its results show that legibility, textual accuracy, and spatial control can diverge sharply.

A new study finds that an LLM’s written chain of thought often does not fully match the strategy detected in its internal computations. The authors introduce CIA to measure this gap and explore whether post-training can reduce it.

As scientific agents quickly master static benchmarks, evaluation can lose its ability to expose meaningful weaknesses. AutoSciBench proposes an automated loop that generates, tests, and revises scientific tasks as agent behavior changes.

A study of test-time training finds that retaining updates learned from a model’s own generations can create a long-horizon degradation loop. Independent evidence before committing an update offers a practical way to reduce the damage.

UndoBench separates an agent’s ability to complete a workflow from its ability to recover safely after tool failures. The benchmark finds that successful completion can coexist with duplicate external effects and high recovery costs.