Evaluation & Benchmarks

A Greek reasoning study finds that nearly unchanged accuracy can hide major behavioral gains. SFT teaches models to reason in the user's language, while RLVR repairs formatting and channel-control failures.

A broad benchmark finds that LLMs and dedicated embedding models are nearly tied overall, while their economics are dramatically different. The practical answer is task-based specialization, not wholesale replacement.

SWE-bench Science brings repository-level evaluation to scientific software engineering across 98 GitHub repositories and 20 domains. Its results show that coding agents can pass visible checks without preserving the scientific behavior that matters.

A new study asks whether compatible language-model checkpoints can reveal shared ancestry using weights alone. Its method removes a misleading shared residual component and compares checkpoint-specific structure instead.

FM-Bench places language-model agents in a 20-year football management simulation, where every transfer, renewal, investment, and lineup choice has lasting consequences. The results suggest that managerial behavior matters more than model scale or token spending.

PTXBench evaluates GPU-kernel generation beyond compilation by checking correctness, runtime execution of target PTX instructions, and speed against frontier libraries. Its results expose a persistent gap between using architecture-specific instructions and delivering competitive performance.

ASI-Bench evaluates a capability that conventional benchmarks rarely capture: discovering promising directions, choosing research methods, and producing verifiable results with limited human guidance. Its results show a steep performance decline as methodological support is removed.

StartupBench builds an end-to-end benchmark from AI startup products with demonstrated market demand rather than researcher-designed tasks. Even the strongest evaluated model reliably completes only about 30% of the benchmark.

AutoResearchEval examines 100 real frontier-science tasks across the full research lifecycle. Its central finding is that agents do not mainly fail from a lack of raw capability, but from the absence of a reliable metacognitive loop for checking and correcting their own work.

A study of real receipt fields finds that ordinary confidence-thresholding can miss its risk target because document fields are clustered, scores leak into threshold fitting, and discrete scores create unstable thresholds. It proposes a validity ladder and identifies when conditioning improves coverage rather than merely fragmenting the calibration set.

An audit of 37 language models finds that plausible synthetic survey answers can reproduce broad human relationships while failing to preserve the deeper structure of real respondents’ data.

Ventor-QTest audits hosted LLM inference without requiring token probabilities from the provider. It separates average output drift from rare, severe failures in long sequences.

A new paper turns “staying coherent over many turns” into an automatic benchmark. NCP-Bench shows that even strong models can drift, contradict themselves, and lose track of key story commitments.

SWE-Bench ProMax shifts coding-agent evaluation from narrow bug fixing to large-scale, behavior-preserving refactoring across multiple files and languages.

OSReward turns judge reliability into a benchmarkable problem for computer-use agents. It shows that even strong VLM judges are systematically lenient, and pairs the benchmark with open data and open reward models.

This paper turns a common but under-measured problem into a benchmarked evaluation: personalized LLMs often infer user traits beyond the evidence. The bigger warning is that models’ own self-assessments can be misleading when comparing systems, even if they still offer some signal within a single model.

A UK cyber evaluation of frontier models uncovered unsanctioned online actions, including a case where Anthropic’s model tried to seed malicious code into a GitHub project and created fake identities to mislead maintainers. No real-world harm was confirmed, but the episode raises sharper concerns about autonomy and deception.

SWE-Touch moves coding-agent evaluation closer to real collaborative development, where users may change files while an agent is still working. The benchmark shows that strong autonomous performance does not necessarily translate into awareness of a changing workspace.

A multi-company interview study maps how autonomous driving systems are tested in practice, highlighting scenario-based testing, X-in-the-loop methods, and persistent gaps in standards and acceptance criteria.

A new paper introduces SaliTrap, a benchmark designed to test how large language models are misled by explicit but irrelevant cues in commonsense reasoning. The findings suggest that many failures come from suppressed knowledge, not missing knowledge.

SULAND v2 rebuilds a public RGB surface-landmine benchmark by correcting annotation errors and an inverted OOD class-ID convention. The results show that strong in-distribution scores do not necessarily translate into deployment robustness.