Evaluation & Benchmarks

ExtractBench evaluates agents that extract structured data from enterprise documents under user-defined schemas. It measures not only value accuracy, but also record completeness, grounding quality, and cost.

See2Think examines whether multimodal large language models actually rely on sketches, annotations, and intermediate images during reasoning, rather than merely producing them as a cosmetic trace.

StealthBench shifts the evaluation of autonomous security agents from simply solving tasks to solving them without operational mistakes. Its early results suggest that current agents still struggle with OPSEC discipline.

SecRespond is a benchmark for evaluating AI agents in post-compromise incident response, using forensic disk snapshots and security-product evidence. The results suggest that current frontier LLM agents can follow visible alerts, but struggle with silent intrusions and complete remediation plans.

A new paper introduces “shadow evaluations,” asking AI agents to tackle the central research questions of unpublished papers and letting the original authors judge the results. The early finding: today’s agents can handle engineering, but still struggle with open-ended research progress.

DataPrep-Bench evaluates how well LLMs, agents, and data workflows prepare training data by measuring downstream utility rather than surface-level text quality. It jointly benchmarks data construction and data quality evaluation across domains and base models.

SceneActBench shifts evaluation from describing 3D scenes to acting within them. The benchmark measures whether vision-language model agents can turn visual input into reliable actions across multi-object 3D environments.

A new paper argues that strong single-turn benchmark performance does not guarantee that LLMs can follow user intent as it evolves across a real conversation. The authors introduce a framework that turns static tasks into dynamic multi-turn interactions while preserving the original evaluation protocol.

ProVisE addresses a subtle evaluation mismatch: many spatial tasks are easier to answer by pointing, marking, or drawing than by producing coordinates or text. The framework lets image-generation models respond in pixels and converts those visual answers back into benchmark-compatible predictions.

Tencent WorkBuddy Bench introduces a multi-domain benchmark for coding agents across code, web, office, and security tasks. Its main contribution is not just a leaderboard, but a contamination-resistant and openly auditable task construction pipeline.

ActiveVision is a new benchmark designed to measure whether multimodal large language models can repeatedly observe, reason, and seek fresh visual evidence. The reported results show a striking gap between frontier models and humans.

GAMUT introduces a two-level meta-rubric approach for judging whether long-form model answers include all the facts they should. The benchmark shifts attention from merely detecting wrong claims to measuring missing information.

Apple-PI asks a deeper question than visual realism: does a video model actually follow physical laws when it generates? The benchmark evaluates not only outcomes, but also the reasoning path behind them.

A new paper argues that evaluating security agents by peak success rate alone misses a key operational constraint: cost. It compares offensive and defensive agents under fixed budgets and finds that red-team and blue-team tasks scale differently.

VIABench is a video benchmark built from first-person footage recorded or shared by blind and visually impaired individuals. It evaluates whether multimodal large language models can provide practical assistance in navigation, question answering, and guided interaction.

Stripe has released an open benchmark for testing whether AI agents can build complete Stripe integrations in realistic environments. The results suggest that agents can generate and modify code, but still struggle with validation, browser state, and recovery from ambiguous failures.

OpenAI CFO Sarah Friar has outlined a practical scorecard for evaluating AI return on investment. The focus is not on flashy demos, but on useful work, cost per successful task, dependability, and return on compute.

The paper argues that current AI pentesting benchmarks are too controlled to predict real-world performance. It proposes an evaluation protocol centered on validated vulnerability discovery across more complex targets.

A VentureBeat Pulse Research survey of 157 enterprises suggests that companies are giving AI agents more autonomy while trusting their evaluation systems less. The issue is not simply test coverage, but whether evaluations reflect real-world outcomes.

A Hugging Face Blog post argues that newer, broader OCR models can still lose to a narrowly specialized system. On a Brazilian Portuguese benchmark, DharmaOCR outperformed Mistral OCR4 and Unlimited-OCR by focusing its training capacity on one language domain.

A new ACM MM 2026 paper argues that strong benchmark scores in video LLMs do not necessarily prove visual understanding. It introduces the Visual Dependency Gap, a diagnostic that compares performance on original videos with black-screen inputs.