Evaluation & Benchmarks

A Hugging Face Blog post argues that newer, broader OCR models can still lose to a narrowly specialized system. On a Brazilian Portuguese benchmark, DharmaOCR outperformed Mistral OCR4 and Unlimited-OCR by focusing its training capacity on one language domain.

The Hindcast paper proposes a time-aware evaluation setup for LLM forecasters, replaying resolved Polymarket questions as if models were standing at an earlier date. Its goal is to separate genuine forecasting from answer leakage through retrieval or training data.

A new arXiv paper introduces CAVE-ABSA, a constraint-aware framework for generating and validating counterfactual examples in aspect-based sentiment analysis. Instead of flipping a whole sentence’s polarity, it aims to change only the sentiment tied to a target aspect.

A new arXiv paper studies serial verifier cascades in LLM harnesses and shows that partial correlation can sharply weaken their reliability gains. Instead of exponential error reduction, cascades may yield polynomial improvement or hit a blind-spot ceiling.

The second International StepUP Competition raises the bar for pressure-based footstep biometrics by testing unseen users, stronger domain shifts, and paired left-right stride verification. The best system reached an 8.00% equal error rate, but unfamiliar personal footwear remains a hard case.

The AIMO Interpretability Challenge aims to move math-model evaluation beyond final-answer accuracy. It asks whether frontier reasoning models rely on robust internal mechanisms or brittle shortcuts.

A new arXiv paper argues that causal inference needs more than identification: it also needs a way to verify whether a proposed observational formula truly identifies a target interventional distribution.

A new arXiv study examines which inputs in chest radiograph classification are prospectively available and which are post-hoc artifacts. It finds that Clinical Indication is highly predictive, while report sections such as Findings and Impression can create substantial label leakage.

EgoProceVQA introduces a benchmark for evaluating procedural reasoning in egocentric video question answering. The work also presents EgoProceGen for data generation and EgoProceAgent, a self-skill-exploration framework for procedural understanding.

A new arXiv paper studies how post-training methods alter confidence signals before, during and after chain-of-thought generation. Its key message: confidence in reasoning models is useful only when matched to the right stage and position.

A new arXiv paper studies grokking at an extreme capacity limit, where a neural network’s expressible functions collapse into a finite-dimensional algebraic variety.

A new arXiv paper audits attribution methods for video models estimating ejection fraction from echocardiography. The models focus on the left ventricle spatially, but fail to highlight the clinically decisive end-systolic and end-diastolic frames.

Hume AI’s Real World VoiceEQ benchmark argues that voice AI should be judged not only by latency and transcription accuracy, but by how well it listens, speaks, and adapts in real conversations.

AgentCompass targets a growing bottleneck in LLM/VLM agent research: fragmented evaluation pipelines that are hard to reproduce and costly to extend. Its design separates benchmarks, execution harnesses, and environments into reusable components.