TB-MathModel Turns Mathematical Modeling into a Verifiable Agent Test

Introduction

Mathematical modeling is not simply the act of solving a well-defined equation. An agent must interpret a messy situation, choose variables and assumptions, inspect supporting data, formulate a computable model, and defend a recommendation. Many existing evaluations isolate one of these abilities. Others ask an LLM judge to grade a complete report, without checking whether the reported numbers are physically or mathematically feasible.

TB-MathModel, featured by Hugging Face Daily Papers, is designed to evaluate this missing layer. The paper describes it as the Terminal-style Mathematical Modeling Bench. It uses 18 problems from the 2023–2025 China Undergraduate Mathematical Contest in Modeling (CUMCM) and Mathematical Contest in Modeling (MCM), and turns them into terminal-based tasks.

What the benchmark changes

• An end-to-end working environment. Each task includes the verbatim problem statement, official attachments, and figures in a containerized terminal. The agent must inspect files, write and execute code, and emit one JSON artifact rather than answer an isolated subproblem.
• Numerical verification instead of report judgment. The benchmark focuses on the numerical core of the final question. Its direction-aware, one-sided scoring does not penalize a result merely because it beats the human reference anchor.
• Multiple human references. The evaluation draws on 94 Outstanding-award papers. Since modeling problems rarely have one unique solution, an agent must remain close to many accepted human solutions rather than match a single answer.
• Feasibility as a hard gate. A programmatic layer checks six decidable forms of invariants, followed by 624 Lean 4 obligations checked by the kernel. If an answer is infeasible, its score is zeroed even when the headline number is numerically close.

Results and limitations

Across seven multimodal entries, with three seeds for each, the strongest multi-reference evaluation reached 64.58%. The source text reports about 64.6% for GPT-6 Astra and 63.1% for Claude Fable 5.1. An open model costing about $2.54 achieved 59.81%, at roughly one hundredth of the cost cited for the leading systems. These results suggest that current agents can complete parts of an end-to-end modeling workflow, but still struggle to do so reliably.

The benchmark also has clear boundaries. Its contest problems are a proxy for real-world modeling, and its main score concentrates on the final numerical core. That leaves assumptions, communication quality, robustness, and the practical value of a recommendation only partly represented. Formal checks can verify constraints that have been made decidable and expressible; they cannot settle every modeling judgment.

Why it matters

The broader contribution is an attempt to connect a uniform environment, reproducible verification, and agent trajectories. This is a stronger foundation than relying only on an LLM’s opinion about a report. It may help researchers study tool use, data handling, multimodal reasoning, and scientific problem solving under a common protocol, while also providing machine-checkable signals for future training.

TB-MathModel does not show that agents are ready for unrestricted real-world modeling. It does, however, make a previously vague capability more measurable: can an agent turn raw problem materials into a numerical answer that is both close to accepted solutions and consistent with explicit constraints?

Source: Hugging Face Daily Papers