DataPrep-Bench Turns Training Data Preparation Into a Measurable LLM Capability

Introduction

Training data is one of the strongest determinants of what a large language model can actually do. Yet the field still lacks a unified way to ask a practical question: if an LLM, an agent, or a data-centric workflow is asked to prepare training data, how good is the result?

DataPrep-Bench addresses this gap by shifting the evaluation target from surface quality to downstream training utility. Instead of asking whether samples look fluent, diverse, or well formatted, the benchmark asks whether the prepared data improves a model after training, and whether a scoring method can predict that improvement before the training run happens.

Key ideas

• Two capabilities under one benchmark: The authors define LLM-driven data preparation as a combination of data construction and data quality evaluation. Data construction turns raw sources into supervised training examples. Data quality evaluation estimates the value of candidate datasets before downstream training.
• Downstream-grounded scoring: For the construction track, different methods consume the same raw sources. Their outputs are used together with Dolly-15k to fine-tune a base model, and the final score comes from downstream performance.
• Multi-domain, multi-model setup: The benchmark spans six domains and multiple base models, making it less dependent on a single task or architecture.
• A skill-guided construction agent: The paper releases Data-Construction-Skill, an agent designed around construction skills. In the Llama-3.1-8B Finance setting, it improves over the Dolly-only baseline by nearly 20 absolute points and remains competitive with strong agent- and DataFlow-based methods in knowledge-extraction-heavy domains.
• A distribution-based quality metric: For the evaluation track, scoring functions are judged by Pearson correlation with downstream performance on a shared candidate pool. The proposed Distributional Alignment Score uses MMD between a candidate dataset and a domain proxy. It achieves the strongest cross-model correlation in four of six domains and is the only metric reported to exceed r > 0.70 simultaneously in Math, Science, and Medical.

Why it matters

DataPrep-Bench is important because it treats data preparation as a first-class capability rather than as an informal preprocessing step. By fixing raw sources, candidate pools, and downstream-grounded scoring, it gives researchers a common protocol for comparing LLMs, agents, and data workflows.

The work also reflects a broader shift in model development. As training and fine-tuning become more expensive, teams need better ways to decide which data is worth using before committing compute. A metric that correlates with downstream performance can be more valuable than one that merely rewards clean text or simple diversity heuristics.

The benchmark does not settle every question about data value. Different domains, base models, and task goals may still require different notions of utility. But its central message is clear: if data quality is meant to improve models, then the evaluation of data preparation should be tied to what happens after training.

Source: Hugging Face Daily Papers