Data World Builds the Data Environment Before Training the Agent

Introduction

Data-analysis agents can write code, inspect heterogeneous files, and produce answers to business questions. Yet many of the questions that matter most are inherently difficult to verify. “Why did food-delivery sales decline?” may have several plausible explanations, and even human experts may disagree about which one is decisive. This creates a fundamental problem for training: real company data is expensive to curate and constrained by privacy and intellectual-property rules, while open-ended findings often lack a known generative truth.

Data World reverses the usual workflow. Instead of collecting files first and trying to infer what happened afterward, it specifies the world behind the files before generating them. The system begins with explicit mechanisms describing entities, variables, and relationships. It then synthesizes the observable data, the analytical task, and the rubric used to judge the answer.

How the system works

• Mechanism-first generation: A hidden world is defined before any files are created, allowing the target conclusion to be controlled at the source.
• Joint environment design: Data files, questions, and evaluation criteria are generated as parts of the same environment rather than as disconnected artifacts.
• Recoverability audits: Independent checks test whether the intended target can actually be recovered from the files available to an agent.
• Training-compatible rubrics: The resulting rubrics support both evaluation and reinforcement-learning rewards.

Human preference-based curation was used to select realistic questions. The resulting DataWorld-Bench contains 100 tasks spanning 50 held-out worlds and 16 domains. For training, the engine generated 2,135 tasks from 1,037 disjoint worlds, helping separate the mechanisms seen during training from those used for evaluation.

What the experiments show

Evaluation across ten frontier models suggests that the benchmark remains challenging, particularly when a task requires discovering an insight rather than merely calculating a statistic. Models may successfully load files, write code, aggregate records, and produce numerical results, yet still fail to explain why the result matters or what causal pattern it suggests. The authors describe this recurring behavior as a computation–insight gap.

The researchers also used the synthetic tasks for RL-zero training, without distilling solver trajectories. Starting from Qwen3.6-35B-A3B, they produced DataWorld-35B-A3B. Under matched evaluation, the trained model improved by 7.06 points on DataWorld-Bench, 14.28 on DAComp, 20.90 on DABstep-Hard, and 12.13 on JobBench.

Why it matters

The central contribution is not simply a larger pool of synthetic examples. It is a framework for engineering verifiable data worlds. Researchers can define the mechanism, generate the observations, audit whether the target is recoverable, and reuse the same structure for training and scoring. This is particularly relevant for agents expected to work with files, execute code, and turn analysis into a practical judgment.

The method also has clear boundaries. Its quality depends on whether the underlying mechanisms and questions resemble the messiness of real work. A conclusion that is recoverable in a controlled world may be harder to establish when real data contains missing values, measurement errors, confounding factors, or several simultaneous causes. Data World is therefore best viewed as infrastructure for connecting controllable training with real-world analysis, not as a complete substitute for real business environments.

Source: Hugging Face Daily Papers