DS-Lighting Makes the Harness Behind Data-Science Agents Explicit

Introduction

Large language models can write code, select tools, and iterate on data-science tasks, but a model is only one part of an end-to-end system. The surrounding harness determines how a task is presented, how intermediate state is retained, which artifacts are allowed, and how success is measured. When those choices remain implicit, results from ostensibly similar agents may not be directly comparable, and a failed run can be difficult to diagnose.

DS-Lighting, introduced in an arXiv paper, treats this surrounding machinery as a first-class research object. Rather than proposing another foundation model, it provides a unified toolkit for making the design of data-science agent harnesses explicit and reusable.

Key ideas

• Four reusable layers. The harness is organized into data, workflow, execution, and evaluation. Together, these layers describe the task interface, the sequence or structure of operations, the runtime state and environment, and the required outputs and metrics.
• Executable operator programs. Agents are represented as programs built from executable operators. This representation can capture a predefined pipeline, while also supporting adaptive search that changes course as execution proceeds.
• A shared benchmark interface. Multiple open-source data-science benchmarks are integrated into an MLE-Bench-style task format. Tasks can therefore be run with a shared interface, sandboxed runtime, and metric protocol.
• System-level analysis. The reported experiments vary agents, harnesses, models, and ablations. The paper argues that explicit harness design improves reproducibility, comparability, and reliability, while reducing failures caused by the organization of the overall system.

Why it matters

The main contribution is a shift in what should be compared. Two agents may use the same language model but behave differently because they receive different task representations, operate under different file or execution constraints, or are judged by different evaluation procedures. If these factors are hidden inside an implementation, it becomes difficult to attribute an outcome to the model, the agent policy, or the harness.

Making the layers explicit offers a more practical debugging path as well. A failed experiment can be examined through its data interface, workflow logic, runtime state, and evaluation contract instead of being treated simply as a model failure. For benchmark designers, a common task format can establish clearer experimental boundaries across heterogeneous data-science problems.

This does not mean that standardization makes every task equivalent. Data-science workflows still differ in their tools, search spaces, and evaluation needs, and a useful harness must balance control with flexibility. DS-Lighting is better understood as infrastructure for that discussion: it exposes the conditions under which an agent operates before drawing conclusions about which model or strategy is stronger. The project also releases its code, creating an entry point for reproduction and further extensions.

arXiv