ProgramDistill Tests Coding Agents on Reference-App Reconstruction

Introduction

Most coding-agent benchmarks describe the desired change through an issue, a specification, or a natural-language instruction. Real web development can be less explicit. A developer may have access to a fully working product and an incomplete implementation, then need to discover the intended behavior by interacting with the product. ProgramDistill, introduced by Microsoft Research, is designed to measure this form of software engineering ability.

How the benchmark works

The benchmark treats a working application as a behavioral reference rather than merely as a source of screenshots or requirements. The researchers factorize applications into features at different granularities and associate each feature with replayable behaviors. Those behaviors can be executed and checked through the corresponding gold patch, creating an objective path for verification.

The automated pipeline, called mine-craft-patch, discovered 1,975 replay-verified behaviors across 26 applications and used them to construct 4,063 tasks without human intervention. This design combines the richness of interactive software with the repeatability required by an evaluation benchmark. It also avoids relying solely on whether an agent appears to satisfy a loosely written textual request.

Results

The study evaluates nine frontier coding agents. In full-application reconstruction, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success, respectively, on cumulative workflows. The results suggest that completing a single feature is not enough: agents must preserve consistency across a sequence of related interactions.

The partial-reconstruction setting makes the difficulty more explicit. As restoration depth increases from 1 to 8, GPT-6 Astra’s success rate falls from 100% to 64.0%, while Claude Opus 5’s drops from 96% to 32%. The decline points to a broader challenge than code generation alone. Agents must observe the reference application, infer dependencies between behaviors, plan changes, and implement them within an incomplete project.

Why it matters

ProgramDistill extends SWE evaluation toward implicit requirements and interactive discovery. Its feature granularity and restoration depth provide controlled difficulty, which can help diagnose whether an agent struggles with discovering behavior, planning a workflow, or carrying out the implementation.

The automated construction process also offers a scalable way to expand benchmark coverage and could support future curriculum-based training. At the same time, the provided material does not establish how broadly the benchmark will generalize across application types. The public release is still being prepared, so further evaluation will be needed to assess its practical scope.

Source: Hugging Face Daily Papers