DuoDiscover Uses Fast Yardsticks to Break the Evaluation Bottleneck

Introduction

Autonomous engineering systems often face an awkward mismatch: an agent can produce a code change in seconds, while determining whether that change really helps may require a full benchmark, a serving trace, or a hardware run lasting hours. When evaluation is this expensive, a search process can afford only a small number of trials. Simply adding more agents does not solve the problem if all of them are waiting for the same slow feedback loop.

DuoDiscover addresses this constraint by making evaluation itself part of the discovery problem. Instead of asking agents only to find better implementations, the system asks them to find a fast way to estimate progress for the particular objective they are pursuing.

How the system works

• Joint discovery of goals and measurements: Each coding agent explores an optimization target and constructs a tailored proxy evaluator for it. The paper calls this evaluator a “yardstick.” Agents can use the yardstick for frequent iteration and hill-climbing, avoiding a costly true evaluation after every change.
• Fidelity checks against the real objective: A proxy can be wrong, and an optimizer may learn to exploit the gap between the proxy and the actual goal. DuoDiscover therefore periodically rechecks candidate rankings with the true evaluation. If a yardstick’s ranking begins to drift, it is invalidated rather than treated as reliable evidence.
• A bottom-up fleet: The architecture does not rely on a central controller to decompose the entire problem. A lightweight orchestrator starts and stops agents, gives running agents brief updates, and seeds new agents with work that other agents have already confirmed. Exploration can therefore proceed in parallel without requiring a single global plan.

Reported results and limitations

The paper evaluates DuoDiscover on two repository-level benchmarks and four GPU and TPU kernel suites covering hundreds of kernels. On SWE-fficiency, the authors report twice the average speedup achieved by human experts. On SOL-Exec, the system ranks first on 121 of 235 kernels and places in the top three on 212. The source material also reports leading performance on an NVIDIA SOL-Exec benchmark and merged pull requests in the FreeToken project.

The broader lesson is that search scale is not just a matter of agent count. If every candidate must wait for an expensive evaluation, a larger fleet can create more queued work without producing faster discovery. A task-specific proxy can increase the number of useful iterations within the same time budget.

However, a yardstick is still an approximation. True evaluation remains necessary for calibration and final confirmation. DuoDiscover does not remove that cost; it changes where the cost is paid and adds a mechanism for detecting when the shortcut has become unreliable. This distinction matters because an optimizer can otherwise become very good at improving a metric that no longer represents the real objective.

Why it matters

DuoDiscover puts the evaluator inside the autonomous discovery loop. Future systems for code optimization, inference serving, and chip architecture search may need to produce not only candidate solutions but also task-specific ways to measure progress. The open questions include how often fidelity checks should run, how proxy failure should be detected early, and how to limit bias introduced by approximate measurements. These issues will become more important as autonomous search moves from controlled benchmarks to long-running real-world engineering workflows.

Source: Hugging Face Daily Papers