FDE-Gym Tests the Last Mile of Real-World AI Delivery

Introduction

Many current agent benchmarks begin with a clean task description. The objective is explicit, the constraints are known, and the acceptance criteria are fixed. Real customer delivery is rarely so convenient. A customer may describe a symptom rather than the underlying problem, leave important restrictions unstated, or request a solution that does not directly target the business outcome that matters. A forward-deployed engineer must clarify the situation, investigate the live environment, implement a change, and demonstrate that the customer’s results actually improved.

FDE-Gym is designed to measure this broader capability. It places an agent in the role of a forward-deployed engineer and provides 95 end-to-end delivery tasks across 20 customer scenarios and 18 sectors. Industry experts adapted the tasks from real delivery projects. Each scenario includes scripted stakeholders with whom the agent can interact, as well as a modifiable customer system designed to resemble a production environment.

Key findings

• The benchmark measures outcomes, not just task completion. A delivery is scored by the improvement it produces on the customer’s own success metric. An unchanged system receives 0, while an expert reference solution defines 1.
• Information gathering has a price. Messages, trial runs, and waiting time all incur penalties, preventing an agent from asking unlimited questions or repeatedly experimenting without consequence.
• Hidden constraints are the main bottleneck. Leading models lose substantial performance when important facts must be discovered rather than explicitly supplied. A misdirected initial request has a smaller impact by comparison.
• Facts alone do not guarantee delivery. Even after receiving every key fact, the best configuration achieved a 0.8 level of improvement on only 10 of 23 tasks.
• Weak agents can create negative value. Some systems leave the customer worse off than the unchanged baseline, making incident prevention as important as average capability.

Why it matters

FDE-Gym highlights a gap between instruction-following benchmarks and practical deployment work. A useful delivery agent must infer the real objective, decide which questions are worth asking, operate under communication costs, modify a live-like system safely, and remain accountable for a measurable business result. This is not simply a matter of providing a longer context window. It requires coordination between uncertainty handling, goal alignment, tool use, and operational safeguards.

For model developers, the result suggests that training should emphasize problem validation before implementation, not only execution of a predefined sequence. For enterprise adopters, impressive demos or isolated coding scores are insufficient evidence. Agents should be tested in environments that include ambiguity, stakeholder interaction, realistic side effects, and a meaningful “do nothing” baseline. Human approval and rollback paths remain especially important when a weak agent can reduce performance.

FDE-Gym does not show that agents are incapable of customer delivery. It offers a stricter definition of progress. A valuable agent is not one that produces a plausible answer in a clean sandbox, but one that can move a customer metric forward when information is incomplete, communication is costly, and its changes can affect the system.

Source: Hugging Face Daily Papers