QuoteBench Shows How Command Paths Can Hide Agent Failures

Introduction

When a coding agent is asked to operate a terminal, its success rate is often treated as a direct measure of model capability. That assumption overlooks the journey between text generation and execution. A Bash command may be serialized, inserted into a wrapper, interpolated into another string, and parsed again before the shell sees it. A quoting mistake at any of these stages can change the command’s meaning.

QuoteBench examines this hidden boundary. Its central question is practical: when a command-issuing agent fails, did the model generate the wrong command, or did the execution interface damage a command that was originally usable?

Key findings

• The benchmark checks final state, not just text. It contains 56 one-shot tasks grouped into 14 families derived from incidents. Exact final-state validation determines whether the task was actually completed.
• The transport boundary is deliberately stressed. Researchers add one unescaped parser between generation and execution, then replay the same model reply through it. Escaping the interpolation point reproduces the raw-path result, allowing the transport-induced loss to be isolated.
• Replaying the same answer causes a steep drop. Across eight configurations evaluated in the same window, success falls by 55.4 to 73.2 percentage points when the added parser is introduced. This is large enough to make a robust model look unreliable in deployment.
• Disclosure enables adaptation, but not uniformly. Telling the model about the boundary recovers 30.4 to 60.7 points in six configurations. The other two show no recovery or a slight decline. Any improvement therefore comes from changed generation, not from the executor repairing the original response.
• Deployment can reorder models. Raw generation is close to saturated at the frontier tested by the paper. The remaining separation comes from adapting to a particular generation contract and transport path. The study reports one clear reversal among comparable model pairs and several additional comparisons decided by a single task.

Why it matters

The contribution of QuoteBench is less about a new shell technique than about what should count as the evaluation target. A matched score normally assumes that model output reaches the environment unchanged. Real agent stacks do not guarantee that assumption: tool protocols, wrapper scripts, interpolation layers, and shell parsers can all alter the path.

This helps explain why two systems with similar offline scores may behave very differently after being connected to different tools. The paper gives a telling example: one configuration has a matched gap of only -3.6 points, yet the added boundary causes 64.3 points of damage and the model compensates for 60.7 points after disclosure. The final score hides both the failure and the adaptation behind it.

For future command-agent evaluations, reports should specify the model configuration, generation contract, execution transport, operating point, and final-state validator. Developers should also test whether fixing quoting or parsing is more effective than replacing the model. Separating raw generation accuracy from boundary robustness would produce scores that are more comparable—and much more useful for deployment decisions.

Source: Hugging Face Daily Papers