Same Capabilities, Different Agent Behaviors: MCP vs. CLI

Introduction: For a language agent, a tool interface is not merely a transport layer between a model and a program. Tool descriptions, command formats, and feedback patterns are all visible to the model during decision-making. A new controlled study asks a simple but important question: if MCP and CLI expose the same underlying operations, will agents behave the same way?

Key findings

• The authors introduce MICA, or Matched Interface Comparison and Analysis. It renders MCP and CLI from a shared semantic action contract and sends decoded actions to the same runtime, reducing the chance that implementation differences explain the results.
• The evaluation spans six models and three interactive environments. Despite matched executable capabilities, the interfaces produce substantial differences in task performance and interaction cost.
• The effect goes beyond malformed calls or invalid syntax. Interface presentation, action realization, and environmental feedback influence semantic action selection, failure pathways, and recovery behavior.
• Paired natural trajectories diverged in 94.3% of 2,004 episodes. Early differences were often propagated through later steps instead of remaining isolated events.
• The study also presents DART, or Decoupled Action Realization and Targeted Repair. DART keeps validated semantic commitments intact while repairing interface-specific realization. It improved benchmark scores in all six evaluated settings, by up to 9.0 points, and reduced MCP token cost by up to 27.1k tokens per task.

Why it matters

The central contribution of MICA is methodological: it separates executable capability from the way that capability is presented to a model. Agent failures are often attributed to the model, the tool, or the environment. If the interface itself changes the model’s decision process, however, a final success rate alone cannot identify the source of the problem. Matching the semantic contract and runtime creates a clearer boundary for measuring interface effects.

For developers, protocol selection should therefore involve more than integration convenience. The interface can shape how an agent searches its action space, interprets errors, and uses feedback. Evaluations may need to report interaction cost and recovery behavior alongside task scores. DART suggests a practical repair strategy: when the semantic decision has already been validated, correct the interface-level realization instead of forcing the agent to reconsider the entire plan.

The findings do not establish that MCP or CLI is universally superior. They show that interface realization is itself an intervention variable. As tool use becomes a standard part of agent systems, designing and evaluating the model-facing interface may be as consequential as improving the model behind it.

Source: Hugging Face Daily Papers