How Claude, Codex and Cursor Choose Tools: Evidence from 17,000 Runs

Introduction

When a coding agent is asked to add payments, email, storage or deployment to a project, should we expect different agents to recommend the same service? Armature tested Claude Code, Codex and Cursor at scale, running 16,893 experiments to examine how they gather information, compare options and implement a final choice.

Methodology

The researchers analyzed public GitHub repositories and balanced a test panel across languages, frameworks, third-party services, deployment platforms, team sizes and codebase age. They then created 75 synthetic repositories covering 10 programming languages. The repositories used fictional companies, histories and credentials, while package lockfiles were checked against real registries.

Tasks were written for several user profiles, from a symptom-oriented “vibe-coder” to an enterprise engineer specifying compliance, procurement and operational constraints. Each run took place in an ephemeral sandbox. A simulated human was also added to the interaction: the agent first recommended a solution, after which the simulated user accepted the top option or asked it to implement the best choice. This reduced the tendency to build everything in-house simply because the agent could not ask for permission to adopt a vendor.

Of the 16,893 runs, 5,292 sessions across 51 repositories and 18 sectors were judged valid for the first publication. The remaining traces may support later analysis.

Key findings

• Search behavior differs sharply. Cursor used the web in roughly two-thirds of sessions. Codex searched in 94% of sessions and commonly used operators such as site: to focus on trusted domains. Claude Code searched in about 30% of cases, but browsed roughly three times as many pages when it did. In newer areas such as sandboxes, its search rate rose to about 80%.
• The agents often disagree. All three selected the same tool in only 42% of cells. In voice-agent tasks, Claude Code favored Twilio, Codex favored the OpenAI Realtime API and Cursor favored Vapi. Claude Code also chose in-house implementations at about 19%, nearly twice the 10% rate reported for Codex and Cursor.
• Repository context changes the winner. For the same email requirement, Resend led in TypeScript repositories, SendGrid in Python, Postmark in Go and Azure Communication Services in Java. Vercel dominated TypeScript projects, especially those using Next.js, but was not recommended in the Python cases where Render was more prominent.
• Mentions are not adoption. PayPal was mentioned 139 times and never selected; Adyen appeared 175 times but was chosen only three times. LangChain, Netlify and Supabase were also frequently discussed yet often lost to alternatives. Visibility helps a product enter consideration, but does not guarantee implementation.
• Small details can flip decisions. Free-plan retention terms, management overhead and bundled features affected rankings. Supabase often lost when the task required only a database because its authentication, storage and realtime bundle looked unnecessary.

Why it matters

The study challenges the idea that coding agents produce a universal vendor leaderboard. Their recommendations combine prior knowledge, web results, repository structure and the language ecosystem. A single requirement can therefore lead to different answers in different codebases.

For developers, an agent’s first recommendation should be treated as a hypothesis, not a procurement decision. Teams should ask for explicit constraints, compare alternatives and manually verify compliance, pricing, migration effort and operational risk. For vendors, searchable documentation and clear plan descriptions may influence agent-mediated adoption almost as much as product capability.

The most useful future question is not simply who wins a category, but why the winner changes when the repository, prompt or information source changes.

Source: Hacker News