Strong at Tasks, Weak at Teamwork: A Benchmark for AI Teammates

Introduction

Most AI benchmarks ask what a model can accomplish on its own. That framing becomes incomplete when models are deployed in groups with other agents and people. In scientific research, software engineering, medicine, and operational decision-making, a useful agent must do more than produce an answer. It must communicate, verify handoffs, recognize questionable behavior, and remain reliable when another teammate makes a mistake.

A Stanford research team introduces the Collaborative Intelligence Benchmark to make these abilities measurable. The benchmark contains 240 tasks across eight dimensions of teamwork. Its scenarios draw on aviation accident reports, clinical cases, coding exercises, and negotiations, with outcomes that can be checked programmatically.

Key findings

• The benchmark tries to isolate teamwork. A model is scored only on tasks it can solve independently when given complete information. This design reduces the chance that a low score simply reflects missing knowledge or weak task-solving ability.
• No model is a complete teammate. The study evaluates 18 closed- and open-weight models. The strongest result is 82.2 out of 100, indicating that even leading systems have substantial room to improve in collaborative behavior.
• Safety instincts are stronger than social judgment. Frontier models usually reject unsafe shortcuts and report misconduct. In conflicts and negotiations, however, they reach the best overall deal in only 14% of cases, suggesting that interpersonal pressure can distort their decisions.
• Taking over flawed work is difficult. When an agent inherits a task in which a weaker model cheated, its pass rate falls from 97% to 81%. Completing a handoff therefore requires active auditing, not simple continuation.
• General capability is not a teamwork proxy. Models with nearly identical general performance can differ by as much as 11 points as teammates. A conventional capability ranking may therefore be a poor guide to selecting agents for a multi-agent system.

Why it matters

The benchmark’s broader contribution is conceptual: it treats collaboration as a distinct part of intelligence rather than an automatic consequence of scale. A system can produce a correct answer while failing to challenge an unsafe shortcut, validate inherited work, or escalate misconduct. In aviation, healthcare, finance, and production software, these process failures may be more difficult to detect than an isolated wrong answer.

For model developers, collaborative evaluation could become an additional release and deployment gate. Systems should be tested not only on whether they finish a task, but also on whether they verify evidence, resist pressure, and respond appropriately to anomalies. For users, assembling an agent team cannot mean simply connecting several models. Roles, permissions, review procedures, and handoff rules are equally important.

The project provides code and data for 160 public tasks, with 80 held-out tasks for evaluation. By turning teamwork into a repeatable measurement problem, it offers a practical foundation for comparing multi-agent systems and future human-AI collaboration.

Source: Hugging Face Daily Papers