Kolab Reframes How Human–AI Collaboration Is Evaluated

Introduction

In practical work, an AI agent is rarely judged by whether it can complete a task in one uninterrupted attempt. Professional workflows unfold through revisions, dependencies, changing priorities, and decisions that are difficult to express in a single prompt. A useful suggestion may be accepted as written, adapted into a larger plan, or discarded because it does not fit the expert’s judgment. Most agent benchmarks are designed around automation, so they do not capture these distinctions.

What Kolab measures

Kolab addresses this gap with recordings of real professional workflows. The dataset covers 50 professionals from 40 occupations who collaborated with agents on realistic, long-horizon problems. It includes more than 115 hours of screen recordings, along with participants’ judgments about collaboration and the quality of the resulting work.

Rather than focusing only on final task completion, the benchmark tests two capabilities:

• Forecasting the fate of an agent contribution: a model must predict whether a professional will adopt, modify, or reject an output, and use that prediction to understand what should happen next.
• Assessing collaboration quality: a model must infer satisfaction and output quality from the workflow as it unfolds, without requiring constant explicit feedback from the expert.

The distinction matters because an agent’s response does not remain inside the chat interface. It may be transferred into a document, analysis, plan, or operational decision. A collaborative system therefore needs to anticipate how its work will be used after delivery. It also needs to detect indirect signals: whether a plan was rewritten, whether a recommendation changed the next action, and whether the user is quietly working around the agent’s result.

Findings

Kolab reveals a broad range of collaboration patterns and shows why strong generation is not equivalent to strong collaboration. The study reports that frontier models, both agentic and non-agentic, overestimate how well they collaborate with experts. They also have difficulty distinguishing high-quality interactions from low-quality ones. Supplying a longer slice of workflow context does not consistently make their predictions better.

Why it matters

The benchmark suggests that agent evaluation should expand beyond completion rates. In long-horizon work, success also depends on timing, uptake, revision, and the system’s ability to recognize when its contribution has failed to move the workflow forward. These capabilities are central to dependable assistance, yet they are largely absent from automation-oriented tests.

Kolab offers a foundation for studying collaboration through both observable professional behavior and user judgments. The available material describes the dataset and its high-level findings, but does not provide detailed results by occupation, model, or task type. It should therefore be read as evidence of an evaluation gap, not as a complete ranking of commercial agents.

Source: Hugging Face Daily Papers