Harbor Builds a Common Infrastructure for Agent Evaluation

Introduction

As language models gain the ability to browse, use terminals, call tools, and complete multi-step workflows, evaluating them requires more than checking whether they can produce a correct answer. Agentic benchmarks place models inside environments with tools, state, and execution constraints. The problem is that many of these benchmarks use different interfaces and harnesses, forcing researchers to rebuild integrations repeatedly and making results harder to compare.

A new arXiv paper introduces Harbor Adapters and Harbor-Index as an attempt to address both the infrastructure and dataset sides of this problem.

Key points

• A shared adapter layer. Harbor Adapters ports more than 80 agentic benchmarks to a common evaluation infrastructure. The authors validate the ports through code review and parity experiments, aiming to preserve the original tasks’ semantics and difficulty after integration.
• A broad evaluation campaign. The study tests eight models spanning multiple capability tiers across 54 benchmarks. Each model is run with Terminus-2 and with one of three native harnesses. This setup makes it possible to examine not only model capability, but also the effect of the agent runtime and the sources of failure.
• A curated meta-dataset. Harbor-Index contains 82 difficult and diverse tasks drawn from 29 benchmarks. The selection process combines difficulty filtering, AI and human audits, and an audit-and-fix loop. The goal is to retain broad coverage while making repeated evaluation less expensive.
• Low pass rates remain the norm. None of the evaluated model-harness configurations exceeds a 30% pass rate on the index. The strongest reported combination, GPT-5.5 with Codex, reaches 28.0%, suggesting that planning, tool use, and interaction with complex environments remain difficult even for strong systems.

Why it matters

Harbor’s main contribution is not simply another leaderboard. It exposes the engineering layer of agent evaluation. A common adapter can reduce the cost of adding new benchmarks and help researchers separate failures caused by the model from those caused by the harness, tool integration, or environment setup. For developers, Harbor-Index offers a compact regression suite for checking whether changes to a model or agent stack improve performance without requiring the full benchmark collection every time.

The infrastructure does not eliminate every evaluation caveat. Benchmarks still differ in their objectives, environments, and success criteria, and an 82-task collection cannot represent every real-world workflow. Harbor is therefore best viewed as a reproducible foundation and a complement to broader testing, rather than a single definitive measure of agent intelligence. By releasing the adapters, results, analysis, and index, the authors also make it easier for the community to audit, extend, and refine the evaluation process.

Source: arXiv