Tencent WorkBuddy Bench Targets Real-World Evaluation for Coding Agents

Introduction

As coding agents move from demos into everyday software workflows, evaluation needs to go beyond isolated programming puzzles. Tencent’s WorkBuddy Bench aims to address this gap by testing agents on a broader set of work-like tasks, including repository-level engineering, front-end development, office and business workflows, and security scenarios.

The benchmark is presented as a unified evaluation suite for coding agents, with a strong emphasis on reproducibility and resistance to benchmark contamination.

Key points

• Four work domains: WorkBuddy Bench is organized into Code, Web, Office, and Security subsets. These cover repository engineering, front-end work, office or business processes, and red-/blue-team security tasks.
• Contamination-resistant task design: Tasks are not simple adaptations of public issue descriptions. Each task is reverse-engineered from a real commit, pull request, or business scenario, then rewritten as a short, colloquial, role-played user request.
• Open release instead of secrecy: The benchmark includes task directories, environment images, evaluation harnesses, tests, and reference solutions. Its contamination resistance relies on task construction and dataset versioning rather than hidden data.
• Uniform execution format: All subsets are packaged in a common task-directory structure and run under a reproducible protocol. The report mentions evaluation on two agent harnesses: CodeBuddy Code and Claude Code.
• No suite-wide average: Because each subset uses a different scoring instrument, scores are not treated as directly comparable across domains, and the benchmark does not report a single overall average.

Interpretation

The most important shift here is from “can the model solve a coding question?” to “can an agent complete a realistic work request?” Real engineering work often requires navigating a repository, understanding implicit context, editing user interfaces, manipulating business artifacts, or dealing with security constraints. A benchmark that spans these categories can reveal different failure modes than conventional code-generation tests.

The contamination strategy is also notable. Public benchmarks are vulnerable to being absorbed into training corpora, which can inflate reported performance. WorkBuddy Bench tries to reduce that risk by transforming real development traces into new user-facing prompts while still keeping the benchmark open for inspection and reruns. This is a pragmatic middle ground between fully private evaluation and fully searchable public tasks.

Why it matters

For model developers, the suite may help separate agents that merely generate plausible code from agents that can complete end-to-end work. For enterprise adopters, the domain split offers a more practical view of where an agent is reliable. For researchers, the open release of environments, tests, and reference solutions makes the benchmark auditable and reproducible.

At the same time, the report’s decision not to publish a single aggregate score is important. Different work domains require different validation methods, so a unified ranking could be misleading. The broader lesson is that coding-agent evaluation is likely to become more contextual: not just which model ranks first, but which agent performs reliably under which type of work.

Source: Hugging Face Daily Papers