GameXpert-Bench: From Game Generation to Real Development

Introduction

Having a coding agent produce a runnable game from a natural-language prompt is no longer merely a conceptual demonstration. Yet creating a playable prototype is very different from completing a game development task like an expert. A real project combines program logic, visual and audio assets, interfaces, interaction, playability, debugging, and repeated revision. GameXpert-Bench, introduced by the Tencent Hunyuan team, is designed to measure that wider gap.

Evaluating the lifecycle, not just the artifact

Many existing evaluations focus on a one-shot result or an isolated repair task. That can miss the continuity of real development: an agent first creates a project, then responds to feedback or finds problems, and finally adds new features without breaking what already works.

GameXpert-Bench turns this lifecycle into three complementary tracks:

• GameGen: generation from scratch. The agent starts in an empty workspace and creates a complete game from one natural-language request. The evaluation considers whether code, assets, interaction, and playability function together.
• GameFix: diagnosis and repair. The agent addresses reported defects or attempts to discover issues that have not been explicitly identified. This tests whether it can locate the actual failure rather than merely produce plausible edits.
• GameOpt: multi-turn optimization. Request chains derived from real user-agent development trajectories test whether the agent can implement cumulative changes while preserving previous behavior through regression checks.

The benchmark also moves beyond static code inspection. Depending on the task, it uses live game interaction, deterministic behavioral tests, or final-product criteria. The suite includes 97 generation tasks spanning 11 genres, 100 repair tasks drawn from 50 human-verified levels, and 17 optimization chains containing six turns and 102 requests. These components are intended to reflect different pressures within an executable game project.

What the benchmark reveals

The central finding is straightforward: current agents are relatively good at establishing a playable foundation and implementing explicit requirements. Their reliability drops when the task demands active defect discovery, confirmation of runtime behavior, or preservation of existing functionality after several rounds of edits.

That distinction matters because game development is not simply a translation exercise from text to code. A feature may appear in the project while its interaction remains broken, its logic conflicts with another system, or a later change silently removes an earlier capability. By testing the process as well as the product, GameXpert-Bench makes these weaknesses visible instead of allowing one successful generation to stand in for sustained engineering ability.

Why it matters

For agent research, the benchmark shifts attention from code output to executable products and long-horizon workflows. For tool builders, it highlights the importance of runtime observation, test generation, fault localization, and regression protection. For users, the results suggest that agents are useful collaborators for rapidly building prototypes, but should not yet be treated as unsupervised game developers.

More broadly, games are a dense test environment for long-horizon agents: logic, presentation, interaction, and playability must all remain coherent in one executable artifact. Stronger systems will need not only to generate code, but also to validate outcomes, interpret feedback, and preserve system integrity as requirements evolve.

Source: Hugging Face Daily Papers