GTAGym Turns GTA V into a Verifiable Environment for Agent Evaluation and RL

Introduction

A game agent must do much more than recognize an image or answer a question. It needs to interpret a goal stated in language, observe a changing 3D world, select actions, and recover when a plan fails. Verifying whether the goal was actually achieved is also difficult: a final screenshot may not reveal the full state of the game. As a result, many game evaluations still depend on human annotation, making them costly and poorly suited to reinforcement learning.

GTAGym, introduced by the OpenMOSS team, uses Grand Theft Auto V as a unified environment for agent evaluation and reinforcement learning. Its central contribution is not simply the use of a popular AAA game. It connects game state, interaction, outcome verification, and task generation into one operational loop.

Key points

• A broad task benchmark. GTAVBench reconstructs 103 online tasks from GTA V main-story missions. They are organized into nine task families, with another 47 image-based shooting and sniping tasks.
• Automatic outcome verification. Agents observe screenshots and act through keyboard-mouse controls. The server checks the underlying engine state to determine whether a task succeeded, reducing reliance on manual judgment.
• A partially concealed instruction setting. In GTAVBench-AGI, the instruction appears only in the first screenshot. A harness then compresses the interaction history, testing memory and sustained control rather than simple instruction following.
• Clear weaknesses in continuous control. The best configuration among six frontier LLMs achieves 70.9% overall accuracy on the AGI branch, but aerial navigation is the hardest area, followed by interception and evasion.
• A path toward scalable RL. GTAGym samples parameters from user-defined ranges and option sets under fixed templates. It can generate batches of 3D navigation and image-based shooting environments with automatic rewards and termination signals.

Experiments and implications

The authors apply LoRA reinforcement learning to Qwen3.8-27B using generated on-foot navigation tasks. Training improves in-domain performance and transfers to out-of-domain task categories. On native GTA V tasks, performance rises from 15% to 23%. The result does not show that long-horizon game control is solved, but it suggests that generated tasks and automatic reward signals can make training more systematic.

GTAGym matters because it evaluates perception, reasoning, action, and verification in a single loop. Compared with static datasets, a game offers richer state transitions and more consequential actions. For reinforcement learning, automatic termination and reward conditions provide an important bridge from demonstration-style testing to repeated large-scale interaction.

The headline accuracy should nevertheless be read with caution. A 70.9% aggregate score can conceal large differences between task families, and the reported weaknesses point to unresolved problems in spatial modeling, temporal planning, and real-time control. Future progress will depend on whether GTAGym can expand its task templates, maintain stable environment reproduction, and transfer training gains to unseen, more complex missions.

Source: Hugging Face Daily Papers