HiSTEVE 1.0 Brings Human-Like Keyboard-and-Mouse Control to Minecraft

Introduction

Making an agent behave like a human player in an open-world game requires more than recognizing a screenshot or issuing an isolated action. The agent must remember what has already happened, decompose a broad objective into manageable steps, determine whether a step is truly complete, and recover when execution goes wrong. HiSTEVE 1.0, presented by a Tencent research team, addresses this combination through a hierarchical vision-language-action (VLA) architecture.

Minecraft is a useful testbed for this problem. Its procedurally generated world makes fixed action scripts less sufficient, while ordinary players interact through what they see on the screen and through a keyboard and mouse. HiSTEVE follows that interface model instead of relying on privileged game states or direct environment APIs.

Core design

• Capability-aware orchestration. The upper-level agent harness uses structured memory to track progress. A vision-language model planner then selects subtasks from the abilities that the underlying VLA can actually execute. This limits the gap between an attractive high-level plan and the controller’s practical capabilities.
• Policy-coupled completion detection. Open-world tasks often lack a single explicit success signal. HiSTEVE therefore introduces a completion-state detector conditioned on the VLA policy context. It helps the harness decide whether to advance, recover from a failed attempt, or replan a subtask.
• Native interface control. One VLA policy handles all subtasks using visual observations and native keyboard-and-mouse actions. It must support first-person movement and interaction as well as GUI operations, rather than switching to privileged skills or internal game commands.
• Separation of interaction regimes. GUI actions and ordinary gameplay actions have different control patterns. To reduce interference between them, HiSTEVE uses regime-specific RMSNorm branches and separate token embeddings for mouse movement.
• A staged training recipe. The policy is trained with world-knowledge injection, imitation learning, and reinforcement learning driven by environment feedback. The stated goal is not only stronger action prediction, but also greater reliability and better recovery after mistakes.

Results and implications

The paper reports 85.0% performance on atomic tasks in MineStudio Bench and a 37.4% average success rate on OpenHA. The complete system also executes a long-horizon task such as crafting an iron pickaxe from scratch. These results distinguish two related capabilities: reliable low-level interaction and the ability to sustain a sequence of decisions over time.

The broader contribution is an explicit division of labor between planning and control. The upper layer maintains memory, selects subtasks, and reacts to execution evidence, while the lower VLA remains responsible for acting through the same interface available to a human. The completion detector is particularly notable because it conditions its judgment on the controller’s context. This reflects a practical lesson: abstract language goals alone are not enough to determine whether an embodied task has been completed.

The current evidence is still centered on Minecraft and the reported benchmarks. Long-horizon stability, transfer to other environments, and recovery from more diverse failure modes require further study. Even so, HiSTEVE offers a useful blueprint for agents that must both plan over extended horizons and operate through real interfaces without privileged APIs.

Source: Hugging Face Daily Papers