Show-Harness Lets General-Purpose VLMs Control Robots Through Semantics

Introduction

A vision-language model may understand objects, scenes, and instructions, yet that understanding does not automatically translate into reliable physical control. Conventional robotics pipelines often require a new action representation, a large demonstration set, or embodiment-specific pretraining for each robot. Show-Harness takes a different approach: it redesigns the interface between the model and the machine. The VLM remains responsible for interpreting the scene and making fine-grained decisions, while an embodiment-aware interpreter turns those decisions into executable local actions.

Key ideas

• A compact semantic action space. Instead of asking a VLM to emit low-level control values directly, Show-Harness exposes discrete semantic action units. These units give the model a space it can naturally reason over while remaining more portable than raw motor commands.
• A clear split between reasoning and grounding. The VLM chooses what physical action should happen. A deterministic interpreter then grounds that choice according to the robot’s body, sensors, and execution interface. This reduces dependence on one robot’s control format without removing the model from the physical decision loop.
• A route for both closed and open models. The paper describes direct, zero-shot robot-control experiments with closed-source frontier VLMs. It also presents adaptation of smaller open-source VLMs with only a limited amount of fine-tuning, pointing toward more affordable deployment.
• GUI-based demonstration collection. GUMI, or GUI Manipulation Interface, extends the semantic action space to graphical interfaces. Humans and agents can “play” a robot through a GUI, making it possible to gather demonstrations without specialized teleoperation equipment and potentially across different embodiments.

Why it matters

The central proposition is that the interface may be as important as the model scale. If semantic actions remain stable across robots, a single VLM can potentially reuse its visual knowledge and planning ability across machines and environments. The embodiment-specific interpreter absorbs part of the engineering burden, while the model continues to make decisions at a level closer to human intent.

This does not eliminate the hard parts of robot control. The action vocabulary must cover sufficiently complex tasks, interpreters must cope with perception errors and physical contact, and GUI-collected demonstrations still need to be reliable and safe. The available summary reports that Show-Harness-equipped agents generalize across tasks, embodiments, and environments, outperforming the representative agentic and VLA approaches evaluated by the authors. Its broader contribution is therefore architectural: rather than requiring a separate policy for every body, it explores whether a better semantic bridge can unlock more embodied capability from existing VLMs.

Source: Hugging Face Daily Papers