Omni-IO Skills: Giving General Agents a Multimodal Work Harness

Introduction

General-purpose agents are increasingly capable of planning, reasoning, and acting over long horizons. Yet their ability to produce and transform media remains fragmented. A task that combines video understanding, music generation, document creation, and visual design may require several specialized models or tools, each with its own interface and output format.

The challenge is therefore broader than tool calling. An agent must know which capability to invoke, track dependencies between operations, preserve intermediate artifacts, and reuse those artifacts when a task continues in a later turn. Omni-IO Skills addresses this systems problem with a plug-and-play Agent Harness placed around an existing general-purpose agent.

What the harness provides

• Hierarchical Skills: The framework includes 27 Skills covering 38 representative tasks. They span seven artifact modalities and four capability families: understanding, generation, reasoning, and retrieval. The goal is to expose heterogeneous tools through a more consistent multimodal execution interface.
• Declarative execution graphs: Multi-asset workflows are represented as graphs whose nodes describe operations and whose edges capture dependencies. Independent nodes can run concurrently, while downstream operations wait for the assets they require. This structure fits workflows such as analyzing a video, composing background music, and turning the results into a slide deck with a cover image.
• Persistent asset management: Successful outputs are recorded in an Asset Registry. Instead of treating each tool result as a temporary response, the harness makes intermediate artifacts available to later steps and future conversation turns.
• Pluggable backends: Execution providers are separated from the agent-facing interface. Replacing a backend can therefore be handled through configuration, reducing the need to redesign the agent whenever a model or service changes.

Reported evaluation

On UniM-90, the harness raised the input-support rate of GPT-5.6 Sol from 40.00% to 100%, and that of Claude Sonnet 5 from 38.89% to 100%. Their Semantic–Quality Coupled Scores increased from 26.99 to 74.94 and from 27.82 to 77.78, respectively. Strict Structure Score reached 100.00 for GPT-5.6 Sol and 99.78 for Claude Sonnet 5.

These results suggest that an agent’s practical multimodal coverage can be constrained by orchestration and interface design, not only by the capabilities of its underlying language model. At the same time, the work should be read as a harness-level composition approach. The system relies on replaceable external execution backends, so final quality still depends on the underlying tools, their availability, and the correctness of the dependency plan.

Why it matters

Omni-IO Skills separates two responsibilities. The host model remains responsible for interpreting intent and planning, while the harness turns that plan into an executable, parallelizable, and reusable artifact workflow. This division could make agent systems easier to extend as new media tools appear, without requiring a foundation-model update for every added modality.

The project also releases code and 20 example workflows, offering a practical basis for studying skill composition and multimodal agent orchestration. Its broader contribution is an infrastructure pattern: persistent assets and explicit dependencies may be as important to long-horizon agents as reasoning quality itself.

Source: Hugging Face Daily Papers