What Shippy Reveals About Building Reliable AI Agents

Introduction

Ai2’s technical write-up on Shippy is a useful case study in what it takes to move an AI agent from demo to operational system. Shippy supports Skylight, a maritime domain awareness platform, by helping analysts query vessel behavior, exclusive economic zones, marine protected areas, vessel tracks, and map links.

The stakes are unusually high. A wrong answer in this context could send a patrol vessel to the wrong location, waste scarce enforcement resources, or create safety risks. That makes Shippy less a story about choosing the most capable language model and more a story about controlling uncertainty around it.

Key points

• The agent is split into soul, skills, and config. The “soul” is the system prompt that defines behavior and boundaries. Skills describe how Shippy should handle specific tasks. Configuration determines the agent harness, model, and runtime settings, so model or framework changes do not require rebuilding the whole agent.
• Skills are versioned as readable Markdown. Shippy uses markdown skill files with structured frontmatter, similar to the agent-skills pattern used by coding tools. These skills cover Skylight API queries, EEZ and MPA boundary lookup, vessel track interpretation, and generation of deep links into the Skylight map.
• Deterministic tools reduce model error. Early prototypes let the agent construct raw API calls, which led to subtle pagination, geometry, and filter mistakes. Ai2 instead built a purpose-made Skylight CLI that exposes typed commands, handles authentication and pagination, and writes structured output to local JSON files.
• Each user session is isolated. Shippy runs on Mothership, an agent hosting platform that provisions a dedicated Kubernetes deployment for each conversation. A user’s Skylight JWT is injected at session creation, files remain inside that session, and network access is limited to required services.
• Evaluation targets the whole agent. Ai2 argues that static question benchmarks do not capture how an agent selects tools, queries live data, acts on results, and knows when to stop. Shippy is therefore evaluated as a combined system of model, skills, and sandbox.

Why it matters

Shippy’s design points to a broader lesson for enterprise AI agents: reliability is an architecture problem. The model matters, but so do schemas, command-line interfaces, permission boundaries, session isolation, and task-specific evaluations.

The most important safeguards are explicit. Shippy is instructed not to make legal determinations about whether a vessel is breaking the law, and not to speculate beyond the available data. In a high-risk workflow, knowing what not to answer is part of being useful.

For other industries, the pattern is transferable. Build predictable tools around complex APIs, keep agent skills auditable, isolate user data, and test the system against real workflows rather than generic prompts. That is how agents become operational infrastructure rather than impressive chat demos.

Source: Hugging Face Blog