AutoVend Tests Whether LLM Agents Can Run a Vending Business

Introduction

Many agent evaluations end when a single action succeeds: an item is found, an order is placed, or a form is completed. Operating a business is a different kind of test. Decisions have delayed consequences, resources are shared across locations, and an action that looks sensible in isolation may reduce future cash flow or create avoidable costs.

AutoVend is designed to measure that broader capability. In the benchmark, one LLM agent operates 20 vending machines distributed across five types of sites through a web-based interface. The simulation runs for 180 days, forcing the agent to make repeated decisions rather than relying on one successful transaction.

What the benchmark tests

• Product and pricing decisions: The agent selects products and sets prices for machines with different demand patterns.
• Procurement and replenishment: It must order stock, account for lead times, and pay for on-site replenishment before warehouse inventory can serve customers.
• Shared-resource coordination: A common warehouse and shared cash position require trade-offs between locations instead of isolated optimization.
• Long-term adaptation: The agent is expected to use accumulated sales, inventory, and cost feedback to revise its operating policy.

The simulator also includes product expiration and recurring operating expenses. These details matter because inventory sitting in a warehouse is not equivalent to inventory on a machine shelf, and excess stock can become a liability rather than an asset.

Main findings

The study compares five LLM agents with a rule-based inventory-control baseline. Four of the five agents perform below that baseline. The result is a useful reminder that language-based reasoning does not automatically translate into reliable operational control. An agent may describe a plausible plan yet fail to execute it consistently, commit inventory that cannot arrive in time, or overlook the interaction between several sites.

The relationship between intermediate metrics and business outcomes is also weak. Improving availability or another individual operating measure does not guarantee stronger economic growth. Inventory holding, expiration, replenishment, and other execution costs can erase the apparent benefit. When returns are divided by inference cost, model rankings change again, suggesting that a costly agent must be judged not only by its gross return but also by the computation required to produce it.

Why it matters

AutoVend shifts agent evaluation from “can the model complete an action?” to “can the model sustain a profitable operation?” That framing is relevant to retail, supply-chain planning, and other online-to-offline systems where digital decisions eventually require physical execution.

The benchmark does not establish how an agent would perform in a real vending network, since it remains a simulator. Its contribution is diagnostic: it exposes weaknesses in feasible action grounding, inventory commitments, cross-site coordination, and policy revision from feedback. For business agents, robust planning and cost-aware execution may matter as much as fluent reasoning.

Source: Hugging Face Daily Papers