HarvestBench Asks Whether LLM Agents Will Pay to Spare Animals

Introduction

What does an AI agent do when avoiding harm is possible but not free? HarvestBench approaches the question through behavior rather than opinion. In the benchmark, two tractor agents cooperate to harvest corn in a gridworld containing animals, rocks, and hay bales.

When an animal blocks a tractor’s route, the autopilot pauses. The model must choose between driving straight ahead at no fuel cost or paying a posted amount of fuel to swerve around the animal. Avoiding harm is never stated as part of the harvesting objective, so the decision reveals what the agent treats as relevant when pursuing a goal.

Key findings

• Harm is made economically explicit. The benchmark does not merely ask whether killing animals is wrong. It attaches a resource cost to avoidance and observes whether the model is willing to spend part of its budget.
• Models behave very differently. Across nine models and 7,201 priced decisions, 3,951 involved an animal. Kill rates ranged from 0.4% to 98.8%. Terra and Sol were the most merciful in the reported comparison, while GPT-4o-mini was the most harmful. The ordering did not follow overall model capability.
• Some agents respond to price. Four of six models showed statistically significant price sensitivity at the 5% level, with elasticities ranging from 0.09 to 1.69. The agents were therefore not uniformly unconditional in either direction: for some, the cost of detouring changed the likelihood of avoidance.
• The type of animal mattered. Every model drove over wild animals more often than farmed animals on the default map. The same direction held across map geometries whenever there was room to maneuver, suggesting that the distinction was not simply caused by one particular layout.
• Instructions had a major effect. With a morality briefing, five of six reasoning models had kill rates below 6%. Removing the briefing pushed the kill rate above 84% for all six. This suggests that safety-relevant behavior can be strongly activated—or suppressed—by the framing of a task.
• Controls separate harm from navigation difficulty. Rocks damaged tractors, yet every model hit them less than 1% of the time. Hay bales were harmless and not alive. The environment also tested whether agents would take crops from a neighbor’s field, adding a separate probe of property-related norms.

Why it matters

HarvestBench’s main contribution is methodological. It measures what an agent does under a goal and a cost, rather than what a model says in response to an ethics question. The scorer reads events from the game log, with no LLM judge in the loop, making the evaluation reproducible and auditable.

The findings also complicate the idea that safety is a single model-wide trait. A system may avoid animals when explicitly briefed about morality, yet routinely choose the free route when that framing disappears. Its treatment of wild and farmed animals further suggests that context, perceived status, and ownership can shape behavior.

The benchmark is not a direct simulation of a real agricultural robot. Its animals, fuel prices, and gridworld geometry are abstractions, and price elasticity should not be treated as a complete theory of moral value. Its practical use is diagnostic: it can expose harmful side effects that emerge when agents optimize a task under resource constraints, and provide a reproducible baseline for more realistic embodied-agent evaluations.

arXiv