LiveMACE Looks Beyond Returns to Diagnose LLM Agent Capabilities

Introduction

The simplest way to evaluate an LLM agent is to inspect its final outcome: did it complete the task, make money, or rank above another system? That approach becomes much less informative when the agent operates continuously in a changing environment. An outcome is then shaped not only by the agent’s decisions, but also by external conditions that evolve during the interaction. A lucky result may not indicate a robust capability, while a poor result may not prove that the underlying mechanism failed.

LiveMACE addresses this problem with a process-aware evaluation framework. Its LiveMACEBench uses live financial markets as a changing testbed, running simulated trading in cryptocurrency and U.S. equity markets. Because future market events have not yet appeared in training data, the setting offers a way to study how agents respond to conditions that are genuinely new rather than merely recalling answers from a static benchmark.

Key points

• Evaluation moves beyond the endpoint: The benchmark measures realized outcomes while also examining the decision and interaction traces that produced them.
• Four mechanisms are isolated: The study focuses on tool use, persistent memory, rule following, and multi-agent collaboration under matched configurations.
• Returns and capability can diverge: Higher realized returns do not consistently imply stronger performance on a particular mechanism, such as memory management or rule execution.
• Similar outcomes may hide different strategies: Agents can reach comparable results while using tools, memory, or collaboration in substantially different ways.
• Access is not effective use: The framework distinguishes whether a mechanism is available, whether the agent uses it effectively, and whether that use improves downstream performance.

Why it matters

The main contribution of LiveMACE is not turning financial markets into another leaderboard. It is using a continuously changing environment as a diagnostic instrument. Trace-level analysis can help researchers ask more useful questions: did an agent fail because it did not call a relevant tool, because it misunderstood the information returned by that tool, or because the information was not integrated into later decisions? Similarly, a long-running system may have access to memory without actually retrieving and applying the right past context. Rule following can also look adequate in isolated tests while degrading over an extended trajectory.

This perspective is relevant beyond trading. Persistent coding agents, operational assistants, and multi-agent systems all face environments in which actions change what happens next. Their evaluation should therefore combine results with evidence about the mechanisms behind those results.

The benchmark does not show that market performance equals general intelligence or reliable investment skill. Financial simulations remain a specific test setting, and 30 days of evaluation cannot settle every question about long-term robustness. Its more defensible lesson is methodological: when agents act in evolving environments, capability, mechanism use, and final performance should be measured separately. LiveMACE provides an open framework for making that distinction concrete.

Source: Hugging Face Daily Papers