FINCOSMOS Tests Whether Financial Agents Can Be Trusted

Introduction

Financial agents can now search the web, call data APIs, write code, and return investment reports. Their most serious failures may not look like failures at all. A report can be polished while implementing a strategy the user never requested, relying on data that cannot be verified, or using information that was unavailable at the time of the trade. In finance, that kind of silent error can matter more than an obvious factual mistake.

FINCOSMOS, short for “An Open-World Evaluation Paradigm for Agents in Finance,” is designed to expose these problems. Rather than asking whether an agent matches a reference answer, it checks whether the work it delivers can be independently verified.

How the benchmark works

• The strategy is hidden. A simulated investor holds a strategy sampled from a compositional grammar, with roughly 3.8 million possible combinations. Details are revealed only when the agent asks, so the interaction tests whether it can clarify requirements instead of filling gaps with assumptions.
• The data is not supplied. The agent must find data on the open web, document its provenance, and construct a backtest under a specified data cutoff.
• The output is recomputed. Programs inspect provenance and point-in-time correctness, recompute every reported metric, and compare the delivered strategy with the hidden specification. The evaluation spans 45 dimensions and includes 15 veto checks, without requiring a reference answer.

What the results show

The study ran 150 sessions across four coding agents and a model without tools. No session was accepted in full. Before veto checks, quality scores ranged from 20.2 to 37.2 out of 100. The simulated user signed off on 35 sessions, yet every one failed a program check. An LLM judge would have accepted 17 of those sessions, illustrating how easily a convincing presentation can conceal technical errors.

The failures occurred at both ends of the workflow. Only 25 of 120 sessions delivered the investor’s intended strategy, and none produced trades that reproduced the returns reported in the final document. Agents also raised only 54–64% of the details that had not been specified to them. In practice, they often failed either to ask the missing question or to preserve the answer consistently through implementation.

Human experts attributed 88.33% of failures to the agents rather than to the benchmark. The paper also reports that an agent with persistent memory learned all ten tasks after receiving FINCOSMOS feedback. That result suggests a path for improvement, but it should not be mistaken for evidence that current systems are already dependable.

Why it matters

The central lesson is that satisfaction is not correctness. A user may approve a report because it is clear and plausible, while a programmatic audit reveals a look-ahead error, a mismatched strategy, or a metric that cannot be reproduced. Financial evaluations therefore need to inspect the entire chain: requirement elicitation, sourcing, temporal validity, implementation, and numerical reconciliation.

FINCOSMOS also points toward a more useful standard for agent development. Systems should be rewarded for leaving a complete evidence trail and surviving repeated checks, not merely for producing fluent analysis. In high-stakes financial work, the question is not whether an agent can write a convincing backtest, but whether it can deliver one that remains correct when every assumption is challenged.

Source: Hugging Face Daily Papers