LLMs Do Not Know What They Know: Rethinking Financial Backtests

Introduction

When a language model is asked to predict a historical stock price, macroeconomic figure, or Federal Reserve decision, is it forecasting—or recalling information that was already present in its training data? The distinction is crucial for financial backtesting. If the outcome is already known to the model, apparent predictive performance may contain hindsight information rather than genuine forecasting ability.

A study highlighted by Hugging Face Daily Papers introduces FinDendro, a benchmark designed to test whether models can recognize the temporal boundaries of their own knowledge.

Key findings

• A broad dated dataset. FinDendro contains 2,226 U.S. facts spanning 1999 to 2026. The benchmark covers monthly macroeconomic series, month-end stock prices, and Federal Reserve decisions. Each fact is converted into several prompt types, including direct value questions, recall-versus-prediction judgments, and prediction requests tied to a historical date.
• Self-reports are unreliable. GPT-5 answered 76% of the controlled facts correctly, yet claimed to know only 38% of them. It also answered 61% of the facts it said it did not know correctly. In practice, “I do not know” does not mean that the model lacks usable information, while “I know” does not necessarily mark a reliable knowledge boundary.
• Knowledge claims can lag behind actual behavior. The researchers found that claims of prior knowledge could persist for as many as seven quarters beyond the measured knowledge boundary. A cutoff inferred from self-description may therefore be either too conservative or too optimistic.
• Historical-date prompts do not establish isolation. Asking a model to answer as of an earlier date removed most claims of prior knowledge, but left accuracy largely unchanged. The instruction changed how the model characterized its answer, not clear evidence of whether later information had been internalized.

Why it matters

The study reframes leakage control in financial backtesting as a behavioral evaluation problem. Practitioners commonly rely on a vendor’s stated knowledge cutoff or ask the model whether it knows an outcome. FinDendro suggests that neither approach can certify that a backtest is free of hindsight information. A historical-date instruction is also not sufficient proof that the model has been isolated from later events.

A more defensible process is to probe the model with dated facts before running the backtest. Developers can measure accuracy, knowledge claims, and guessing behavior across time, then choose a testing window based on demonstrated behavior rather than documentation alone. This turns the question from “What cutoff does the model report?” into “During which periods does the model show repeatable factual knowledge?”

For financial AI teams, backtest reports may need to document temporal-boundary tests alongside returns and risk metrics. The broader lesson extends beyond markets: whenever a model is evaluated on historical or time-dependent tasks, its claimed ignorance should be tested rather than assumed.

Source: Hugging Face Daily Papers