Hindcast Replays Prediction Markets to Test Whether LLMs Can Really Forecast

Lead

Forecasting is becoming an appealing way to test whether large language models can reason under uncertainty. Yet for LLMs, ordinary backtesting has a hidden weakness: the answer may already be somewhere the model can retrieve, or may have appeared in its training data. The paper “Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters” introduces a framework designed to close those leaks and evaluate models as if they were operating at a specific point in the past.

Key ideas

• Backtesting is not enough for LLMs. Conventional forecasters are often evaluated by replaying resolved questions and asking what probability the system would have assigned before the result was known. But with LLMs, resolved events are frequently documented online afterward. A retrieval-enabled model may turn the task into a lookup, while a newer model may have absorbed outcome-related data during training.

• The paper highlights two leakage channels. The first comes from retrieval: search or RAG can surface reports written after the event. The second comes from model development itself: a question that was in the future for an older model may sit inside the training distribution of a newer one. In both cases, the benchmark risks measuring recall while claiming to measure foresight.

• Hindcast fixes a historical cutoff. For each resolved Polymarket market, the framework chooses a past time, t0, before the outcome would have been available through either channel. The model can only consult a frozen snapshot of public Reddit and only posts written before that cutoff.

• The comparison includes a market baseline. Hindcast scores each model forecast not only against the eventual outcome, but also against the Polymarket price at t0. That market price represents a human collective forecast made from information available at roughly the same time, making it a meaningful reference point.

Why it matters

The central contribution is methodological. Hindcast argues that time is not a minor detail in LLM evaluation; it is part of the test environment. If the evaluator does not control what information existed at the moment of prediction, strong benchmark performance may simply reflect later knowledge leaking into the model or its tools.

The paper also reports a nuanced view of retrieval. Once leakage is closed, retrieval can still help many models, but mainly when Reddit had relevant pre-event discussion. When the archive contains only loose speculation, retrieval may hurt rather than help. That is an important reminder for RAG-style systems: access to more text is not automatically better if the text is noisy, thin, or temporally misaligned.

For AI evaluation, Hindcast offers a more realistic setup for testing probabilistic judgment. Models must reason from limited public information available at a past date, then compete not only with reality but with contemporaneous human market expectations. Because the archive is frozen and the cutoff is set per market, the benchmark can be rerun as models improve without immediately becoming stale.

Source: arXiv