Agon turns rival models into implicit graders of reasoning

Lead

Modern reasoning models often rely on reinforcement learning from verifiable rewards: if the final answer can be checked, the model receives a clear training signal. Methods such as GRPO have made this recipe practical, but they also expose a limitation. The reward typically evaluates the final result, not the quality of the reasoning path that produced it. On difficult problems, this may push models to generate longer traces without necessarily learning stronger reasoning strategies.

Agon reframes the problem. Instead of asking humans to label good reasoning steps, or training a separate reward model to judge them, it lets two models compete on the same task. Each model becomes part solver, part implicit evaluator for the other.

Key ideas

• Competition replaces explicit process labels: In Agon, two models attempt the same problem. They alternate roles: one model drafts a solution, while the other reads that draft and solves the problem with that additional context.
• The reward is about out-solving a rival: A model is rewarded for doing better than the other model. To win, it is not enough to produce a verbose chain of thought; the reasoning must be useful enough that a rival who has seen it cannot easily surpass it.
• No separate reward model is required: The paper emphasizes that Agon does not need process annotations or a learned process reward model. The value of a reasoning trace is inferred through the competitive outcome.
• The opponent improves over time: Since both models are optimized, each faces a progressively stronger rival. This creates a moving target that single-model RL does not naturally provide.
• Training and inference match: At deployment, the pair is used as a two-stage cascade: one model drafts, and the second produces an answer after reading the draft.

Results and implications

According to the paper, on the hard split of DeepMath with Qwen3, Agon doubles GRPO’s pass@1. The authors also report that this is roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base model. The same ordering is said to replicate on competitive-programming code tasks and across model families such as Qwen3.5 and Gemma 4.

The broader significance is that Agon turns process supervision into a mechanism-design problem. Rather than explicitly defining what good reasoning looks like, it creates pressure for reasoning that can withstand an informed competitor. This is a different path from collecting high-quality traces, training reward models, or relying on static self-consistency tricks.

There are also practical questions. A paired-model setup may increase training and inference cost. The method assumes the two models are similarly capable but behaviorally different, which raises the question of how to choose and maintain good rivals. For now, the models communicate in text; the paper points to latent-space collaboration as a future direction.

If the approach scales, reasoning model training may look less like a single student practicing against answer keys, and more like two evolving competitors learning to expose and improve each other’s thinking.

Source: Hugging Face Daily Papers