Co-RL: Diverse Multi-Agent Cohorts Enable Unsupervised Reasoning

Why unsupervised reasoning is difficult

Reinforcement learning has become an effective way to improve the reasoning behavior of language and vision-language models. Yet the most successful recipes usually depend on a reward that can be checked against a reference answer or supplied by human annotators. That dependence becomes harder to sustain as tasks grow more open-ended and model outputs become more difficult for people to evaluate reliably.

Self-rewarding RL appears to offer an escape route: a model can judge its own completions and turn those judgments into training signals. The problem is that the same model acts as both solver and evaluator. Its existing preferences, shortcuts, and mistakes can therefore be reinforced rather than corrected. Over time, exploration may shrink, outputs may become more alike, and training can move toward collapse.

The Co-RL idea

Co-RL introduces cooperative multi-agent training. Instead of asking one model to supervise itself, the framework optimizes several decoupled models at the same time. The agents do not share parameters; each maintains its own behavior and internal representation. Their peer-generated feedback is used to derive rewards for reinforcement learning.

This design changes the source of useful disagreement. A model is no longer evaluated only through its own assumptions, but through signals produced by other members of the cohort. The agents can consequently provide alternative reasoning patterns and expose errors that a single self-evaluator might repeatedly approve.

Diversity is not optional

The paper’s central claim is that the quality of peer supervision depends strongly on cohort diversity. Co-RL increases that diversity with heterogeneous model families and sizes, as well as rephrased versions of training samples. These variations make it less likely that all agents will fail in exactly the same way.

The main takeaways are:

• Multiple parameter-independent models are optimized through peer-derived rewards;
• Training does not use ground-truth labels or externally verified answers;
• Heterogeneous agents and rephrased data reduce correlated errors;
• Diversity helps preserve behavioral variety and mitigate training collapse;
• The framework is evaluated in both text-only and multimodal settings.

Results and broader implications

According to the paper’s reported summary, Co-RL delivers average gains of 3.0% to 8.6% across seven text-only benchmarks for language models. Across four multimodal benchmarks, the reported average gains range from 2.3% to 7.2% for vision-language models. The authors state that Co-RL consistently outperforms the base models and previous label-free approaches, while matching or surpassing supervised methods without access to ground-truth labels.

The broader implication is not simply that more agents produce better scores. Co-RL treats disagreement as a training resource. When external supervision is expensive or unavailable, differences among models can support exploration and provide a form of mutual error checking. This suggests a path toward training systems for reasoning tasks where no single evaluator can reliably define the correct process.

Peer feedback is not automatically trustworthy. If all agents inherit the same data bias, architectural weakness, or evaluation convention, they may still converge on the same wrong judgment. Co-RL therefore makes diversity a core part of the training design rather than a byproduct. Future work will need to clarify how cohort composition, feedback aggregation, and task structure affect stability, and whether the approach remains effective as reasoning becomes more open-ended.

Source: Hugging Face Daily Papers