TTPO Lets Language Models Improve at Test Time Without Ground-Truth Labels

Introduction

Post-training techniques such as reinforcement learning and on-policy self-distillation have become important tools for improving mathematical reasoning in large language models. Their common limitation is the need for ground-truth answers or another form of verifiable supervision. That requirement makes them difficult to apply during test-time training, where a model usually receives a new problem without an answer key.

The paper TTPO: Test-Time Policy Optimization proposes a label-free alternative. The model samples several reasoning rollouts for the same problem, uses a majority vote to produce a provisional answer, and then routes the rollouts into different learning objectives according to whether they agree with that answer.

Key ideas

• A vote is treated as a weak signal, not an oracle. If the majority answer is wrong, using it as a universal teacher can spread an error across every token in every selected trajectory.
• The objective is deliberately asymmetric. Rollouts that agree with the pseudo-label are trained with on-policy self-distillation (OPSD). Rollouts that disagree are handled by a grouped reinforcement-learning objective.
• Disagreement still contains useful information. The authors argue that a rollout contradicting the majority answer is generally likely to be wrong, whether or not the vote itself is correct. This makes disagreement a usable negative signal without requiring certainty about the pseudo-label.
• Learning is selected at token level. The distillation branch reduces the weight of positions that have already converged, while the reinforcement-learning branch focuses on errors made with high confidence. The goal is to avoid spending equal optimization effort on every generated token.
• Self-supervision becomes tighter as the model improves. Better rollouts should make the majority vote more reliable, which in turn improves the routing of examples into the two branches.

Results and implications

The abstract reports that TTPO matches label-supervised OPSD on five competition-level benchmarks without using labels. In a test-time training experiment, Qwen3-1.7B improves from 38.0% to 45.2%. The paper also reports gains of 25.2% to 36.4% in a setting without thinking and describes strong cross-task generalization.

The central contribution is therefore not simply replacing an answer key with a vote. It is the decision to preserve different kinds of information from consensus and disagreement. Agreeing trajectories can serve as teacher data, while disagreeing trajectories can expose likely mistakes; combining both under the same objective would make a wrong vote more dangerous. The asymmetric design attempts to keep the useful part of each signal while limiting the damage caused by noisy pseudo-labels.

TTPO also illustrates a broader direction for test-time adaptation: supervision may come from the model’s own sampled behavior, provided that the training rule accounts for uncertainty and error asymmetry. The approach still depends on the quality and cost of majority voting, and the supplied material does not establish its long-term stability or efficiency across broader settings. It is best understood as a promising framework for label-free reasoning improvement rather than a complete solution to test-time training.

Source: Hugging Face Daily Papers