LongZero Trains Long-Context Reasoning Without External Data

Long-context reasoning is not simply a matter of increasing the number of tokens a model can read. A capable system must connect facts scattered across documents, follow multi-hop evidence chains, and preserve the right clues through several rounds of retrieval. Reinforcement learning can strengthen these behaviors, but it requires a steady supply of long, diverse, challenging, and verifiable tasks. Human annotation is expensive, naturally occurring examples are limited, and conventional synthesis often depends on external corpora or a stronger teacher model.

LongZero addresses this bottleneck by asking the model to create its own training tasks without external data. Its main design choice is to separate task specification from long-context generation. First, the model writes a question, an answer, and the evidence supporting that answer in a compact structured format. Only afterward does it expand the evidence into a long context made up of multiple documents. This decomposition avoids asking the model to simultaneously invent a valid task, derive a correct answer, and organize a lengthy document collection. It also makes the intended evidence chain explicit before the context is made noisy and long.

The resulting training set contains two complementary forms of self-generated tasks. Multi-hop reasoning tasks require the model to combine several evidence pieces into an answer. Multi-round retrieval tasks focus on finding relevant information step by step and carrying useful state across turns. LongZero trains on these tasks with GRPO and evaluates the approach on two language models across five long-context reasoning benchmarks.

The reported results suggest that the synthetic tasks are not confined to the format in which they were created. On Qwen3-4B-Thinking-2507, the average score across the five benchmarks increases from 54.7 to 57.4. The method also outperforms reinforcement learning based on data synthesized from external corpora, according to the paper’s summary. These findings indicate that a model can use internally generated, evidence-grounded tasks to practice behaviors relevant to real long-context evaluations.

The broader importance of LongZero is methodological. It does not imply that a model can acquire arbitrary new knowledge from nothing; instead, it explores how post-training can reduce dependence on curated examples by focusing on structure, verification, and task difficulty. Important open questions remain around self-training bias, the quality of generated evidence, and whether synthetic distributions adequately represent real workloads. Still, LongZero suggests that better task construction may be as important as larger datasets for advancing long-context reasoning.

Source: Hugging Face Daily Papers