How Data Mixing Helps Language Models Reason in the User’s Language

Introduction

A multilingual model may understand a question written in Chinese, Arabic, or another language, yet still produce its internal reasoning in English before returning an answer in the requested language. This behavior is often overlooked when evaluation focuses only on final-answer accuracy. However, reasoning in a different language can make the interaction less natural, obscure whether the model understood the original wording, and miss knowledge or cultural distinctions that are easier to express in the target language.

A new study from Cohere Labs examines this problem through the lens of L2 reasoning: the ability to reason consistently in the language used by the prompt. Rather than making model scale the central solution, the authors focus on how supervised fine-tuning data should be composed and scheduled. Their main question is whether in-language reasoning requires extensive reasoning supervision for every language, or whether it can be transferred through a carefully designed multilingual mixture.

Key points

• A compact multilingual model: The study introduces Tiny Aya L2-Thinker, a 3.35B-parameter model evaluated on six benchmark areas, including mathematics, commonsense reasoning, instruction following, open-ended generation, and cultural reasoning.
• High reported language consistency: The abstract reports an in-language reasoning rate above 93% across 60 languages while maintaining strong task performance. The result measures more than whether an answer is correct: it also considers whether the reasoning follows the prompt language.
• Coverage matters more than duplicating supervision: Broader language coverage appears to be more useful than creating a complete set of reasoning examples for every language. Multilingual data without explicit reasoning traces also contributes to language competence and transfer.
• English reasoning remains part of the recipe: The findings do not suggest discarding English reasoning data. Instead, a sufficiently capable English reasoning backbone can provide a transferable basis when combined with multilingual data.
• Potential transfer to held-out languages: The study reports that carefully mixed data can help L2 reasoning generalize to languages not directly represented in the reasoning supervision, reducing the need for language-by-language annotation.

Why it matters

The work offers a practical alternative to the assumption that every language needs its own large reasoning corpus. Reasoning may be partly language-agnostic, while the ability to express that reasoning naturally depends on multilingual exposure. Under this view, English reasoning examples, ordinary multilingual text, and a smaller amount of multilingual reasoning data can play complementary roles during fine-tuning.

This approach could lower the cost of extending reasoning models to languages with limited labeled data. It also shifts attention from model size to training composition: which languages are represented, how much non-reasoning text is included, and when different data types are introduced may be as important as adding parameters.

The reported 93%+ figure should still be interpreted carefully. In-language reasoning consistency is not identical to factual accuracy, cultural appropriateness, or equal performance across all languages. The supplied material does not provide the full benchmark breakdown, so it cannot establish that every language performs equally well. Further evaluation will be needed for low-resource settings and culturally specific tasks.

Cohere Labs has released the Tiny Aya L2-Thinker and Tiny Aya En-Thinker weights, along with multilingual reasoning data covering 44 languages. These releases make the proposed data-mixing strategy easier to reproduce and test.

Source: Hugging Face Daily Papers