Large Language Models

A Nanjing University team proposes a progressive contraction framework that identifies the reasoning trajectories most useful for learning. By combining teacher-side uncertainty profiling with learner-aware online routing, the approach sharply reduces post-training exposure while preserving or improving reasoning performance.

Balanced token traffic does not necessarily mean that every expert is well prepared for the inputs it receives. DRMoET adds a distributionally robust training objective that emphasizes high-loss routing outcomes while preserving standard sparse MoE computation.

A new study finds that the first tokens of a model’s response can steer the reasoning behavior that follows. Carefully fixing those tokens can bring base models surprisingly close to reinforcement-learning-trained counterparts.

Queen combines a silent chess expert encoder with an instruction-tuned language model, aiming to unite strong play with understandable explanations. The paper reports an Elo rise from 1782 to 2697 after seven rounds of iterative distillation.

Dust replaces analytic gradients with activation perturbations and reward-weighted search. The method reaches backpropagation-like performance in transformer language-model pretraining, although it is not yet a cheaper practical alternative.

A new study suggests that many pretrained Transformers do not lack the capacity for long reference chains; they simply stop using it early. A rank-8 LoRA placed at one early layer can make frozen middle layers carry the computation much farther.

A Meta research team proposes Loop Scaling Laws, a framework that models recurrence, MoE sparsity, model size, and data together. The results suggest that repeated computation and sparse capacity can reinforce each other rather than compete.

A new study finds that a post-trained language model may pass useful capability to another model through single-word answers to unrelated prompts. The authors call the subtle pattern left by post-training a “behavioral shadow.”

Negative Self-Distillation introduces a reverse form of self-distillation: rather than copying a privileged solution trace, a model is trained to move away from its own flawed reasoning. The approach aims to preserve exploration and self-correction without requiring external labels.

Cohere Labs studies how multilingual training data can move reasoning away from an English-only default. Its 3.35B-parameter Tiny Aya L2-Thinker reports more than 93% in-language reasoning across 60 languages.

NCP-ArchPreview adds Next Concept Prediction to conventional next-token prediction, giving the model a discrete latent space for multi-token semantic units. The technical report presents an 8.9B-parameter system that targets both representation quality and training efficiency.

A study of instruction-tuned models from 1.7B to 70B parameters suggests that many models recognize structurally unanswerable math and code prompts before generation. Their failure is better explained by a routing gap between recognition and abstention than by an inability to detect the problem.

A new paper examines Qwen3.8-Flash-Next, a 125B-parameter sparse mixture-of-experts model with only 6B parameters activated per token. Its central lesson is that model quality, compute cost, inference efficiency, and training stability should be optimized together.

SPOT improves on-policy distillation by deciding where to spend extra probing budget and what targets the student should learn. Instead of relying only on local teacher probabilities, it calibrates distillation with verifier-scored downstream outcomes.

The paper introduces U-OPSD, an unsupervised on-policy self-distillation method that builds pseudo-solutions from a model’s own multiple generations. By training on disagreements with those self-consistent pseudo-solutions, it improves mathematical reasoning without ground-truth labels or stronger teachers.

A new technical report argues that interpretability does not have to be bolted onto language models after training. Instead, it can be built into the training process and optimized alongside language modeling itself.

The Skaling law introduces a single interaction exponent to couple model capacity and training data, addressing systematic errors in conventional neural scaling laws.

ByteDance is reportedly training a model that could reach 10 trillion parameters, putting it in the same league as the most advanced frontier systems. The move signals a more aggressive push by Chinese tech giants to compete at the highest level of AI.

GradCuit introduces a test-time latent optimization method that inserts trainable continuous states into an intermediate Transformer layer. By routing reward-weighted gradients from the generated continuation back to those states, it aims to make latent reasoning more robust and interpretable.

Weak-to-Strong On-Policy Distillation proposes a way to improve a strong LLM without relying on an even stronger teacher. The method builds a proxy teacher from contrasts between smaller models in logit space.

Multi-Head Attention Residuals (MHAR) rethink how Transformers read information across depth. Instead of forcing all feature subspaces to share one residual routing distribution, MHAR gives each subspace head its own softmax over layer history.