Large Language Models

The paper introduces U-OPSD, an unsupervised on-policy self-distillation method that builds pseudo-solutions from a model’s own multiple generations. By training on disagreements with those self-consistent pseudo-solutions, it improves mathematical reasoning without ground-truth labels or stronger teachers.

SPOT improves on-policy distillation by deciding where to spend extra probing budget and what targets the student should learn. Instead of relying only on local teacher probabilities, it calibrates distillation with verifier-scored downstream outcomes.

A new technical report argues that interpretability does not have to be bolted onto language models after training. Instead, it can be built into the training process and optimized alongside language modeling itself.

The Skaling law introduces a single interaction exponent to couple model capacity and training data, addressing systematic errors in conventional neural scaling laws.

ByteDance is reportedly training a model that could reach 10 trillion parameters, putting it in the same league as the most advanced frontier systems. The move signals a more aggressive push by Chinese tech giants to compete at the highest level of AI.

GradCuit introduces a test-time latent optimization method that inserts trainable continuous states into an intermediate Transformer layer. By routing reward-weighted gradients from the generated continuation back to those states, it aims to make latent reasoning more robust and interpretable.

Weak-to-Strong On-Policy Distillation proposes a way to improve a strong LLM without relying on an even stronger teacher. The method builds a proxy teacher from contrasts between smaller models in logit space.

Multi-Head Attention Residuals (MHAR) rethink how Transformers read information across depth. Instead of forcing all feature subspaces to share one residual routing distribution, MHAR gives each subspace head its own softmax over layer history.

This paper proposes TOPL, a training paradigm that does not directly imitate off-policy outputs. Instead, it teaches the model to classify whether each token is correct, which helps improve faithful generation under distribution shift.

A new paper featured on Hugging Face Daily Papers introduces Loopie, a Mixture-of-Experts looped Transformer family. The authors claim it outperforms standard Transformer baselines under the same compute budget and gains strong reasoning ability after post-training.

DeepLoop tackles a subtle issue in looped Transformer architectures: reusing the same physical blocks changes how residual updates behave. The paper proposes scaling rules that account for repeated parameter visits rather than only nominal layer depth.

GigaWorld-Policy-0.5 tackles a key bottleneck in World Action Models: costly video generation during inference. It keeps future visual dynamics as a training signal while switching to action-only decoding for deployment.

A new arXiv paper introduces Deep Interaction, a human-in-the-loop method that lets users edit faulty reasoning steps in an LLM’s original answer and reuse the corrected chain to steer future output.

A new arXiv paper reframes familiar Transformer feedforward-block choices as mechanisms for preserving gradient rank across depth. Skip connections, normalization placement, and width expansion are interpreted as part of a shared tradeoff among rank collapse, composition, and parameter cost.

DeltaMerge-LowRes explores whether low-resource NLP adaptation can avoid costly joint language-task fine-tuning. It trains a language delta and a task delta separately, then studies how different merge rules change model behavior.