How Near-Zero Token Embeddings Can Quietly Stall Gated Model Training

Introduction

A token embedding whose norm is almost zero may look like a harmless corner case. According to Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models, it can instead become a hidden failure point in training. Rare or template tokens may receive no input gradient for long periods. When input and output embeddings are untied, AdamW weight decay can continue shrinking those untouched rows until they become numerically negligible.

Key findings

• The authors examined 291 open-weight language-model repositories containing 280 distinct embedding tables. Among untied gated and hybrid checkpoints, 20 of 38 had reachable input rows with norms below 10^-6. No tied model in the survey showed this condition.
• The risk depends strongly on the architecture. In the tested Transformer settings, using a non-initial position or a BOS token prevented the observed failure. In Mamba, Gated DeltaNet, and xLSTM, however, a near-zero token could amplify input-embedding gradients by more than 30 orders of magnitude, reaching 35 orders in the reported experiments.
• The proposed mechanism links several numerical effects. At nearly zero input, current-token readout factors such as SiLU gates or linear queries can vanish, suppressing access to historical state. Residual RMS values may then fall below the square root of the RMSNorm floor across layers, causing Pre-RMSNorm to magnify backward signals.
• Once a gradient entry exceeds roughly 1.84 × 10^19, FP32 global-norm clipping in PyTorch, Hugging Face Trainer, and FSDP2 may overflow to infinity. The resulting scaling can zero the gradients for that batch. In a Telugu continual-pretraining test with Nemotron-3-Nano-4B, about 20% of steps lost updates, leaving the affected rows frozen.

Why it matters

The work suggests that training diagnostics should inspect individual reachable embedding rows, rather than relying only on loss curves or aggregate gradient norms. This is particularly important for untied embeddings and gated sequence architectures, where a rare token can interact with gates, normalization, and clipping to create a system-level failure.

For released checkpoints, the authors propose GroupClip. It separates input-embedding clipping from the rest of the model and computes the relevant norm with either 64-bit arithmetic or a rescaled formulation. The rescaled version eliminated update loss in the reported tests and repaired triggered rows in one step, with negligible measured end-to-end overhead. For pretraining, GateFloor changes the SiLU gate to SiLU(z) + δ, with δ between 0.01 and 0.1. This keeps the gate positive at zero input and reduced near-zero gradients by 28 orders of magnitude in experiments of up to 370 million parameters, while producing perplexity comparable to the baseline.

The broader lesson is practical: rare tokens deserve numerical health checks before and during training. A nearly unused embedding row is not necessarily inert; in a gated model, it can become the path through which otherwise silent numerical instability spreads.

Source: Hugging Face Daily Papers