xHC: Scaling Transformer Residual Streams Beyond N=4

Introduction

Most discussions about scaling large language models focus on parameter count, depth, width, data, and compute. xHC looks at a more specific but potentially important part of the Transformer: the residual stream. Hyper-Connections (HC) previously proposed expanding the residual stream into N parallel streams, giving the model another form of internal memory capacity. Yet in practice, HC-family methods have generally stopped at N=4. This paper asks whether residual stream width can become a usable scaling axis at larger values.

Key points

• The promise is real, but scaling has been limited. The strong gains from moving from N=1 to N=4 suggest that residual stream expansion can improve model capacity. However, the authors find that scaling mHC beyond that point produces diminishing returns while training cost rises quickly.
• Two bottlenecks explain the slowdown. First, as the number of streams grows, the system lacks sufficient information for effective write-back into all streams. Second, residual-mixing generation becomes expensive because its cost scales cubically with N.
• xHC combines richer write-back with sparse updates. The method introduces temporal feature augmentation to provide more informative write-back signals. It also uses a sparse residual-stream architecture: with N=16 streams, only k=4 are updated at a time, while the model still keeps dense access to the full residual state.
• The reported results are strongest in large MoE pre-training. On an 18B MoE model, xHC reduces final training loss from 1.776 to 1.758 compared with DeepSeek mHC, and raises the average downstream score from 44.8 to 48.8. It adds only 4.1% training FLOPs over the vanilla baseline. On a 28B MoE model, xHC further outperforms mHC by 3.1 average points.
• Efficiency is central to the proposal. Scaling-law experiments indicate that vanilla and mHC models require 1.50× and 1.19× the compute of xHC, respectively, to reach the same loss. The paper also introduces xHC-Flash, which reduces per-sublayer memory traffic from 73.5C to 40C, close to the 34C required by mHC at N=4.

Why it matters

The contribution of xHC is not merely that it uses more residual streams. The more important point is that it diagnoses why mHC becomes unattractive beyond N=4 and proposes a sparse update mechanism to keep larger N practical. This turns residual-stream expansion from an architectural curiosity into a more credible engineering option for large-scale training.

If the results generalize beyond the paper’s own setup, residual stream width could join depth, width, expert count, and data scale as another meaningful dimension in LLM scaling. For MoE pre-training in particular, xHC suggests a way to increase representational capacity without a proportional increase in training FLOPs. Still, the evidence currently comes from the authors’ experiments, so independent replication, tests on other model families, and interactions with other efficiency techniques remain important open questions.

Source: Hugging Face Daily Papers