Rethinking Transformer Depth Through the Lens of Rank Preservation
A new arXiv paper reframes familiar Transformer feedforward-block choices as mechanisms for preserving gradient rank across depth. Skip connections, normalization placement, and width expansion are interpreted as part of a shared tradeoff among rank collapse, composition, and parameter cost.
Read more