Can Architecture Potential Be Estimated Without Training? NSC Offers a Spectral Approach

Introduction

Transformer design and model compression often come down to the same allocation problem: with a fixed budget, should capacity go into depth, hidden width, attention heads, or feed-forward layers? Parameter count and FLOPs remain the default measures, but they mainly answer how large a model is and how much computation it requires. They do not adequately distinguish architectures that spend a similar budget in different ways.

A paper titled Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone proposes NSC as a structure-aware alternative for comparing and designing neural networks.

How NSC works

The method starts from the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko–Pastur law provides a theoretical description of that spectrum based on matrix shape and initialization variance. The authors use this relationship to derive a closed-form capacity measure that can be calculated directly from an architecture specification.

This means no model needs to be instantiated, trained, or evaluated on data. Gradients are not required either. The score is intended to reflect structural capacity before the costly part of the machine-learning workflow begins.

NSC also has a useful computational property: its contribution can be summed layer by layer. The authors exploit this decomposition in NSC-DP, a dynamic-programming solver for architecture selection under parameter or FLOPs budgets. Rather than repeatedly training candidates or relying on a black-box proxy, the solver returns the architecture that globally maximizes NSC within the specified search space and budget.

Reported results

• On FlexiBERT architecture ranking, NSC outperformed parameter count and representative training-free proxies. For pairs whose parameter counts differed by less than 10%, NSC retained meaningful ranking ability while parameter count became much less informative.
• On WikiText-103, NSC-DP found a Transformer-XL architecture in about two seconds on a CPU core. The resulting perplexity was better than that of the human-designed baseline reported in the paper.
• For LLaMA-7B pruning, the method selected a 5.7B model without calibration data. It achieved the best aggregate result across eight commonsense reasoning tasks and ran roughly 5,900 times faster than the strongest training-free proxy baseline cited by the authors.

Why it matters

The broader contribution is the connection between architecture measurement and architecture optimization. If a score can be computed from specifications alone and decomposes across layers, architecture search can become a constrained combinatorial optimization problem instead of a sequence of expensive training experiments.

That promise comes with important limits. NSC is grounded in assumptions about random initialization and theoretical spectral behavior. It measures a form of structural capacity, not a universal guarantee of downstream accuracy. Training dynamics, data distribution, optimization choices, hardware behavior, and task-specific sensitivity may all change the final ranking. The reported experiments nevertheless suggest that NSC could serve as an efficient first-stage filter for model families, compression plans, and budget allocation.

The next question is whether spectral capacity can be combined with data-aware signals and deployment costs without losing its speed or additive structure.

Source: Hugging Face Daily Papers