Are Embeddings Overpriced? Frozen Token Codes Shrink Small LMs to 46 MB

Introduction

For a small language model, the vocabulary table can be larger than the transformer body. In Gemma 3 270M, the embedding table contains about 168 million of the model’s 268 million parameters. Once the rest of the network is quantized, embeddings often remain at higher precision, making them an even larger share of the final file.

A paper highlighted by Hugging Face Daily Papers asks whether this cost is necessary. Instead of learning a conventional embedding matrix, the authors assign every token a frozen 512-bit code. The codes are produced once from a pretrained embedding using principal-component projection followed by sign binarization. After initialization, the vocabulary is not updated during training.

Key findings

• Lower training cost: The researchers trained a Gemma-shaped model from scratch on the same 2.05 billion tokens, with and without the learned vocabulary. The learned table improved validation loss by only 0.20 nats, while requiring about 2.6 times as many trainable parameters.
• Competitive downstream results: The frozen-table model matched or outperformed the learned-table version on LAMBADA, HellaSwag, and PIQA. ARC-Easy was the only listed benchmark where the learned table had the advantage.
• Better behavior on the long tail: On rare-word evaluations, the frozen approach led by 1.7 to 2.3 nats. When 10,000 tokens were added after training, its quality cost was reported at roughly 3%, compared with a 43% collapse for the learned-table model.
• A compact ternary model: Combined with a BitNet b1.58 ternary body, the full model occupies about 46 MB. On a laptop CPU, it generates 615 to 740 tokens per second while using about 62 MB of memory. The paper reports a five- to six-fold speed advantage over the 8-bit GGUF version of the float model in llama.cpp, with roughly one-seventh of the memory footprint.

Why it matters—and what remains open

The result challenges the assumption that a small language model must devote most of its parameters to a continuously trainable vocabulary. A fixed codebook can redirect the storage budget toward the transformer body, while also offering a practical path for rare words and vocabulary expansion. That combination is particularly relevant to offline applications on phones and laptops.

The evidence is not a blanket argument against trainable embeddings. The experiments focus on a small model and a specific token budget. At larger scales or with substantially more data, a trainable vocabulary may provide useful flexibility that the frozen representation cannot. The evaluation should also be broken down by language, script, domain, and character rarity rather than judged solely by average validation loss.

Future work needs to clarify how new tokens are encoded, whether the projection must be refit, and whether the frozen table becomes a bottleneck as the transformer receives more training. For now, the method is best viewed as a strong on-device default for compact and low-bit models—not proof that embeddings are universally overpriced.

Source: Hugging Face Daily Papers