Daedalus-150M Reverses the Usual Recipe for CPU-Friendly Inference

Introduction

Small language models are often built by shrinking the architecture used for large models and only later adapting the result to CPU inference through quantization or runtime optimizations. Daedalus-150M takes the opposite route. Its authors first fixed the intended operating point—one user, one token at a time, 4-bit weights, and an ordinary CPU—and then selected a network structure around that constraint. The central goal is not merely to reduce parameter count, but to limit repeated reads from a growing key-value cache during long-context decoding.

A selective use of attention

Daedalus-150M has roughly 160M parameters and 18 blocks. Six retain full attention, while the other 12 use short depthwise convolutions. Their recurrent state is only two time steps wide, so these layers do not repeatedly revisit an expanding KV cache as the conversation grows. Attention remains available for global interactions, while convolution handles part of the local and sequential processing workload.

That division creates a specific systems advantage. At an empty context, the hybrid and all-attention models show almost no speed difference. As the context grows, however, the hybrid’s advantage increases. This is the behavior expected when cache traffic is the bottleneck; it is different from a model that is simply smaller or has fewer operations overall.

Training and controlled comparison

The model was trained from scratch on 59.9B tokens. It scored 47.31 across HellaSwag, ARC-Easy, PIQA, OpenBookQA, and WinoGrande, exceeding a pre-set benchmark bar of 42.20. It also outperformed GPT-2 124M, Pythia-160M, OPT-125M, and GPT-Neo-125M, despite those references having seen roughly three to six times more data. Its validation bits-per-byte was 0.8685.

The more informative experiment was a parameter-matched all-attention twin trained on the same data and under the same conditions. The winning criterion was specified before either model was evaluated. The hybrid improved the selected quality metric by 0.81%, matched the twin on downstream tasks, and produced a 4-bit file 6.3% smaller. At a 2,048-token context, it decoded 1.76 times faster than the twin and 2.08 times faster than an external peer of similar size. A bandwidth-only estimate predicted just 1.17 times, suggesting that reduced memory volume alone does not account for the measured gap.

What remains uncertain

The report is useful partly because it documents failed attempts. Quantization-aware training initially incurred an unmitigated 4-bit quality cost. About half of the convolution channels became inactive, but the stock runtime’s shape checks prevented straightforward pruning. The authors also found that a larger vocabulary was not justified for a model of this size.

There is an important practical limitation in the available material: it does not provide a specific CPU model, batch size, or absolute latency and tokens-per-second figures. Relative speedups show the architectural trend, but deployment decisions require measurements on processors such as a particular laptop or server core and under a defined runtime configuration.

Why it matters

Daedalus-150M presents a broader lesson for efficient inference: deployment constraints can be the starting point for architecture design rather than the final stage of compression. It does not establish that convolutions can generally replace attention. Instead, it suggests that for small models, streaming generation, and long contexts, a hybrid may align better with CPU memory behavior than a uniformly shrunken Transformer. Reproduction across processors, runtimes, and real workloads will determine whether the idea translates into consistently noticeable user-facing gains.

Source: Hugging Face Daily Papers