DeepSeek-V3 in 190 Lines of Pure JAX
Introduction
Large-model architectures are often difficult to study because their research ideas are buried inside extensive training and systems code. The deepseek-v3-jax project takes a different approach: it implements selected components from the DeepSeek-V3 technical report in about 190 non-blank, non-comment lines of pure JAX.
The code does not use Flax, Equinox, or PyTorch. Parameters are stored as ordinary dictionaries of arrays, and the model can be read from top to bottom. Alongside the model file, the repository includes scripts for data preparation, training, evaluation, sampling, plots, and tests, making the project both a compact implementation and a reproducible experiment.
Key points
- Multi-head Latent Attention (MLA): Instead of caching full keys and values for every token, the implementation caches a smaller latent representation. In the published configuration, the cache is reduced from 480 to 144 values per token. A decoupled RoPE component keeps the rotary key representation outside the compressed latent.
- DeepSeekMoE: Each token is processed by one shared expert and routed to four of 16 additional experts. To keep the code simple, every expert still computes on every token and the unselected outputs are zeroed. This matches the routing logic but not the efficiency of a production sparse-dispatch system.
- Load balancing: A per-expert bias nudges routing toward underused experts without changing their weights. The project also implements the sequence-wise balance loss described in the paper.
- Multi-token prediction: An auxiliary module predicts the token after next, with its objective weighted by 0.3.
- Local training: The default model contains 36.2 million parameters and activates about 13.6 million per token. The corpus contains roughly 21.7 million training tokens from 207 public-domain detective novels. On a 24GB MacBook Air M5, 12,000 steps took about 2.6 hours at a median speed of roughly 10.7k tokens per second.
Results and limitations
Validation loss fell from 5.02 at step 250 to 3.51 in the reported run. At step 2,000, expert usage with the routing bias ranged from 5.7% to 6.9%, close to the ideal 6.25% share. Without the bias, usage ranged from 0.09% to 18.6%. However, validation loss was almost identical in the two runs at that point: 3.921 with balancing and 3.923 without it. In this small experiment, the bias clearly changes utilization, but does not produce an obvious loss improvement.
The repository is explicit about what it leaves out. MoE computation is dense rather than truly sparse, MLA does not use weight absorption, and inference explicitly decompresses the latent representation. FP8 training, DualPipe, node-limited routing, and YaRN are also absent. The project should therefore not be treated as a complete production reproduction of DeepSeek-V3.
Why it matters
The main contribution is accessibility. Readers can inspect how attention caching, expert routing, balancing, and cached generation fit together, then reproduce the reported ablation with local hardware. It also separates two often-confused layers of modern model design: a research mechanism can be expressed in a small, readable prototype, while making it fast and scalable requires sparse dispatch, parallel training, low-precision arithmetic, and substantial infrastructure.
For people learning JAX or mixture-of-experts models, this is best viewed as an executable technical walkthrough with measurements, not as a new foundation model release. The published checkpoint is a Python pickle and should only be loaded from a trusted source.
Source: Hacker News
Comments
Checking sign-in status...
Loading comments...