Introduction
Thinking Machines announced Inkling 2 Mini alongside day-one support in vLLM for NVIDIA Blackwell, Hopper, and DGX Spark. The notable engineering challenge is not simply serving another model: Inkling 2 Mini repeats a group of middle layers with shared weights, while passing gated messages between executions. That turns inference into a multi-pass computation with distinct cache needs, rather than a straightforward traversal of the network once.
Key points
- A three-pass loop with separate state. The model has 24 layers: six form a prelude, six a coda, and the 12 middle layers run three times. The loop passes share weights, but each keeps its own KV cache and convolution state. A later pass starts from the prelude output and reads messages from the previous pass; learned controllers gate how those messages are merged into the inputs to each sublayer.
- Multimodal MoE and speculative decoding. Inkling 2 Mini has 26 billion total parameters, with 3.5 billion active, and supports context up to one million tokens. It includes a vision encoder and a native DSpark speculator. vLLM reuses fused kernels from Inkling 2 Small and adds fused operations for message reads, controller projections, gated merging, and normalization to reduce the overhead of the loop.
- A configurable research model. The TML-d24-varloop-research checkpoint lets operators set the number of passes and optionally share a KV cache across them. Fewer passes run faster; more may improve quality. With cache sharing, later passes overwrite the cache entries for their tokens, while convolution states remain separate. These options do not apply to the fixed-loop Inkling 2 Mini checkpoint, and DSpark requires its default pass count.
Performance and implications
In vLLM’s forward-only test on one GB200 at TP1 with a 4K context, one pass took 1.51 ms per decode step at batch 1 and 3.81 ms at batch 32. Three passes took 3.85 ms and 8.11 ms, respectively. Reported KV-cache capacity fell from 352K tokens with one pass to 175K with three. Sharing the cache across three passes produced similar decode times and raised capacity to 194K tokens. These figures describe that specific setup, not a universal performance guarantee.
For numerical behavior, vLLM says it retains the reference implementation’s data types, quantization steps, and FP32 message merges; the main difference is reduction order. MoE outputs match bitwise for identical inputs. Evaluations included needle-in-a-haystack tests up to one million tokens, MMMU-Pro, HLE, and GSM8K. The broader significance is practical: a serving framework can represent each loop pass as ordinary cache layers, while kernel fusion helps contain the added work. Deployers still need to weigh pass count, cache capacity, and task quality for their own workloads. vLLM also supports serving the model for reinforcement learning with SkyRL.
Source: vLLM Blog
Comments
Checking sign-in status...
Loading comments...