Thinking Machines Releases Inkling, an Open Multimodal Model Near the Trillion-Parameter Scale

Lead

Thinking Machines has introduced Inkling on Hugging Face, an open multimodal model built to handle text, images, and audio as native inputs. Its headline specifications are hard to ignore: 975B total parameters, 41B active parameters, and a context window of up to 1M tokens. Rather than presenting itself as a text model with auxiliary perception modules, Inkling is positioned as a foundation model for reasoning across modalities and for domain-specific fine-tuning.

Key points

• Large but sparse: Inkling is a decoder-only Mixture-of-Experts model. While its total parameter count is 975B, only about 41B parameters are active during a given forward pass. The architecture uses 256 experts, combining routed experts with always-active shared experts.
• Native multimodal inputs: The model accepts text, images, and audio. For images, a hierarchical MLP patchifier progressively merges pixels into patch embeddings. For audio, the waveform is converted into a discretized mel spectrogram representation and then embedded for the model.
• Long-context architecture: Inkling supports a 1M context window. Instead of RoPE, it uses relative attention, learning position information inside the attention logits. Its layers alternate between sliding-window attention and global attention in a 5:1 pattern, with the final layer using global attention.
• Local modeling via SConv: A short 1D convolution module reads the current token and nearby previous hidden states. This is intended to help capture local patterns while leaving attention and MoE components to focus on broader representations.
• Inference ecosystem: Hugging Face highlights day-one support in transformers, along with compatibility across major inference engines such as SGLang and vLLM. The release also points to ggml quantization options for llama.cpp-based deployment.

Why it matters

Inkling reflects a broader shift in open AI models: scale, long context, and multimodality are increasingly being combined in a single release. For developers building applications that must reason across screenshots, documents, speech, and text prompts, a native multimodal model can reduce the complexity of stitching together separate encoders and language models.

The model is also notable because its multimodal towers are described as relatively simple compared with systems that rely on heavy standalone encoders. That design could make Inkling especially interesting for research into unified multimodal architectures and for teams that want to adapt a large open model to specialized domains.

The practical constraint is compute. The BF16 checkpoint requires around 2TB of VRAM, while the NVFP4 variant requires around 600GB of VRAM. Most users will therefore encounter Inkling through hosted inference, inference providers, or quantized deployments rather than direct local use of the full model.

Overall, Inkling is less a consumer-ready model than a new open foundation for ambitious multimodal systems. Its real impact will depend on benchmark behavior, community tooling, and whether developers can make its cost-performance profile work for real applications.

Source: Hugging Face Blog