Chord Open-Sources an INT4 MoE Kernel for Kimi K2.x

Introduction

MoE inference is governed by more than the throughput of a matrix multiplication. The number of routed tokens assigned to each expert, padding behavior, and the ability to keep GPU warps occupied can all change which kernel schedule performs best. Novita AI’s open-source Chord project addresses that problem with CUDA kernels tuned for INT4 Kimi K2.x serving shapes.

Chord uses a W4A16 design: BF16 activations are multiplied by INT4 weights with group-32 scaling factors. The current codebase contains two independent kernel families. The indexed family is derived from the public Humming implementation and consumes vLLM routing tensors such as sorted IDs, expert IDs, and padded token counts. The second family, named grouped_contiguous for prefill and grouped_masked for decode, follows a separate SM90 design influenced by DeepGEMM. It uses grouped routing inputs and a different packed-weight layout.

Key findings

• Against the matching public Humming path, Chord reached roughly 1.11x–1.20x on H200 EP8 prefill and 1.17x–1.33x on H200 TP8 single-instance serving.
• H200 EP8 decode results were about 1.16x–1.24x, with the down projection reaching 1.31x in the reported measurements.
• On B300 EP8 decode, the reported range was 1.81x–2.15x. This comparison uses Humming’s default, untuned configuration because the public release has no SM100/SM103 tuning table, so it is not a tuned-versus-tuned comparison.
• The grouped H200 paths produced ranges of about 1.00x–1.31x for EP8 prefill and 1.16x–1.35x for EP8 decode, with additional tables for EP16 and EP32.

The central optimization principle is shape-aware scheduling. A single MoE kernel is unlikely to be ideal for both prefill and decode because routed tokens per expert can differ by orders of magnitude. Chord’s indexed resolver uses that quantity to influence block-M selection and occupancy decisions. In parts of the H200 prefill range, it sizes tiles around padded expert rows; for selected medium-sized tiles, a register limit is used to increase resident warps and hide gather and dequantization work. The implementation also overlaps weight loading, INT4 dequantization, and WGMMA operations through a pipelined main loop.

Integration and caveats

On compatible vLLM revisions, users can install Chord and explicitly select the Humming backend. The indexed path is designed to reuse the existing integration, while grouped integration with vLLM’s Humming backend remains a work in progress. The distribution supports the uint4, group-32, BF16-scale format and the compressed INT4 checkpoint layout used by Kimi K2.x. Unsupported schemes are intended to fail at load time rather than silently choosing an incompatible kernel.

The results should be read as kernel-level measurements, not promises of equivalent end-to-end acceleration. Real deployments will depend on expert parallelism, batch composition, routing balance, weight layout, GPU generation, and vLLM revision. Chord’s broader significance is that MoE inference backends are moving toward workload-sensitive scheduling instead of relying on one static kernel configuration for every request.

Source: vLLM Blog