QuadTok Uses Quadtrees to Rethink Visual Tokenization for Image Generation

Introduction

Visual generative systems commonly begin by converting an image into discrete or continuous visual tokens before handing them to a sequence model. A fixed grid is convenient, but it assigns roughly the same representational budget to every part of an image. A uniform sky or wall may receive as many tokens as a face, object boundary, or textured surface. QuadTok proposes a different allocation strategy: organize image regions as a hierarchical quadtree and refine only where additional detail is needed.

A content-adaptive visual representation

The tokenizer starts with coarse spatial regions and recursively splits selected areas into smaller ones. Visually intricate regions can therefore receive finer-grained nodes, while relatively homogeneous regions remain represented at a coarser level. The resulting token set is not simply a regular collection of patches; it also encodes a hierarchy of spatial relationships.

This design addresses a tension between two conventional representations. A two-dimensional grid preserves locality and spatial binding, but it is less naturally aligned with sequence-based autoregressive modeling. A one-dimensional token stream is convenient for a GPT-style model, yet flattening can weaken the original spatial organization. QuadTok attempts to combine both properties: parent-child relationships preserve the image’s spatial hierarchy, while the tree can be traversed as an ordered sequence for autoregressive prediction.

Reported results

The paper summary reports that the ImageNet-trained QuadTok tokenizer saves approximately 10% of tokens on ImageNet compared with a fixed 256-token grid. When transferred zero-shot to COCO, it saves approximately 9%. Reconstruction fidelity remains comparable according to the authors’ summary, suggesting that part of the regular grid’s budget may be redundant in visually simple regions.

For generation, the team trains a 947M-parameter GPT-style model conditioned on a quadtree topology supplied before sampling. The model obtains a 2.08 gFID on the ImageNet 256×256 benchmark. The authors also report zero-shot spatially controlled image generation, attributing this capability to the strong spatial correlations retained by the quadtree representation. The supplied material does not include the detailed control protocol, baselines, or a full evaluation table, so the scope of that capability requires further inspection of the paper.

Why it matters—and what remains open

QuadTok’s main contribution is not merely token reduction. It makes the spatial decomposition itself part of the generative interface. The topology can act as a coarse layout or scaffold, after which the autoregressive model fills in visual content across different regions and levels. This could make compute allocation more selective and create a natural interface for region-aware editing or layout-conditioned generation.

The approach also introduces practical questions. A topology must be available or specified before generation, so topology prediction and topology errors may affect final quality. Variable-length tree sequences and traversal rules could complicate batching, training, and inference compared with fixed grids. In addition, the reported evidence centers on ImageNet and COCO; it does not yet establish how the method behaves at higher resolutions, in more crowded scenes, or across broader generation tasks. Even so, QuadTok presents a clear alternative to uniform visual tokenization: let the token structure adapt to the image rather than forcing every region into the same spatial budget.

Source: Hugging Face Daily Papers