BYOD Uses LoRA to Turn Autoregressive Models into Diffusion LMs

Introduction

Diffusion language models generate text differently from conventional autoregressive systems. Instead of predicting tokens strictly from left to right, they begin with a fully masked sequence and progressively denoise several positions at once. This can enable bidirectional context, flexible decoding schedules, and a tunable inference budget. The trade-off is that building a diffusion language model from scratch, or fully fine-tuning an existing model, can be expensive.

BYOD, short for Build Your Own DLM, explores a cheaper alternative. It keeps the pretrained autoregressive model fixed and trains LoRA adapters to induce diffusion-style generation.

Key findings

• The recipe was applied to Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B. The authors describe the Ministral result as the first reported AR-to-DLM conversion for that model family.
• All pretrained weights remain frozen. The adapters account for only 4.1% to 5.8% of the final parameter count, avoiding full-model fine-tuning and architecture-specific conversion work.
• Each primary conversion ran for 25,000 iterations with at most 102.4 million tokens. The reported runs took 3.1 to 4.0 hours on a single Google Colab GPU; the paper experiments used a 96 GB NVIDIA RTX PRO 6000.
• Every converted model can generate from a fully masked sequence. Controlled attention tests also indicate that the models can use bidirectional context, while retaining a substantial amount of knowledge from their parent models.
• BYOD-Gemma achieved 50.6% average accuracy across nine tasks, close to the 51.9% reported for LLaDA 8B Instruct in the same evaluation harness.

The remaining gap

The method’s strongest case is accessibility, not outright superiority over established diffusion models. Open-ended generation remains the main limitation. When generating 128 tokens in 64 steps, BYOD models recorded perplexities of 8.3 to 11.1, compared with 4.1 for LLaDA and 2.2 to 3.0 for their autoregressive parents. The adapters therefore appear to teach the mechanics of diffusion generation, but the resulting models still struggle to match the fluency and distributional fit of their reference systems at relatively low network function evaluation budgets.

Why it matters

BYOD reframes AR-to-DLM conversion as an adapter-training problem rather than a full pretraining project. Researchers can test different base models, datasets, and decoding strategies without altering the original weights. Developers can also reuse the knowledge embedded in instruction-tuned models while investigating parallel updates and variable denoising schedules.

The approach does not eliminate practical constraints. Large parent models still require considerable memory, and the paper-scale configuration uses a paid GPU. More importantly, the balance among generation quality, denoising steps, and speed is unresolved. BYOD is best understood as an accessible research platform, not yet a replacement for autoregressive decoding. The accompanying notebook, code, and browser demos make that platform easier to inspect and extend.

Source: Hugging Face Daily Papers