A Scaling Law for Synthetic Rewrites Links Data Mix to Compute

Introduction

Synthetic data is now a routine part of large-language-model pretraining. A model can rewrite a natural document into a target style, format, or distribution, and the resulting text can be added to the training mixture. Yet two practical questions remain unresolved: how much compute should be spent producing those rewrites, and can a model trained on them eventually outperform the rewriting model itself?

The arXiv paper Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints treats both questions as a scaling problem.

What the study models

The authors propose a rewriting scaling law that predicts training loss as a function of:

• target-model size;
• the number of training tokens;
• the fraction of tokens that are synthetic rewrites;
• the number of passes over the natural dataset; and
• the compute required to generate the rewritten data.

Including generation cost is the central methodological choice. Synthetic tokens are not free: producing them consumes inference compute, and that cost should be compared with the compute used to train on natural data. The law is therefore intended to identify an end-to-end tradeoff, not merely the best mixture after a synthetic corpus has already been created.

In the compute-limited setting, natural data is assumed to be plentiful. Rewrites come from a third-party rewriter trained with more compute than the largest target-model run. The estimated optimum shifts substantially with the budget. At roughly 3×10^18 FLOPs, the best rewrite fraction is below 10%; at 10^22 FLOPs, it reaches 32%.

The researchers then extrapolate the fitted law to a 22-billion-parameter model. After a 33-fold increase in training FLOPs, the observed final loss is within 1.1% of the prediction. The resulting model also achieves better bits per byte than its rewriter on several web domains, while using one-tenth of the rewriter’s training compute. This is evidence that a downstream model need not simply inherit the ceiling of the system that generated its training text.

When natural data is scarce

The second setting assumes a fixed seed set of natural data. The rewriter is the same size as the target model and is pretrained on that seed set. Here too, the estimated optimal rewrite fraction increases with the available compute budget.

The study further reports that models trained on rewrites of their own pretraining data, generated by a same-size rewriter, outperform the best natural-data-only run at every tested model size up to 3 billion parameters. The result does not imply that all synthetic data is beneficial; rather, it suggests that a carefully chosen rewrite process can change the effective training distribution in a useful way.

Why it matters

For practitioners, the main lesson is that synthetic rewriting should be budgeted jointly with model training. A high rewrite fraction may be wasteful at a small budget because generation costs dominate. As the budget grows, however, increasing the synthetic share can become the more attractive allocation.

The paper also weakens the simple teacher-student intuition that a trained model cannot exceed its data generator. Rewriting can reshape data for the target objective, allowing training to extract value that is not directly visible in the rewriter’s own evaluation.

The available material is an abstract-level summary, so the generality of the law across tasks, prompts, architectures, and data-quality regimes remains uncertain. Risks such as bias amplification, duplicated information, and factual errors also require separate evaluation. The safest interpretation is that synthetic rewriting is not free data expansion; it is an optimization problem spanning data quality, generation cost, and training compute.

Source: arXiv