Identical Training, Different Jailbreak Failures

The overlooked variable in jailbreak testing

Jailbreak benchmarks usually evaluate a single checkpoint and treat its successful attacks as evidence about the model’s training data or safety tuning. This study asks a more basic question: if the same data and safety-training procedure are run again with a different pretraining seed, will the model fail on the same jailbreaks?

The answer is often no. The study suggests that jailbreak behavior is tied not only to a model’s data recipe or safety objective, but also to the particular pretraining run that produced the checkpoint.

Experimental setup and main findings

The researchers used 25 pretraining data recipes. For each recipe, they trained models with three pretraining seeds and applied the same safety post-training with two additional seeds. This produced 150 models at 1B parameters. The same pattern was also observed at 150M and 300M parameters. Each model generated 20 responses to a shared collection of jailbreak prompts, allowing the researchers to compare non-refusal rates prompt by prompt.

The main results were:

• Two safety-trained copies from the same pretraining run had an average per-prompt correlation of 0.984.
• Changing only the pretraining seed reduced the correlation to 0.811.
• Changing the data recipe lowered it only slightly further, to 0.779.
• Among 91 attack templates, the template ranked best on one checkpoint remained best after a new pretraining seed in only 27% of model pairs, compared with 67% after a new safety-training seed.

The numbers indicate that safety-training randomness matters, but a fresh pretraining run can reshape the model’s jailbreak profile almost as much as changing the underlying data recipe. The best-template comparison should also be interpreted carefully: even two safety-trained copies of the same base run disagreed about the top template in roughly one-third of pairs. The result therefore points to instability in attack rankings, not to the claim that every attack changes completely.

A behavioral fingerprint for each run

The researchers also found that pretraining runs leave distinct behavioral signatures. Across 3,211 prompts, a model’s per-prompt non-refusal rates identified its source among 75 pretraining runs every time in the reported experiment. A random subset of 50 prompts still achieved 98% identification accuracy.

This has practical implications for model evaluation and procurement. A red-team report conducted on one checkpoint may contain a hidden run-level error: the result can reflect properties of that particular training run that are invisible from the model label alone. If a vendor silently refreshes its base model, previous jailbreak results may no longer describe the deployed system, even when the name and broad training recipe remain unchanged.

The fingerprint should not yet be treated as a ready-made vendor monitoring tool. The experiment matched models using the same post-training recipe and identified them among 75 known runs. Real updates may also change fine-tuning or safety procedures, and the study did not test whether the fingerprint alone can distinguish such updates. It is better understood as a trigger for renewed evaluation.

Why it matters

The study shifts the unit of analysis in jailbreak research from the general model family to the individual training run. Safety reports may need to record run-specific information and repeat testing after model updates, rather than relying only on a model name or a single benchmark score.

At the same time, correlations and attack-template rankings describe statistical behavior across prompts; they do not imply identical answers or provide a complete measure of deployment risk. The authors released code, weights, and per-prompt labels, making it possible for future work to test how well these findings transfer to larger models and real update pipelines.

Source: Hugging Face Daily Papers