Why Poison Selection Can Matter More Than Poison Count

Introduction

Backdoor evaluations for language models often fix the number of poisoned examples and then sample those examples randomly from a candidate pool. That procedure is convenient and apparently controlled, but it can hide an important source of variation: poison examples do not contribute equally, and their interactions may depend on the set in which they appear. The paper Pick Your Poison argues that random selection can substantially underestimate a model’s worst-case vulnerability.

Key findings

• The poison budget is only part of the story. In three LLaMA-3-8B backdoor settings, the model, clean fine-tuning data, and number of poisoned examples were held constant. Yet attack success ranged from 3% to 80% solely because the selected poison set changed. A meaningful evaluation therefore needs to distinguish poison quantity from poison composition.
• Selection is a set-level optimization problem. The authors formulate the attacker’s task as finding a high-impact subset under a limited oracle budget. The ideal attacker would fine-tune and evaluate many candidate sets, but doing so exhaustively is too expensive.
• SAILS combines learning with auditing. SAILS, or Set-level Audit-Informed Iterative Learned Selection, first uses a few hundred fine-tune-and-evaluate runs to learn a scorer for complete poison sets. It then ranks millions of candidates and audits only a small shortlist, reducing the number of expensive experiments needed for search.
• The claim extends beyond one benchmark. According to the supplied material, SAILS improves held-out attack success by an average of 30 percentage points over the strongest influence-based baselines. It also transfers from small-scale to full-scale fine-tuning and is tested in code-generation, agentic, and API-only backdoor settings.

Why it matters

The main lesson is not simply that attackers can add more effective poisoned data. It is that the sampling strategy used in a safety benchmark can determine whether the benchmark measures average behavior or something closer to worst-case behavior. If experiments rely on random poison sets, a low attack-success rate may reflect an unlucky sample rather than genuine robustness.

For model developers, this suggests that backdoor testing should include set-level search, repeated evaluation, and checks across training scales. Defenders should also look beyond individual examples. A sample that appears harmless in isolation may become influential when combined with other examples sharing a trigger or target behavior. Dataset auditing therefore needs to consider composition and interaction, not only row-level anomalies.

SAILS also illustrates a practical compromise between exhaustive search and random testing: use a limited number of expensive training runs to learn where promising candidates are likely to be, then spend audit resources on that shortlist. The supplied material does not establish how stable the method is across additional architectures, training pipelines, or defenses, so broader independent replication remains important. Still, the study offers a clear warning: for LLM backdoors, choosing the poison set may be as consequential as choosing the poison count.

Hugging Face Daily Papers