SecOPD Uses Token-Level Distillation to Mitigate Adaptive Prompt Injection

Why prompt injection remains difficult

AI agents increasingly read webpages, files, and emails before deciding what to do. Those sources may contain more than useful information: an attacker can insert instructions that tell the model to ignore its original task and perform an unauthorized action. Because the malicious instruction is embedded in otherwise legitimate external data, simple input filtering or keyword matching is unlikely to address every variation. Adaptive attackers can also modify their wording after observing how a defense behaves.

The limitation of sequence-level feedback

Many defensive fine-tuning recipes use preference or policy-optimization methods such as DPO or GRPO. Their feedback is commonly applied at the level of an entire response. A response may therefore be treated as broadly good or bad, even though only a small portion contains the unsafe behavior. This coarse signal does not clearly tell the model which output decisions should be changed.

The paper argues that this credit-assignment problem helps explain why existing defenses can remain vulnerable to adaptive prompt injections. A model may learn a general refusal pattern without learning which local generation choices allow an injected instruction to redirect its behavior.

How SecOPD works

Secure On-Policy Distillation, or SecOPD, moves the training signal to the token level:

• The model receives an example containing an injected instruction and produces an on-policy rollout.
• The initialization model evaluates the rollout tokens while being given the corresponding clean input.
• These token-level scores are used to guide defensive fine-tuning.
• Because the rollout is generated from the attacked context, training focuses on the behavior the model may actually exhibit under injection rather than only on static demonstrations.

The underlying idea is to distinguish the safe and unsafe parts of a response more precisely. Instead of labeling an entire completion as wrong, the method attempts to identify the local decisions that should be discouraged or recalibrated.

Reported results

According to the paper, SecOPD-trained Qwen3.6-27B reaches a 9.0% attack success rate against the PISmith state-of-the-art adaptive prompt injections. The cited prior baseline, Meta-SecAlign, records 94.0% in the same comparison. In an agentic tool-calling domain that was completely unseen during training, SecOPD reaches 4.7%, compared with 5.5% for Meta-SecAlign.

These figures suggest that finer-grained feedback may improve robustness beyond memorized refusal examples. The unseen tool-calling result is particularly relevant because real agents often transform model outputs into external actions, making a successful injection more consequential than an unsafe textual answer alone.

Significance and limitations

SecOPD frames prompt-injection defense as a credit-assignment problem. If a model can learn which part of its generation opened the door to an injected instruction, safety fine-tuning may become more targeted and less dependent on treating every response uniformly.

The reported numbers should nevertheless be interpreted within the scope of the supplied material. They come from particular evaluations, models, and attack settings, and do not establish universal protection for deployed agents. Attack capabilities, tool permissions, context construction, and the definition of attack success can all affect the outcome. Operational systems should still combine model training with least-privilege tools, confirmation gates, clear separation of external data from instructions, and runtime monitoring. The authors provide code and a model, enabling further reproduction and testing across environments.

Source: Hugging Face Daily Papers