LLM-Attacker Turns Split-LLM Privacy Testing into a Reproducible Process

Introduction

A split language model can keep the user’s raw text on the client while sending intermediate activations to a server. During training, gradients may travel in the opposite direction. Neither message should automatically be treated as harmless: both can contain signals that help reconstruct private text. The problem is compounded by the way privacy studies are usually compared. One evaluation may give the attacker activations only, another may include gradients; researchers may also differ in model knowledge, datasets, and definitions of successful recovery.

LLM-Attacker focuses on making the evaluation process inspectable rather than presenting only another extraction technique. The framework records the conditions of an experiment, allows compatible attacks to reuse captured messages, and stores predictions so that scores can be checked or recomputed independently.

Key points

• The attack surface is explicit. A report should state which messages the attacker sees and whether model weights or other system knowledge are available.
• Captured observations can be reused. Compatible attacks can work from the same recorded messages, reducing differences caused by changing inputs or data.
• Predictions remain available. Saving outputs, rather than only publishing aggregate scores, supports independent scoring and later measurement changes.
• Defenses need a linked baseline. A defense should be compared with an associated undefended reference before it is allowed to support a privacy claim.
• Untested attacks are visible. Reports identify gaps in coverage instead of presenting a limited test set as a complete security argument.

What the experiment shows

In a controlled GPT-2 experiment, giving the attacker gradients returned during training increased token recovery from 94.20% to 97.38%. Exact-document recovery increased from 13.71% to 37.77%. The token-recovery gain was evaluated against a two-percentage-point threshold fixed before testing, and its 95% interval remained above that threshold.

The result should be read as evidence about evaluation sensitivity, not as a universal leakage rate for every split model. The experiment changes what the attacker can observe and shows that this design choice can materially alter the measured privacy risk. The framework also includes further checks involving corpus selection, compression defenses, an adapted published attack and its reference implementation, and the measurement code. These activities distinguish implementation checks from measured attack effectiveness.

Why it matters

The main contribution is methodological. By separating attack effectiveness from implementation validation and evaluation coverage, LLM-Attacker makes privacy claims easier to audit. Developers are reminded that protecting raw client text is only one part of the design: activations, gradients, compression, and their combinations also deserve examination. Researchers gain a common record for comparing attacks and defenses.

The framework does not remove every limitation. Conclusions still depend on the attacker model, dataset, model configuration, and recovery metric. An attack that was not tested cannot be treated as ineffective. A credible privacy claim should therefore explain what the attacker saw, how recovery was scored, which controls were used, and which attacks remain outside the evaluation.

Source: Hugging Face Daily Papers