Split-LLM Training Has a Gradient Privacy Blind Spot

Introduction

Split-LLM training is often presented as a way to keep sensitive text on a trusted device while outsourcing part of model computation to the cloud. In that design, privacy reviews commonly focus on the forward wire: whether the activations sent to the cloud can be mapped back to the original input. A new systems-security case study argues that this view is incomplete. The backward wire can expose the structure of the private batch even when the forward-channel check passes.

How the leak appears

The examined system has a Trusted Local Node (TLN) and an Untrusted Cloud Node (UCN). The TLN sends protected activations to the UCN, which runs its portion of the model and returns the output. Because the loss and labels remain local, the TLN computes the output gradient and sends that gradient back to the cloud.

To create a larger anonymity set, the transmitted frame mixes real rows with synthetic decoys. The problem is that the loss ignores the decoys. Their gradients are therefore exactly zero, not merely small or noisy. The cloud can inspect the returned tensor and distinguish rows with nonzero support from rows with zero support. The mechanism intended to hide the real examples consequently reveals their positions.

What the experiments showed

• The researchers fixed the evaluation protocol in advance and injected a leak of known strength to verify that the measurement could detect a real signal.
• Across nine seeds, the zero pattern identified the real rows in every frame: 4,096 out of 4,096 per run.
• An attack aimed at frame contents recovered roughly one additional token per hundred compared with a constant-guess baseline, with gains ranging from 0.65 to 1.50 percentage points.
• Shuffled-label controls produced no recovery, helping rule out a detector that simply reports leakage everywhere.
• The same behavior appeared in a configuration that kept model quality within the stated budget, so the issue was not limited to an unusable research setting.

The most important evaluation result is procedural. The system passed both the forward-channel privacy check and the quality check. Once the returned gradient was included, it failed that same privacy check. A review limited to activations therefore gave a misleadingly positive result.

Mitigation and broader implications

Clipping and adding noise to every gradient row, including the decoys, closed the tested structural leak at a cost of about 0.01 nats of held-out cross-entropy. That is a useful mitigation, but not a complete security guarantee. Gradients can still carry information about private inputs or labels, and repeated training steps may allow an observer to separate persistent real rows from randomly generated decoys through temporal statistics.

The broader lesson is that the privacy boundary in split training must cover the entire bidirectional protocol, not just whether raw text leaves the local node. Audits should inspect activations, gradients, masking rules, support patterns, and cross-step correlations. They should also include fixed controls that distinguish an actual leak from an artifact of the evaluator. Decoys do not create anonymity by themselves: unless their computational treatment is indistinguishable from that of real rows, the training protocol may disclose their identity through its gradients.

Source: Hugging Face Daily Papers