How Peer Pressure Breaks Conformal Guarantees in Multi-Agent LLMs

Introduction

Conformal prediction is often used to attach a statistical safety layer to model outputs. Under its assumptions, a prediction set should contain the correct answer at or above a chosen rate. This makes the method useful not only for producing answers, but also for deciding when a system should abstain, escalate, or request human review.

A new arXiv paper identifies a failure mode in multi-agent LLM systems. A model may satisfy its conformal guarantee when answering independently, yet lose that guarantee after seeing other agents’ responses. The question, correct label, and task distribution remain the same. What changes is the way the model scores candidate answers.

Key findings

• The shift is behavioral, not necessarily distributional. The authors call the phenomenon a “score-mechanism shift.” Calibration data collected from isolated answers describes one scoring process, but says little about the model’s scoring process after it receives social evidence from peers.
• Unanimous mistakes reduce coverage. Across open-weight models and multiple-choice question-answering tasks, coverage at the standard alpha=0.10 operating point fell from a calibrated 90% to 74% when peers unanimously supported an incorrect answer.
• Aggregate metrics can hide targeted failures. When an attacker targets items that still appear covered but carry relatively low confidence, coverage falls from 87% to 47%. The overall average can therefore make a selective attack look less severe than it is.
• The failure reaches the action policy. A system that should escalate when uncertain may become sufficiently confident after exposure to wrong peer answers and proceed with the attacker’s choice.

Why standard fixes may not work

Typical conformal remedies focus on recalibrating thresholds, changing the significance level, or handling shifts in the input distribution. Those interventions do not directly address this case. The questions have not changed; the model’s response mechanism has changed because the interaction context has changed.

This distinction matters for multi-agent designs. Voting, debate, and cross-checking are often introduced to improve reliability. Yet a group of agents can also create a channel for correlated error. If several agents confidently repeat the same wrong answer, the target model may interpret agreement as evidence and alter its internal ranking of choices. Collaboration then becomes a source of pressure rather than correction.

Monitoring average coverage is also insufficient. A selective adversary does not need to damage every example. It can search for cases where the certificate is fragile, then supply misleading consensus only there. Subgroup analysis based on confidence, peer agreement, and escalation status is therefore essential.

Implications

The study does not show that conformal prediction is useless for LLMs. It shows that its guarantee must be tied to the information environment in which calibration was performed. A deployed system may need separate evaluations for isolated answering, exposure to peer answers, unanimous agreement, and other communication patterns.

For safety engineering, the relevant question is not simply whether a model is calibrated. It is whether the model remains calibrated under the social and contextual inputs it will encounter in operation. Multi-agent systems should test score changes, selective attacks, and escalation behavior—not just final-answer accuracy or average coverage. Otherwise, a statistical certificate can create confidence precisely when the system most needs caution.

Source: arXiv