MARS Uses Textual Refusal Directions to Steer Multimodal Safety

Multimodal large language models can interpret images and video, but that broader input space also creates a more difficult safety problem. Conventional alignment often depends on harmful examples collected for each modality, followed by post-training or activation-based steering. Such datasets are expensive to build, difficult to cover comprehensively, and potentially sensitive from ethical and legal perspectives.

The paper Harnessing Textual Refusal Directions for Multimodal Safety explores a different premise. Refusal may not be only a surface-level linguistic behavior; it may also correspond to a relatively stable direction in the model’s internal representation space. If so, a direction extracted from text could potentially influence responses to visual inputs as well. The authors report preliminary evidence that this transfer is possible, while emphasizing that it depends on how the intervention is applied.

Key points

• Refusal directions can transfer across modalities. The method derives refusal-related directions from the language-model backbone used by a multimodal model, avoiding the need to construct a separate multimodal safety dataset for every modality.
• Transfer is sensitive to implementation choices. Layer selection, steering strength, and cross-modal alignment all affect the outcome. Misalignment can push benign multimodal inputs toward refusal, creating a false-positive safety response.
• MARS adds selective control. Modality-Agnostic Refusal Steering uses activation centering to correct modality shifts, adjusts the steering magnitude inside a geometrically defined trust region, and chooses an intervention layer. The intervention is applied at the first generated token and requires no additional training.
• Safety must be evaluated with utility. Across five state-of-the-art multimodal language models, the evaluation covers safety, utility, and video jailbreak settings. MARS shows consistent safety improvements while retaining utility until over-refusal becomes significant.

Why it matters

The broader contribution of MARS is a hypothesis about where multimodal safety may live. Some safety-relevant structure could already be shared through the language backbone, even when the original input arrives as an image or a video. If that pattern holds across more architectures and risk categories, safety engineering may not need to build entirely separate alignment pipelines for every modality.

The result should not be interpreted as evidence that a refusal direction provides complete safety understanding. Strong steering can reduce useful responses, and representation mismatch can cause benign requests to be rejected. Video also introduces temporal context and combinations of events that a single-token intervention may not fully capture. MARS is therefore better viewed as a lightweight safety enhancement than as a replacement for data-driven alignment and evaluation.

From an engineering perspective, its appeal lies in parameter-free deployment and the ability to intervene without retraining the model. Future work will need to test its stability across model families, real-world use cases, and more nuanced safety policies. A central challenge will be moving beyond a binary refuse-or-answer behavior toward more accurate risk judgments and constructive responses.

Source: Hugging Face Daily Papers