Why English-Only Safety Tests Miss Multilingual Model Risks

Introduction

Large language model safety alignment has been developed and measured predominantly in English. That focus creates a practical blind spot: a model that refuses a harmful request in English may not respond safely when the request is expressed in another language, especially a low-resource language. Differences in phrasing, cultural context, topic distribution, and available training data can all affect the result.

A team from the Swiss AI Initiative proposes a template-based augmentation pipeline and releases MSafeTemp, a multilingual corpus intended to make safety training and evaluation broader and more culturally aware.

From reusable templates to multilingual data

The approach is more involved than translating an English dataset sentence by sentence. The pipeline extracts reusable safe templates from harmful prompts, translates those templates, and then fills their slots with entities that are considered culturally authentic. The slots can represent both harmful and benign content, allowing the process to generate refusal examples as well as ordinary, non-harmful requests.

The researchers also use verified cultural cards to guide localization. This is important because risk is not distributed identically across regions. A dataset built only around English-speaking contexts may omit locally relevant topics, names, institutions, or social situations. Template reuse provides scale, while localized filling is intended to add contextual diversity.

The resulting MSafeTemp corpus spans 60 languages and contains roughly two million multilingual safety alignment prompts. It is designed for both model training and evaluation, and the team has open-sourced the data and the associated pipeline to support further research.

What English-only evaluation misses

The central finding is that English safety scores are not a reliable proxy for multilingual safety. For some models, attacks that fail in English can succeed after being translated into another language. The gap becomes clearer when prompts are culturally localized: these variants can expose weaknesses that a direct translation does not reveal.

This suggests that multilingual safety is not simply a translation problem. A translated prompt may preserve the words while losing the assumptions, references, or social meaning that shape how a local user would express the request. Conversely, a culturally adapted prompt may activate behavior that is absent from a literal translation.

Why it matters

For model developers, MSafeTemp offers a practical way to increase the scale and diversity of safety data without writing every example from scratch. Templates can provide structural consistency, while language-specific and culture-specific entities broaden the situations covered by alignment training. The paper reports that this kind of augmentation improves the diversity and quality of alignment and leads to better safety performance beyond English.

For evaluators, the work argues for a wider test design. Safety reports should include multiple languages, low-resource settings, and localized scenarios rather than treating English performance as the default global measure. This is especially relevant when models are deployed across regions with different risk topics and user expectations.

Template-based generation is not a substitute for validation. Translation quality, cultural authenticity, harmful-content labeling, and the possibility of repeated template patterns still require scrutiny. MSafeTemp is best understood as an open foundation for that work: it places multilingual and culturally situated safety closer to the center of alignment research.

Source: Hugging Face Daily Papers