HarmProfile: Why Frontier LLM Safety Needs a Risk Distribution

Introduction

Many LLM safety evaluations reduce a failure to a binary event: a model either refuses a harmful request or a jailbreak succeeds. This is useful for measuring attack success, but it leaves a larger question unanswered: what does a model actually produce when its safeguards fail? HarmProfile shifts attention from the attack alone to the distribution of harmful outputs.

Key findings

• Safety failures are treated as distributions. The benchmark characterizes harmful outputs by their content, severity, and variation. This creates a model-level risk profile that captures more than the number of successful attacks.
• A large cross-model corpus is provided. HarmProfile includes more than 80,000 validated artifacts collected from 23 frontier LLMs and 13 model families. The artifacts are organized into 15 harm categories and 57 subcategories, making cross-model comparisons more systematic.
• Models fail in different ways. Frontier LLMs can reliably generate harmful content at scale under failure conditions, but their outputs are not identical. Distinct models show distinct risk profiles, suggesting that aggregate safety scores may conceal meaningful differences in the types of content they produce.
• Capability and risk complexity may grow together. The study reports that both harmfulness and harmful-output diversity increase with model capability. A model can therefore appear well aligned in ordinary interactions while retaining a broader and more varied set of dangerous knowledge beneath that surface.

Why it matters

The main contribution of HarmProfile is a change in the unit of analysis. Refusal rates and jailbreak success rates remain useful, but they do not reveal the composition of a model’s failures. A structured corpus can help researchers compare model families, identify concentration in particular harm areas, and design more targeted red-team and alignment evaluations.

The benchmark should not be treated as a final safety verdict. Its conclusions will depend on how samples are collected, validated, and categorized. It is best used alongside attack-success metrics, refusal-quality assessments, and scenario-specific testing. The broader lesson is that model safety is not only about whether a system refuses; it is also about which directions its failures take, how severe they are, and how varied the resulting outputs can become.

Source: Hugging Face Daily Papers