Turning Transcription Style into a Control Knob for Verbatim ASR

Introduction

Automatic speech recognition is usually described as the task of converting speech into text. In production systems, however, the harder question is often not only what was said, but how it should be written down. A speaker’s words can be transcribed verbatim, preserving fillers, repetitions, false starts and repairs, or they can be converted into an intended transcript that reads more cleanly. The nyra labs paper argues that modern ASR systems trained on mixed annotation sources often fail to control this distinction.

Key points

• Transcription style is a hidden source of error: The paper treats the choice between verbatim and intended transcription as a latent variable. If a model is trained on heterogeneous labels without being told which policy to follow, its output can fluctuate between preserving and removing disfluencies.
• WER can be confounded by annotation policy: According to the abstract, up to 60% of reported WER may be attributable to transcription-style mismatch. That means some measured “recognition errors” may actually be policy disagreements between the model and the reference transcript.
• The capability may already exist inside the model: The authors’ core claim is not that ASR models need to learn an entirely new behavior from scratch. Instead, they suggest that models already encode both styles, and the challenge is controlled activation.
• Task tokens provide an explicit control signal: The method trains coverage-aware decoder task tokens on parallel verbatim and intended transcript pairs. With English-only training, the system raises German disfluency F1 from 10% to 79% in a zero-shot setting, indicating that the control mechanism can transfer beyond the training language.
• Timing is part of the problem: Word-level timestamps are especially fragile when speech contains hesitations and repairs. The paper introduces supervised cross-attention fine-tuning and reports improvements over forced-alignment baselines on disfluent speech.
• A new data task: verbatimize: The authors also propose “verbatimize,” a task aimed at creating and enriching speech corpora with high-quality canonical verbatim transcripts.

Why it matters

This work is relevant for transcription products, captioning, meeting notes, medical documentation, speech data pipelines and TTS dataset creation. Many applications prefer readable intended transcripts, but audit trails, clinical workflows, linguistic analysis and training-data construction often require faithful verbatim records. A controllable ASR system could support both modes rather than forcing teams to choose one annotation style for every use case.

The paper also raises an important evaluation issue. If benchmark references mix policies, a single WER score may blur model quality with annotation convention. Making transcription policy explicit could lead to cleaner datasets, fairer ASR comparisons and more reliable downstream tools that depend on word-level timing.

Source: Hugging Face Daily Papers