VibeVoice-ASR-Streaming Brings Real-Time “Who Said What” Transcription

Introduction

For meetings, support calls, and conversational agents, converting speech into text is only part of the problem. A useful transcript also needs to indicate who said each utterance. Conventional speaker-attributed ASR systems commonly divide the workflow into automatic speech recognition and speaker diarization. That separation can work well offline, but it makes low-latency interaction harder because the system may need to wait for more audio or for a second processing stage to finish.

VibeVoice-ASR-Streaming, presented by a Microsoft research team, takes a unified end-to-end approach. Rather than transcribing first and attaching speaker labels later, the model is designed to produce “who said what” while the conversation is still unfolding.

Key points

• Streaming input design. The model processes fixed-size audio chunks and adds a small amount of lookahead audio, allowing it to balance immediate output with limited future context.
• Text history as context. Previous generated text is interleaved with incoming audio so the model can maintain continuity across successive chunks.
• Joint speaker-attributed ASR. Recognition and speaker attribution are handled within one model, removing the need for a separate diarization stage in the proposed pipeline.
• Two released model sizes. The project provides 1.5B and 7B model weights, along with inference code for experimentation and integration.
• Strong reported results. The report says the 7B model achieves the lowest average WER/CER over five evaluation sets and reaches the best or tied-best result in 12 of 13 speaker-attribution settings.

Why it matters

The most direct application is real-time voice interaction. A voice assistant or agent can receive a speaker-labeled transcript before a conversation ends, creating more timely input for meeting notes, live summaries, call analytics, and multi-party workflows. A unified model may also reduce the coordination burden between recognition, segmentation, diarization, and transcript alignment components.

The approach still involves practical trade-offs. Lookahead audio can improve decisions but introduces some delay. Long conversations require stable context handling, while overlapping speech, background noise, language variation, and limited device compute can all affect deployment behavior. The reported aggregate benchmarks are encouraging, but teams will need to test the released models under their own acoustic and latency constraints.

The broader contribution is a shift in how streaming speech systems are structured. Instead of treating speaker identity as a downstream label attached to text, VibeVoice-ASR-Streaming attempts to model content and attribution together as speech arrives. With weights and inference code available, it offers a useful foundation for research into end-to-end, multi-speaker voice agents.

Source: Hugging Face Daily Papers