Decode Before You Hear: Speculative Listening Speeds Up Duplex Speech Models

Introduction

Real-time speech interaction is constrained not only by recognition quality, but also by when a model can begin producing its response. Many full-duplex speech language models process conversation in fixed-duration chunks. They listen to one chunk, generate or act on it, and then move to the next. This schedule creates an uneven workload: computation may sit idle while the system waits for incoming audio, while speaking and tool-call phases must generate tokens serially and can miss real-time deadlines.

A team from The Chinese University of Hong Kong, Shenzhen proposes Speculative Listening, a way to use that idle period. Before the next audio chunk arrives, the model drafts a possible continuation while treating the incoming chunk as silence. Once the actual audio is available, the system verifies the draft and reuses the matching portion.

The observation behind the method

The approach is based on measurements from four models. When the next chunk was replaced by silence, the model produced the same output as it did with real audio in 95.3% of chunks. When the output did change, 88% of those divergences started with the first output token.

These results suggest that the next audio segment often does not affect the immediate computation, and that the exceptional cases can frequently be detected at the beginning of generation. Speculative Listening therefore drafts the upcoming chunk during the listening phase. If the first-token decision is uncertain, it also creates a second branch from the runner-up candidate. The drafts are stored in a cross-chunk hypothesis pool and verified together when the actual audio arrives.

The procedure is training-free. It does not ask the model to learn a new prediction task, and verification preserves the output that would have been generated from the real input. In effect, the system moves likely computation earlier rather than accepting an approximate answer.

Results and implications

On the eight-task TickTock-Bench evaluation, the method accelerated speaking chunks by 2.6–2.8× and tool-call chunks by as much as 23.4×. The average decoding delay before a response began fell by 73%, reaching 74–241 milliseconds. That range is close to the roughly 200-millisecond turn-taking gap cited for human conversation. Maximum lag behind real time was reduced by 94.4%.

The broader implication is architectural. Lower latency may come not only from a faster model or more hardware, but also from scheduling computation around the model’s input timeline. Idle listening periods become an opportunity to prepare speech or tool actions before they are urgently needed.

The gains will depend on chunking choices, model behavior, and the cost of maintaining and verifying multiple hypotheses. Even so, the technique offers a practical path for speech assistants, real-time translation, and tool-using agents that need to respond while a conversation is still unfolding. Instead of waiting to hear everything before decoding, a system can listen and prepare in parallel.

Source: Hugging Face Daily Papers