Fish Audio Drama 3 Moves Speech Generation Toward Directed Performance

Introduction

Most text-to-speech systems are framed as a one-shot conversion: provide text, select or reference a voice, and receive an audio clip. Fish Audio’s Drama 3 aims at a broader target. Its technical report describes an open-source model designed to turn text into directed performances. In practice, that means the system is expected to handle not only speaker identity, but also performance instructions, scene context, revisions, and continuity across a sequence of turns.

What the model brings together

• Multi-reference voice cloning: Drama 3 can use multiple reference recordings to establish a target voice, rather than relying on a single example.
• Natural-language direction: Users can describe how a line should be performed in ordinary language. The supplied material does not provide a detailed set of examples, so the practical limits of this control still need to be verified through demonstrations and testing.
• Iterative revision: After a take is generated, a later instruction can act as a note on that take. This resembles a recording workflow in which a director asks for a new delivery instead of starting the entire session from scratch.
• Multi-character scenes: The model is intended for dialogue and dramatic scenes, not only for single-speaker narration.
• Speech infilling and streaming: Existing speech can be edited by filling in a selected region, while long-form synthesis and streaming text input support continuous playback and interactive applications.

Architecture and inference

Drama 3 is built on a 35-billion-parameter mixture-of-experts backbone, with roughly 3 billion parameters activated for each token. This design separates total model capacity from the computation used for an individual step. The backbone combines hybrid linear attention with a new codebook multi-token-prediction audio decoder. Dedicated decoder layers predict the acoustic codebooks for each frame while attending across the full sequence.

The architecture is intended to support both sequence-level coherence and efficient decoding. However, the supplied abstract does not include the full evaluation protocol, training-data scale, language breakdown, or detailed listening-study results. Parameter count alone therefore cannot establish that Drama 3 will sound better in every setting.

The report states that the inference engine decodes a single stream at 271 frames per second on one H200, compared with 177 frames per second for Fish Audio S2. That comparison is notable because Drama 3 has more than seven times as many parameters as S2. At the same time, frame rate should not be confused with end-to-end latency or a universal real-time factor: audio representation, context length, batching, and deployment settings also affect the user experience.

Why it matters

The main shift is from one-shot speech synthesis to an editable performance process. For audiobooks, games, short-form drama, dubbing, and interactive characters, one model handling voice identity, character turns, direction, and local corrections could reduce the need to move audio between separate tools. The same capabilities also raise familiar concerns around consent, impersonation, attribution, and labeling of generated voices.

Drama 3’s weights are available through GitHub and Hugging Face, and inference is supported through SGLang-Omni. More evidence is needed from the complete report and independent testing, but the abstract points to a meaningful change in the open-source speech landscape: the question is moving from whether a model can generate a voice to whether it can participate in a revisable, directed performance.

Source: Hugging Face Daily Papers