NVIDIA Nemotron 3.5 Brings Promptable Multilingual Speech AI

Introduction

Speech applications rarely stop at converting audio into words. A meeting system must identify who spoke, a global support call may need translation, and medical dictation has to preserve specialized terms. NVIDIA’s Nemotron 3.5 Transcribe is designed as a single open-weight model for these related tasks, with natural-language prompts controlling how the same recording should be analyzed or formatted.

What the model offers

• Several tasks in one checkpoint: Nemotron 3.5 supports transcription, speaker attribution, translation, and audio question answering. It transcribes 40 languages, translates 31 languages to or from English, and provides speaker attribution for 13 languages.
• Long and multi-speaker audio: The model handles recordings up to about one hour and can label up to eight speakers. Speaker cues are supplied alongside acoustic features to the language model, allowing attribution during generation rather than relying entirely on a later post-processing stage.
• Prompt-controlled output: Developers can ask for a speaker-labeled transcript, an English translation, or an answer about decisions made during a meeting. For unfamiliar terminology, a prompt can provide candidate words while instructing the model to use them only when supported by the recording.
• A path to specialization: Granary v2 includes multilingual data and a reusable curation pipeline, while NeMo Speech provides fine-tuning recipes. In NVIDIA’s medical case study, fine-tuning reduced word error rate on held-out speakers by 45.8% relative to the best prompting setup.

Evaluation and caveats

The source article reports a 5.33% average word error rate on the English track of the Hugging Face Open ASR Leaderboard among open-weight models. It also cites 4.62% on a seven-language multilingual comparison and 10.98% on a long-form comparison. Artificial Analysis reports a 2.8% AA-WER, with roughly 4B active parameters out of about 30B total parameters. The language model uses a hybrid Mamba–Transformer mixture-of-experts design.

For diarization, NVIDIA says the model recorded the lowest error rate in David AI’s 21-language comparison and ranked first in VoiceArena’s 14-system benchmark. These results should be read in context: each leaderboard uses different data, metrics, and participating systems, so they do not prove universal superiority in every deployment. The model guidance also recommends checking names, speaker switches, and audio-question answers against the original recording.

Why it matters

Nemotron 3.5 connects two forms of control that are often separated: developers can describe the desired output in a prompt, then fine-tune the system when prompts are no longer sufficient for specialist vocabulary, accents, or writing conventions. This creates a practical progression from quick experimentation to domain deployment.

The model is available through Hugging Face and ModelScope. NVIDIA says quantized versions are planned for local devices, while larger deployments can use data-center GPUs. For classroom replays, meeting intelligence, multilingual communication, and medical documentation, the combination of task breadth, promptability, and an open adaptation pipeline makes it more than a conventional transcription endpoint.

Source: Hugging Face Blog