Inside GPT-Live: How OpenAI Builds Stateful, Low-Latency Voice Interaction

Introduction

A voice assistant is judged not only by whether it understands speech, but also by whether the conversation remains fluid while tools, external services, or background tasks are running. In its GPT-Live engineering report and related interview, OpenAI describes an architecture built around a simple principle: keep the voice path short and predictable, then isolate the rest of the application behind an asynchronous boundary.

A deliberately narrow real-time path

GPT-Live’s latency-sensitive path contains the media pipeline and the inference loop. Receiving and processing audio, together with generating the model response, must continue without being blocked by work whose duration can vary. Task delegation, tool use, persistence, and other application logic therefore run after an asynchronous RPC boundary.

This separation allows engineers to optimize the components that users feel immediately, rather than allowing a slow external dependency to stall the audio channel. It does not make the other components unimportant. Delegating work to a more capable model and connecting speech input to safety systems still require dedicated designs. The architectural point is that they can be improved independently instead of becoming part of the most timing-sensitive path.

Stateful inference with context migration

Continuous dialogue requires state, yet pinning every conversation permanently to one machine makes scaling and recovery harder. GPT-Live uses a dedicated stateful inference mechanism. Each session reserves capacity on an assigned instance. If that instance is being drained, or if a conversation is approaching its context limit, the system can move the session context to another model instance and direct new sessions toward available capacity.

The approach balances continuity with operational flexibility. A conversation does not have to restart simply because infrastructure is being rebalanced, while the platform can adjust the number of instances as demand changes. The trade-off is additional operational complexity: capacity reservations, migration timing, and context consistency all need careful management. This is more involved than operating a completely stateless inference service, but it better matches the requirements of ongoing voice interaction.

Why keep WebRTC?

Rather than replace WebRTC, OpenAI retained it as the media foundation and introduced the WebRTC Abridged Roundtrip Protocol, or WARP, together with an instant-connect mechanism to reduce startup delay. WebRTC already offers a battle-tested low-latency media stack with built-in recovery behavior. Newer transport efforts may be promising, but they may cover only the transport layer and still lack parts of a complete media pipeline, including congestion control and RTT-based path selection.

WARP’s improvements can be deployed and evaluated separately, and existing WebRTC applications can benefit without code changes. That incremental strategy lowers the risk of replacing client and server transport stacks at the same time, an important consideration for a large-scale real-time service.

Testing with real traffic, without changing user output

Before launch, OpenAI used a “silent test.” The application service ran in read-only mode without user credentials; real inbound voice streams were sent through the media and inference systems, while generated responses were discarded. This made it possible to test production behavior without affecting what customers heard.

Compared with prerecorded or synthetic speech, real traffic contains broader variation in speech patterns, geography, and load. The exercise exposed latency caused in some regions by GPUs and the CPUs feeding them being deployed in different facilities. That example shows why real-time AI bottlenecks can come from deployment topology, not only model speed. For similar products, isolating the real-time path, supporting state migration, and validating changes with representative production traffic may be as important as improving the model itself.

Source: InfoQ 中文