Serving◆ AI-generated · Sourced
Introducing the Realtime API: OpenAI’s Low-Latency Streaming Interface for Speech-to-Speech Interaction
TL;DR
OpenAI has launched the Realtime API, enabling millisecond-level end-to-end speech-to-speech streaming interactions—integrating audio input/output, bidirectional event channels, and built-in LLM response logic for real-time conversational applications.
Core Positioning and Capability Boundaries
The Realtime API is OpenAI’s new streaming service interface designed specifically for real-time voice interaction. Unlike traditional split ASR/TTS pipelines or text-only interfaces like GPT-4 Turbo, it unifies audio ingestion, LLM inference, and audio generation into a single low-latency streaming pipeline—with end-to-end latency under 300ms (median ~220ms in benchmarked conditions) and native support for persistent conversation state and dynamic prompt injection.
Key Technical Features
- Full-duplex audio streaming: Clients connect via WebSocket to simultaneously stream microphone audio (PCM/WAV, 16kHz mono) and receive synthesized speech (μ-law encoded, 8kHz), without requiring utterance segmentation or full-sentence buffering;
- Integrated voice stack: Powered by Whisper-v3 for real-time speech recognition, GPT-4o (2024-05-20 release) for response generation, and a proprietary TTS model (distinct from Azure Neural TTS or ElevenLabs), with voice selection support (currently 6 voices: alloy, nova, echo, etc.);
- Event-driven architecture: Defines 12 structured event types (e.g.,
response.audio.delta,conversation.item.created,input_audio_buffer.committed) enabling developers to inject custom logic at any stage (e.g., PII redaction, RAG trigger); - Conversation context management: Automatically maintains a tree-structured conversation item graph, supporting atomic operations including
add_message,create_response, andcancel_response—eliminating manual history concatenation errors.
Sources (compliance trail)
https://openai.com/index/introducing-the-realtime-api