OpenAI DevDay 2024 | Multimodal apps with the Realtime API

OpenAI’s Realtime API is built to deliver natural, low-latency “speech-in, speech-out” experiences through a single interface—removing the multi-step glue work that previously made voice assistants feel slow and brittle. Instead of stitching together separate systems for speech capture, transcription, text generation, and text-to-speech, the Realtime API uses a model that natively understands and generates speech. That design enables faster turn-taking, more fluid interruption behavior, and richer audio nuance because the system can process audio directly rather than converting everything into text first.

The session contrasts the “old way” with the new approach. Traditional pipelines typically require a user-gesture or end-of-speech detection step, then transcription (using OpenAI’s Whisper model), then a language model response (example given: GPT-4o), and finally a third model to synthesize speech. Each stage adds delay and can degrade conversational flow—especially when users interrupt mid-response. The Realtime API collapses those steps into a single real-time stream, leveraging the same underlying speech capability that powers advanced voice mode in ChatGPT.

A key technical shift is how the API maintains responsiveness: apps open a stateful WebSocket connection to a Realtime endpoint (V1 realtime). Audio input can be streamed as it’s spoken, while model output audio is streamed back as soon as it’s produced. Messages use JSON types that include audio deltas for playback and events that signal when the user starts speaking. That event support is what makes interruptions practical: when a user cuts in, the client stops the currently playing audio and sends interruption metadata (such as the played offset and context identifiers) so the model can align its next output with what was actually heard.

The session also highlights practical developer building blocks. A live coding walkthrough shows a browser-based client using the Web Audio pipeline: it decodes PCM16 audio (24 kHz) returned by the API, plays it, and records microphone audio to send back as input audio buffer append messages. The API supports different audio codecs (including G711), and it can handle function calling alongside speech.

To demonstrate real-world integration, a 3D solar-system tutoring app uses tool calls to trigger visual changes when users ask about planets and to display charts when answers depend on structured data. The app also fetches live information—such as the International Space Station’s current latitude/longitude from a real-time endpoint—and feeds that data back into the conversation via a tool workflow. The model can stream tool-related information progressively, such as showing Pluto’s moons while the tool call expands.

Finally, the session addresses cost and rollout. The Realtime API is in public beta, currently supporting speech, text, and function calling, with more modalities planned. It also introduces prompt caching for text and audio inputs: cached text inputs cost 50% less, cached audio inputs cost 80% less, and a typical 15-minute conversation is expected to cost about 30% less than at launch. The overall message is that speech-to-speech apps can be built with lower latency, better interruption handling, and deeper app integration—setting the stage for more natively multimodal capabilities in future frontier-model releases.