Introducing gpt-realtime in the API

OpenAI is rolling out a new GPT-realtime speech model and an upgraded real-time API in general availability, aiming to make voice interactions with AI feel more natural, more reliable, and low-latency enough for real-world agents. The centerpiece is a speech-to-speech model that natively takes in audio and produces audio—avoiding the usual split between transcription and a separate voice system. That design choice is positioned as faster (one model), more expressive (it can capture laughs, sighs, and a wider emotional range), and more flexible (it can switch languages mid-sentence), which matters for customer support, tutoring, and healthcare-style conversations where tone and timing affect outcomes.

A live demo highlighted three practical capabilities. First, the model’s emotional range and voice quality: it delivered a lottery-ticket scenario with clear shifts from upset to excitement, then produced a short rhyming poem while switching between English, Spanish, and Japanese. Second, instruction following under constraints: when given a policy not to issue refunds above $10, it stayed within the limit even as a user escalated the stakes—claiming a boss was watching and urging an exception. The response remained politely evasive, steering toward alternatives rather than breaking the rule. Third, the system gained image input in the real-time API, letting developers send an image for the model to “see what you see.” In the demo, it described a child near a stuffed unicorn and offered safety-oriented guidance, tying visual details to actionable advice.

Behind the scenes, OpenAI attributes the improvements to post-training and data quality. The speech model was trained with high-quality voice data plus specialized reward models to increase naturalness, alongside reinforcement learning designed to be sample efficient and more effective at shaping behavior. Instruction following is also described as more steerable—supporting adjustments to pace and tone, roleplay, and better adherence in hard multi-turn instruction scenarios. On benchmarks, the new audio instruction-following performance is reported as over 30% accuracy on an audio version of the Scale Multi-Challenge evaluation, and function calling reaches 66% accuracy on complex audio phone-bench scenarios.

On the platform side, the real-time API’s GA adds features aimed at production deployment: image input, EU data residency, asynchronous function calling, and improved context management tools that work better with caching. It also adds SIP telephony support for phone-based voice applications and introduces MCP support, described as a way to plug capabilities into the model—particularly well-suited to voice because the system can interpret what it hears and trigger MCP tools in a natural conversational flow.

T-Mobile joins as a real-world test case, using the model to streamline a device upgrade process. The assistant guides customers through a branching set of questions—eligibility, device selection, and plan compatibility—while maintaining a more responsive, human-like interaction. T-Mobile’s leadership frames the shift as moving beyond incremental IVR improvements toward rebuilding customer journeys from scratch, using AI to fit the company’s “uncarrier” culture: reduce trade-offs and put an “expert in your pocket,” including when the customer’s path is unpredictable.