Build Hour: Voice Agents

Voice agents are moving past “transcribe-and-reply” toward audio-native systems that can sound more like real representatives, handle ambiguity, and preserve emotional cues—while still delegating high-stakes work to smarter models. OpenAI’s Build Hour on Voice Agents ties that shift to a practical stack: real-time speech-to-pech models, the Agents SDK (now with TypeScript parity), and platform tooling that records audio and tool calls for debugging.

The session starts with a definition of an “agent” as a dynamic execution environment that bundles an AI model, instructions, and tool access—allowing the system to decide when to stop and when to hand off. From there, the case for voice AI centers on three advantages: flexibility (less deterministic behavior than earlier voice agents), accessibility (voice interaction is easier than text, especially on the go), and personalization (audio-native models can pick up tone and cadence that transcription often loses). OpenAI frames voice agents as “APIs to the real world,” aimed at solving last-mile integration problems.

Two building approaches are contrasted. A chained pipeline uses speech-to-text to produce a transcript, a text LLM (such as GPT-4.1) to generate a response, and text-to-speech to speak it back—easy to mix-and-match but prone to losing nuance. The newer approach uses speech-to-pech models that understand audio natively and generate audio output tokens directly. These models power advanced voice features in ChatGPT and OpenAI’s real-time API, and they’re positioned as emotionally intelligent precisely because they avoid the lossy transcription step.

Recent product updates are presented as reducing integration friction for real-time voice. A TypeScript version of the Agents SDK brings feature parity with the Python SDK and adds first-class real-time API support. Real-time model activity now appears in the platform’s Traces tab, automatically logging input/output audio and tool calls—critical because debugging audio-token behavior requires access to the full conversation audio. The real-time API also received a best-model snapshot (noted as the June 3rd snapshot), with reported gains in instruction and tool-calling accuracy plus a “speed” parameter to control how quickly the AI speaks.

The demo builds a home-remodel “workspace” assistant in layers. A workspace manager agent creates and updates UI tabs via tools. Switching to voice improves iteration speed, but early behavior is improved by prompting the agent to use filler phrases before tool calls so users understand what’s happening. The system then adds a dedicated interior-design “designer” agent, justified by best practice: split agents by well-defined roles to narrow their focus and improve output quality. The designer agent can also search the web for trends and then hand off workspace updates through encapsulated tools.

To handle open-ended design workflows, the demo upgrades the designer using a voice-agent meta prompt (authored by Noah) that generates identity, tone, and a state-machine-like conversation flow. It also introduces an estimator agent for budget and scheduling, isolating calculations from design so the designer doesn’t drift into estimation. The session highlights a key architecture pattern: real-time conversational agents can hand off to slower, higher-reasoning models (including O3 for more complex tasks) while preserving conversation context.

Stability and safety are treated as engineering problems. Output guardrails run on transcripts as they stream, interrupting the agent when moderation constraints are triggered and feeding back why it failed so the agent can correct itself. For evaluation, the platform’s Traces enable audio playback and tool-call inspection, and the roadmap points toward turning trace logs into eval inputs. The Q&A adds implementation details: real-time configuration choices (model snapshot, Whisper-1 transcription, audio codec, VAD options like semantic VAD), temperature guidance, and mobile architecture using WebRTC with ephemeral tokens.

Overall, the central takeaway is that production-ready voice agents come from combining audio-native models with tool-driven agent design, strong prompting/state management, and platform-grade observability—so teams can iterate quickly without sacrificing reliability or safety.