Gemini 2.0 - How to use the Live Bidirectional API

Gemini 2.0’s Live Bidirectional API is built for real-time, two-way multimodal interaction—letting users talk back and forth with voice, stream audio/video input, and receive spoken or text responses instantly. The practical payoff is a conversational assistant that can respond while simultaneously processing what’s happening in front of the user, whether that’s a screen, an app workflow, or a live camera feed.

Getting started begins in AI Studio by selecting the “stream real time” option and choosing one of three interaction modes. The first mode is straightforward voice or text conversation with the model. Users can switch to the Gemini 2.0 flash model, pick an output format (audio or text), and select a voice. They can also add tools and—crucially—write custom system instructions to steer behavior. In the demo, a system prompt turns the assistant into a Westworld-style character (“Sarah Winters”), and the conversation follows that persona: it answers questions about recent guest interactions, describes maintenance requirements on a schedule, and agrees to power down when asked. The key point is that system instructions meaningfully shape the assistant’s tone and role, making the experience more than a generic chat.

The second mode demonstrates screen understanding. With the model connected to a chosen window, the assistant can identify what’s on-screen (the demo recognizes Figma), then guide a user through unfamiliar tasks using keyboard shortcuts and interaction tips. When the user asks how to zoom and how to drag/pan, the assistant provides concrete steps—like Command + / Control + for zooming and holding the space bar to temporarily switch to the hand tool for panning. It then helps improve a specific design card by asking clarifying questions about the “April 1st” element and suggesting the card’s content might need adjustment depending on whether it’s an event, blog post, or other use case. This mode highlights a hands-on workflow: the model can interpret UI context, answer “how do I do this?” questions, and iterate on design decisions.

The third mode brings live video into the loop. With camera access enabled, the assistant describes the scene in detail, identifies objects (including a Funko pop figurine of Bernard Lowe from Westworld), and answers follow-up questions about what it sees—such as interpreting the number of fingers held up. The interaction stays conversational while the model continues to observe, effectively combining real-time dialogue with ongoing visual description.

Beyond the live multimodal modes, the setup supports standard prompting and optional tool features like structured outputs, code execution, and function calling. The transcript also points to a unified SDK and a Gemini 2 cookbook with examples ranging from basic text prompting to the live bidirectional multimodal API. The overall message: developers can prototype quickly in AI Studio, then move the same capabilities into apps using the SDK, with Gemini 2.0 flash as the recommended starting model for experimentation.