Gemini 2.0 - Video Analyzer with Code

Gemini’s “Video Analyzer” turns uploaded videos into structured, time-coded outputs—captions, spoken transcripts, visual scene descriptions, key moments, tables, and even counts—by combining prompting with function-style JSON results. The practical payoff is metadata you can search, index, and feed into retrieval systems, not just a single transcript.

In AI Studio’s starter apps, the workflow starts with uploading a video through the Files API. Once the upload completes, the system tokenizes the content for downstream prompting. From there, users can generate A/V captions that return scene-by-scene objects tied to timecodes. Each scene’s output can include both what appears on screen and any spoken text (with spoken segments placed in quotation marks). The interface lets viewers jump to specific timestamps and see descriptions that track visual changes—such as identifying a 3D animation of a window with plants on a dark blue background—while also capturing the audio content.

Beyond captions, the same underlying function-based approach can be steered to produce different formats. Users can request paragraph-style narrative responses for each scene, ask for “key moments” summarized with timestamps, or generate a table that organizes detected objects and visual elements (including emojis/objects) by time. Counting is another supported mode: prompts can instruct the model to count people visible per scene, returning numeric values aligned to timecodes. The demo also shows prompt-driven customization—switching from counting people to counting phones, and then to counting trees—highlighting that the counting logic can be repurposed for different visual targets.

A key detail is that these outputs aren’t purely free-form text. The system relies on function calls such as setTimeCodes, setTimeCodesWithObjects, and setTimeCodesWithNumericValues, each designed to return structured data (timecodes plus text, objects, or numbers). The transcript emphasizes that the function call behavior depends on including the function name in the prompt; otherwise, the model may return text without triggering the structured function output.

To reproduce the demo in code, the walkthrough uses Colab with a “new unified SDK.” The process mirrors the UI: upload a longer video (noting the UI seemed sensitive to length), poll the upload state until processing finishes, then call Gemini with a system prompt that instructs it to call the relevant function only once with appropriate timecodes and text. The Python implementation defines “tools” (function declarations) and helper functions to print results. Initially, removing the function mention from the user prompt yields mostly transcript-like text; adding the function name back enables proper structured outputs.

Finally, the code extends the approach by adding a custom function, setTimeCodesWithDescriptions, to separate spoken text from visual descriptions. The result is a richer indexable representation: time-coded transcripts plus distinct visual metadata (e.g., describing a cartoon llama and on-screen UI elements). The practical conclusion is that these structured captions and visual descriptions can be converted into text chunks for RAG systems, enabling question answering over both what was said and what appeared on screen at specific moments.