Gemini 1.5 Pro for Video Analysis

Gemini 1.5 Pro can extract highly specific information from a long video—down to approximate timestamps for when key topics appear—making video-based Q&A and slide-by-slide summaries practical even with a near-1 million token context window. The most consequential takeaway is that the model’s strength comes from pairing visual slide content with text (especially Whisper-style transcripts), enabling targeted “needle-in-the-haystack” queries like identifying whether Noam Shazeer is cited, and pulling details such as Gemini training data and evaluation themes.

The workflow starts with preparing a ~50-minute cut of Jeff Dean’s recent talk on machine learning trends, then uploading it to Google AI Studio. Tokenization alone can take time, and inference latency grows as the context approaches the 1 million token limit: answers often take roughly 1.5–3 minutes, with the slowdowns becoming noticeable near the maximum context. Audio is stripped because Gemini 1.5 Pro (in this setup) doesn’t support audio input, so early questions rely purely on the slides.

Even without transcript or audio, Gemini can answer basic identity and logistics questions. It correctly identifies Jeff Dean as the talk’s author and returns a general summary focused on large language models, multimodal models, data quality, and evaluation methods. More importantly for retrieval-style use cases, it can locate when Gemini first appears: Gemini is mentioned at about the 25-minute mark (around 25:25), and the system sometimes narrows this to within seconds to half a minute.

The slide-only approach also supports precise, high-signal queries. When asked whether Dean cites work from Noam Shazeer, Gemini returns “yes,” pointing to Shazeer’s Transformer-related contributions—consistent with Shazeer’s name appearing on the relevant “Attention is All You Need” slide. That said, some returned claims may blend slide evidence with internal knowledge; for example, Gemini also mentions Shazeer’s work on PaLM, even though the transcript suggests the name may not appear directly on a slide.

Adding a Whisper transcription changes the quality and depth of extraction. With the transcript included, Gemini can generate a breakdown of each slide with the time it appears and what was discussed, and it can pull details from Dean’s spoken wording that aren’t visible in the visuals. The transcript-plus-slides setup increases total token load (to roughly 817k–801k tokens for video plus ~15.5k for transcript) and further increases latency, but it yields more complete answers—such as Dean’s description of Gemini training data: large collections of web documents, books, and code, plus quality filtering using heuristics and model-based classifiers.

Finally, the same extracted material can be repurposed into other outputs, including a generated blog post summarizing the talk’s key themes. The practical message is clear: long-context video understanding isn’t a replacement for retrieval-augmented generation (RAG), because processing cost and latency still favor RAG for most production tasks—but for deep video interrogation and slide-level information mining, Gemini 1.5 Pro is demonstrably effective.