The Gemini Lie

Google’s Gemini Ultra is being marketed as a leap beyond GPT-4, but the most consequential takeaway is that the flashy “real-time video” demo is heavily curated—and the benchmark comparisons behind the hype may not be apples-to-apples. That matters because it shapes public expectations of how capable the model truly is, and those expectations can drive both investment and misuse.

The discussion starts with Gemini’s performance claims. Gemini Ultra is described as outperforming GPT-4 across multiple benchmark categories—reading comprehension, math, and spatial reasoning—while only falling short on a narrow interaction test: completing each other’s sentences. The bigger attention-grabber is a hands-on demonstration where the AI appears to watch a live video feed and play games such as “one ball three cups.” The key point is that this “video interaction” doesn’t reflect autonomous perception in the way viewers might assume. Instead, it relies on multimodal prompting: combining text instructions with still images extracted from the video stream.

A central critique follows: the demo’s apparent intelligence is partly a product of prompt engineering and editing. The analysis compares Gemini’s rock-paper-scissors example with GPT-4’s ability to handle similar multimodal prompts. When given an explicit hint that the task is a game, GPT-4 can identify the game correctly. In Gemini’s blog example involving hand signals, the prompt includes an encoded message—an additional layer that’s harder to decode. The transcript claims GPT-4 fails that version, while Gemini succeeds according to Google’s write-up, though the exact nature of the encoding remains a point of uncertainty.

The transcript then pivots to benchmark methodology, where it argues the headline “surpass human experts” claim is shaky. The benchmark cited is Massive Multitask Language Understanding (MMLU), a multiple-choice test spanning 57 subjects. The controversy centers on how models are evaluated: GPT-4 is compared using Chain of Thought up to 32 intermediate reasoning steps, while Gemini’s comparison is framed against a different setup (a “5-shot” condition). In the transcript’s accounting, GPT-4 performs higher under Chain of Thought (87.2%) but Gemini’s performance under 5-shot drops to 83.7%, undercutting the clean narrative of Gemini’s superiority.

Finally, the transcript urges skepticism toward benchmarks, especially when they come from non-neutral sources. It argues that the only reliable evaluation is direct interaction—“vibe with it”—because benchmark setups can reward specific prompting styles rather than general capability. The overall conclusion is cautious: Gemini Ultra may be powerful, but the public-facing demos and comparisons are not transparent enough to treat as definitive proof of readiness, particularly given the lack of a clear timeline for broader access and the risk of over-trusting polished demonstrations.