New Google Model Ranked ‘No. 1 LLM’, But There’s a Problem

Google’s newly released Gemini experimental 1.5 (Gemini experimental 1114, dated Nov. 14) has landed at No. 1 on a human preference leaderboard—but the top ranking doesn’t settle the bigger question: whether today’s LLM progress is coming from smarter scaling or from controllable presentation factors that can mask deeper limits.

The leaderboard in question is built on blind human votes comparing two model answers over time. A key wrinkle: humans tend to prefer “flowery” language and longer responses, and those stylistic and length preferences are adjustable. When the comparison is constrained to remove style and length as variables, Gemini’s position drops to fourth place—below Claude 3.5 Sonnet for general tasks. The ranking shifts again under narrower conditions: for mathematical questions and “hard prompts,” OpenAI’s o1-preview takes the lead, while Gemini remains competitive but not dominant.

That mismatch matters because it highlights how benchmark-style wins can be sensitive to what people reward. It also exposes a communications gap. Unlike earlier Gemini launches that came with prominent benchmark scores and polished messaging, this release arrives with limited public testing access. The API reportedly had technical difficulties, preventing straightforward third-party evaluation. A workaround test using a small public “Simple Bench” sample suggests Gemini is correct on about three out of ten questions, while o1-preview and Claude perform closer to four or five out of ten. The speaker stresses this is anecdotal and that a full evaluation would require repeated runs and averaging.

Beyond the immediate ranking, the transcript ties Gemini’s uneven showing to broader industry signals: multiple reports in the last 48 hours describe diminishing returns across leading labs. Bloomberg reports OpenAI’s internally named GPT-5 (known as Orion) didn’t hit desired performance targets and may not represent the same leap GPT-4 did over earlier models. Google sources also describe disappointment with incremental gains. Meanwhile, Anthropic has reportedly reduced emphasis on Claude 3.5 Opus and released Claude 3.5 Sonnet instead.

The central takeaway is that “naive scaling” alone—more parameters, more data, more compute—may not be enough. Even Dario Amodei (via an interview referenced in the transcript) pushes back on the idea of fixed “scaling laws” as universal guarantees, calling them empirical patterns rather than laws of nature. The transcript links this to the 01-family approach: improvements increasingly rely on test-time strategies like more deliberate reasoning, “thinking time,” and related paradigms rather than only bigger training runs.

The discussion broadens into the race over what comes next: whether LLMs are plateauing on benchmarks (“eval saturation”) or whether the next gains will be less predictable and more engineering-heavy. OpenAI researchers and staff quoted here express confidence in a path to AGI—defined as replacing most economically valuable human work—while critics point to unresolved evaluation challenges like the ARC AGI benchmark. The Gemini No. 1 headline, in this telling, is less a victory lap than a snapshot of a field moving from predictable scaling to a more complicated mix of reasoning methods, evaluation limits, and product-facing behavior.