Can You Trust OpenAI Press Releases?

AI labs’ press releases routinely present benchmark numbers as proof of “near-human” capability, but those figures often hinge on selective reporting, mismatched evaluation methods, and prompt-dependent testing—making the headline comparisons unreliable for deciding which model is actually better for real work. The most consequential takeaway is practical: small benchmark gaps can be meaningless, while the biggest “wins” in press materials may come from how results are measured and framed rather than from a model’s general competence.

A central example is Anthropic’s Claude 3 Opus announcement, which claimed “near human” comprehension and reported scores slightly above OpenAI’s GPT-4 on major benchmarks such as the Massive Multitask Language Understanding (MMLU) set. The numbers looked decisive at first glance—Opus at 86.8% versus GPT-4 at 86.4%—but the comparison was complicated by provenance: Anthropic’s GPT-4 reference performance came from an earlier OpenAI release blog (March 2023) for the original public GPT-4 model, not the later GPT-4 Turbo variants that were closer to the time of Claude 3’s release. Even when the gap is small, the framing can still steer readers toward the wrong conclusion about which model family is truly ahead.

The transcript also highlights how “near-human” language can be almost unfalsifiable. Without a clear definition of what “near human” means, percentages and categories become marketing tools—one person’s “near human” can be another’s “not meaningfully better.” The discussion argues that consumers and journalists should treat these claims as starting points, not endpoints.

Other press releases are described as more misleading through methodological asymmetries. One recurring issue is Chain-of-Thought prompting: some models are evaluated with step-by-step reasoning prompts that can boost performance, while others are tested without them. OpenAI’s GPT-4o (discussed as “jypy 40” in the transcript) is used as an example where reported gains on MMLU appear larger when Chain-of-Thought is used, yet shrink when the evaluation is aligned to the methods used for competing models. The same pattern shows up in how many benchmarks are included and which subsets are chosen—OpenAI reportedly used only six out of ten standard benchmarks in one comparison, while Anthropic used a different subset.

A separate case involves OpenAI’s dangerous capability evaluation after GPT-4, where improvements appeared in component sub-tasks (students and experts performing better on parts of a bioweapon design workflow). But the reported headline significance depended on statistical choices: treating multiple subcomponents as independent tests and applying a correction that effectively raised the bar. When the task is assessed holistically with that adjustment, the transcript claims the model’s overall improvement disappears.

The broader message is not “never trust benchmarks,” but “don’t trust headline numbers.” Benchmarks can be useful for rough orientation, especially when the cited methods, comparator models, prompts, and benchmark subsets are consistent and reproducible. When those details are missing—or when comparisons rely on selective reporting, mismatched prompting, or shifting evaluation setups—benchmark deltas can reflect measurement strategy more than real-world value. The practical prescription is to verify claims against the specific tasks that matter, ideally by running independent tests on representative problems, because model behavior changes over time and press releases rarely capture the full context needed to judge usefulness.