Deep Research by OpenAI - The Ups and Downs vs DeepSeek R1 Search + Gemini Deep Research

OpenAI’s newly released “Deep research” agent—built on its most powerful o3 model—delivers a noticeable leap in web-based, needle-in-a-haystack research, especially on benchmarks that reward stitching together obscure facts. In early hands-on testing, it surged from low baseline performance on a key “useful assistant” benchmark to roughly 67–72%, while still trailing human experts who can reach about 92% when they put in effort. The practical takeaway: for hard, information-dense questions that require searching, filtering, and synthesizing, Deep research often finds the right trail faster than competing systems.

That strength showed up most clearly on “Humanity’s last exam,” a benchmark centered on obscure knowledge and the ability to assemble scattered details into correct answers. With web access enabled, Deep research’s performance jumped sharply, reinforcing the idea that its core advantage is not just answering, but hunting down niche sources and assembling them into a coherent response. The same pattern appeared in a real-world-style test using a newsletter archive: Deep research located posts where a dice rating changed to 5 or higher and then extracted and summarized the relevant sections—exactly the kind of targeted retrieval that would otherwise require manual scanning.

The caveat is that the agent’s reliability is uneven, and its interaction style can be frustrating. On a “common sense / spatial reasoning” style mini-benchmark, it repeatedly asked clarifying questions instead of directly answering, and the results showed little improvement—suggesting it can struggle to “grasp the real world” even when it can browse. More importantly, multiple shopping and fact-check tests raised concerns about hallucinations and citation fidelity. In one case, Deep research claimed it visited a specific price-history site, yet the returned price history did not match what the site actually showed. In another, it produced plausible-sounding details that were wrong, and DeepSeek R1 with search hallucinated even more aggressively (including inventing battery-life claims).

When Deep research was compared against DeepSeek R1 with search and Google’s Gemini Deep research, the overall pattern favored OpenAI for task completion—often “better pretty much every time”—but not for factual cleanliness. DeepSeek R1 with search was less annoying in conversation (fewer clarifying-question loops) but still produced errors, including fabricated specifics. Gemini Deep research, in these tests, failed to retrieve relevant newsletter data at all.

The transcript’s broader warning is that small, repeated hallucinations remain a thin line of defense for white-collar workflows. Even when models produce deep, well-structured research—like analyzing dozens of references in a research paper—incorrect details can slip in, and sometimes the summary language treats hypothetical or uncertain information as if it were factual. The result is a system that can dramatically accelerate research and synthesis, but still demands verification when decisions depend on accuracy.