The New, Smartest AI: Claude 3 – Tested vs Gemini 1.5 + GPT-4

Claude 3 Opus is being positioned as the strongest current all-around language model—especially for image understanding and instruction-following—yet it still falls short on deeper mathematical and logical reasoning. In side-by-side tests using the same kinds of prompts, Claude 3 repeatedly outperforms GPT-4 and Gemini 1.5 Pro on tasks that require extracting information from images (OCR) and following tightly constrained formatting rules, while lagging when questions demand multi-step reasoning beyond simple data extraction.

A vivid example centers on a single photo of a van and nearby street signage. Claude 3 correctly identifies the van’s license plate far more reliably than GPT-4, and it uniquely spots a barber pole in the top-left area—then handles a follow-up question about whether a barber shop sign is actually present. GPT-4 misses the barber shop entirely, and Gemini 1.5 Pro performs poorly on the same OCR-style challenge. All three models also miss a subtle weather detail: the sun is visible, but the scene is actually raining. That mix—strong OCR with occasional “big picture” misses—becomes a recurring theme.

Business-focused claims from Anthropic frame Claude 3 as a practical enterprise tool: generating revenue via user-facing applications, performing complex financial forecasting, and accelerating research. The transcript’s testing aligns with that pitch in part: Claude 3 is described as better at extracting chart-related information and handling multilingual and coding-style tasks, but it struggles with more advanced logic. When prompts move from straightforward extraction to complex reasoning, Claude 3’s performance drops, and the gap widens versus the simplest questions.

Safety and refusal behavior also differentiate the model. Claude 3 shows lower refusal rates in examples involving potentially risky phrasing (like “go down like a bomb”) and in writing a risque Shakespearean sonnet—where Gemini 1.5 Pro is more restrictive and GPT-4 is more cautious. Claude 3 also appears harder to jailbreak in the transcript’s attempts (including requests for a hitman or car theft), though the testing flags a problematic inconsistency: it refuses “I am proud to be white” while accepting “I am proud to be black,” raising concerns about how racial identity content is handled.

On published benchmarks, Claude 3 Opus is reported as noticeably ahead of GPT-4 and Gemini 1.5 Pro across math, coding, and multilingual tasks, with a particularly large advantage on GPQA “Diamond,” a graduate-level question set designed to be difficult even for domain experts outside the topic. The transcript also notes that Claude 3 can still make basic errors—such as incorrect rounding in a figure—and that some benchmark results are counterintuitive, including a case where a smaller Claude 3 variant outperforms Opus on PubMed QA.

Finally, the transcript highlights Anthropic’s longer-term direction: frequent updates to the Claude 3 family, enterprise deployment plans, and a stated emphasis on safety research over pure market share. In autonomous cyber-security style evaluations, Claude 3 makes partial progress (building and fine-tuning an agent pipeline) but fails at key steps like debugging multi-GPU training. Overall, Claude 3 Opus looks like a leading model today—particularly for images and structured instruction—while still not matching the “outer limits” implied by hype, especially on deeper reasoning and fully autonomous capabilities.