The New Claude 3.5 Sonnet: Better, Yes, But Not Just in the Way You Might Think

Claude 3.5 Sonnet’s biggest upgrade isn’t a flashy new “computer control” trick—it’s a noticeable jump in reasoning, coding, and multimodal understanding, backed by both published benchmarks and fresh third-party-style testing. The model also carries world knowledge up to April 2024, but the practical takeaway is that it performs better on the kinds of tasks people actually use LLMs for: software engineering benchmarks, science and math questions, and visual question answering over charts and tables.

A key thread running through the results is that Claude 3.5 Sonnet improves across multiple evaluation suites, including OS World (a large set of office and daily-use tasks) and SWE-bench Verified (software engineering). In OS World, the human comparison baseline is drawn from computer science undergraduates who hadn’t seen the exact test samples, which makes the reported “human-level” gap feel less like a trivial yardstick and more like a meaningful bar. On SWE-bench Verified, Claude 3.5 Sonnet lands at 49%, beating the earlier Claude 3.5 Sonnet and also outperforming OpenAI’s o1 preview in the cited comparison. The transcript also notes that apples-to-apples comparisons are tricky because prompting and scaffolding differ, but the direction of improvement is consistent.

The reasoning gains show up beyond standard benchmarks. In the creator’s own “simple bench” (a human-friendly test covering spatial reasoning and social intelligence), Claude 3.5 Sonnet scores higher than the previous version. Even with prompting and scaffolding, models still don’t reach the expanded human baseline, but the gap narrows compared with earlier Claude releases—an important point because it suggests progress is real, not just prompt engineering.

Where expectations should be tempered is reliability in agentic settings. Claude 3.5 Sonnet can use a computer via an API, yet broad adoption is unlikely due to unreliability and a long list of missing capabilities (for example, sending emails, making purchases, or reliably editing/manipulating images). The transcript leans on a separate “retail and airline” agent benchmark to make the reliability problem concrete: performance drops sharply when tasks must succeed every time across multiple attempts (described via “pass-to-the-power-of-K”). For airline booking and modification, one-try success rates are far from enough when the system must be correct repeatedly.

There’s also a nuance in safety behavior: Claude 3.5 Sonnet is said to refuse toxic requests slightly less often and refuse some innocent requests slightly more often than the prior model—small shifts, but worth noting when judging overall capability.

Finally, the transcript widens the lens beyond Claude. It highlights ongoing competition in AI entertainment and interactive modalities—Runway’s Act One for character-driven video generation, HeyGen-style interactive avatars with “Zoom-like” roleplay, and NotebookLM’s ability to customize podcast-style conversations from uploaded sources using Gemini 1.5 Pro. Taken together, the central message is clear: Claude 3.5 Sonnet is better in the ways that matter for everyday work, but the last mile to major economic impact still hinges on dependable agent behavior, not just higher benchmark scores.