Open AI SHIPS! o1 FULL & ChatGPT Pro - First Impressions

OpenAI’s o1 model is out of preview—and the release is positioned as a clear step up in reasoning benchmarks, especially for math and coding—while also finally delivering multimodal input support that was missing from the earlier preview. In a short launch livestream, OpenAI reported that o1’s performance jumps from 56.7 (o1 preview) to 83.3 on competition math, and from 62 (o1 preview) to 89 on competition code. For PhD-level science questions, accuracy lands around the same level as the preview (roughly 78–79), but the error-rate and “fewer-shot” behavior are presented as improvements, suggesting the model is getting to correct answers more reliably even when the evaluation is stricter.

Alongside the base o1 release, OpenAI introduced “o1 Pro mode” inside a new ChatGPT Pro subscription priced at $200 per month. Pro mode is described as using more compute for harder questions, with reported gains in competition math (about 85) and PhD-level science (about 79). OpenAI also highlighted a “worst of four” benchmark approach: the model attempts each question four times, and a problem counts as solved only if all four attempts are correct—an attempt to reduce cherry-picking and show consistency rather than one-off wins.

The demos leaned on live, human-verifiable tasks. One example involved an AST physics problem written on paper with a deliberately omitted parameter; o1 not only solved the math but also inferred the missing value, with researchers claiming the inferred parameter was accurate. Another demo tested biological reasoning by asking the model to guess a protein matching constraints like length (210–230 amino acid residues) and chromosome context; the model took about a minute and produced a correct guess.

The transcript then shifts from launch claims to hands-on impressions. Multimodal capability is treated as the biggest practical upgrade: the tester ran a “Where’s Waldo” image search and reported that o1 located Waldo precisely and described nearby visual structures (tents, umbrellas, shoreline crowding) in a way that regular GPT-4o was said to lack. The tester also pushed o1 into coding workflows: recreating a Spotify-like interface in HTML/CSS/JS from a screenshot, generating interactive hover/selection behavior, and producing an infinite-runner game with graphics and scoring. A further step converted the runner into a 3D scene using three.js, with the caveat that the game had minor issues after refresh and lacked a restart button.

Logic and real-world reasoning tests were mixed but generally positive. A classic “marble in an upside-down glass” puzzle was answered correctly with step-by-step reasoning, while a North Pole navigation puzzle—previously failing on o1 preview—was reported as solved, with the answer argued to be 2π kilometers. Other tests included identifying an engine bay as a mid-2000s Volkswagen TDI-family engine and a GeoGuessr-style location guess that landed close (Uruguay rather than the initially inferred Argentina).

The central takeaway is that o1’s release combines stronger benchmark performance with multimodal input and more capable coding-from-images, while ChatGPT Pro’s $200 price tag makes the compute-heavy “o1 Pro mode” a premium product that may only be worth it for heavy users. The transcript ends with an expectation of continued testing during OpenAI’s “12 days of shipping” run-up to Christmas.