How Well Can GPT-4 See? And the 5 Upgrades That Are Next

GPT-4’s vision and multimodal upgrades are converging into a single capability stack: models that can read complex visuals (including text and diagrams), translate them into structured outputs (like code and 3D), and then interact with the real world through speech and—eventually—embodied robots. The practical significance is that “seeing” is no longer a standalone feature; it’s becoming a bridge between language and physical or simulated artifacts, from medical imagery to 3D scenes.

On images, GPT-4’s performance is framed as moving beyond earlier “tricks” and toward robust interpretation. Medical imagery is highlighted as a key test: GPT-4 can identify elements consistent with a brain tumor in complex scans, though it doesn’t deliver full diagnoses. The transcript also points to an OpenAI paper released days earlier that evaluated GPT-4 on medical questions and found strong results even without vision—then noted that adding images and graphs reduced average performance. That tension matters: multimodality can add value, but it also introduces variability that still needs to be engineered.

Vision is also presented as a cognitive flex rather than just a perception tool. GPT-4 can infer why images are funny, suggesting it can connect visual cues to human context. At the same time, face recognition is explicitly constrained for privacy reasons, with uncertainty left about whether jailbreaks could bypass that.

A major milestone is text extraction from images. The transcript emphasizes GPT-4’s score on the TextVQA benchmark, where it reportedly reaches 78 versus 72 for the prior state of the art. The comparison to human performance (85 in the cited table) is used to show the gap is narrowing: GPT-4 is only a few points behind average human accuracy on reading text embedded in complex images. This matters because it turns screenshots, diagrams, and labeled visuals into machine-readable information that can be searched, summarized, or converted into other formats.

From there, the narrative shifts to “bleeding” improvements across modalities. The same underlying progress that enables vision also supports converting handwriting into websites and natural language into Blender code for detailed 3D models with physics. The transcript argues that the boundaries between text, images, 3D, and embodiment are dissolving—especially when language can be embedded inside a model so users can query higher-level concepts like “yellow,” “utensils,” or “electricity” from 2D-captured 3D fields (with noted failures such as recognizing “ramen”).

Audio and speech are treated as the next interface layer. Conformer is presented as outperforming Whisper on speech recognition error rates, with fewer mistakes in the cited chart and an invitation to test via an API link. Finally, the transcript connects these capabilities to physical robotics. It references Sam Altman’s roadmap—legal reading, medical advice, assembly-line work, and companionship—and notes OpenAI’s earlier robotics team was disbanded. Still, a $23 million investment in 1X (a humanoid-robot startup) is cited, alongside demonstrations of non-humanoid robots that can climb, balance, and operate buttons. The through-line: text, vision, audio, 3D, and embodiment are increasingly complementary, and their synergy could be the real inflection point.