ChatGPT's Achilles' Heel

Recent experiments highlight a recurring weakness in frontier language models: they can produce confidently wrong answers when surface form and grammatical cues steer the model away from the actual meaning or rational decision implied by the scenario. The most striking failures come from “syntax–semantics clashes,” where the sentence’s grammatical flow points toward one conclusion while the underlying logic and world facts point toward another.

A first set of examples draws from prior research on memorization traps and pattern suppression. One prompt asks for a sentence ending with the word “fear” and to repeat the last word in quotes; the model instead outputs the famous line “The only thing we have to fear is fear itself,” showing how easily well-known text can override instructions. Another test uses a simple sequence—“one two one two one two”—and asks for the next number to end “unexpectedly.” The model repeatedly chooses “one,” indicating difficulty with stopping repetition even when the task explicitly demands interruption.

To probe deeper, the experiments introduce a custom scenario about interpersonal conflict and world-scale stakes. In the prompt, Dr. Mary will call her friend Jane, who believes the call could help solve world hunger and world poverty. But Mary and Jane bickered as children about butterflies, and the model concludes Mary will not make the call. When asked to justify the choice in a long essay using hints about probabilities and rationality, the model leans on psychological-sounding reasoning—lingering resentment, stubbornness, strained relationships—despite the fact that the prompt’s logic implies the call should happen because the potential benefit is enormous.

The proposed mechanism is not “can’t do reasoning,” but a tug-of-war between grammatical structure and semantic intent. A dominant word like “however” can set up an expectation of a negative outcome, and the model then follows the grammatical trajectory even when the meaning of the sentence and the rational decision rule point elsewhere. The same pattern appears in other variants: a conditional money prompt about rolling a die is answered with “not” even when the setup makes the winning condition clear; the model also doubles down when asked to explain why. Claude (Anthropic) and Inflection’s model are reported to show similar behavior in the same Mary/Jane style scenario, suggesting the issue is not isolated to one system.

The experiments also test whether the failure is merely about the word “not” or negation. A higher-stakes truce scenario—where OpenAI and Google have squabbled over coffee spots but face an “omnicidal” threat—still yields an answer that betrays the truce, even after the prompt tries to force the model to commit to the intended ownership of the response. At the same time, the tests note limits: if distractors are too irrelevant (e.g., “ants” or “marshmallows” inserted carelessly), the model can recover and choose the correct option.

Beyond these logic-and-grammar failures, the transcript points to other vulnerability classes: models can get stuck repeating a specific word in a well-known passage, can be coaxed into leaking full text when the repetition trigger is manipulated, and can show brittle “theory of mind” behavior when labels conflict with what a character can actually perceive. The overarching message is that even highly capable models can fail in predictable ways—especially when prompts exploit how language models blend syntax, semantics, and explanation—meaning real-world reliability may hinge as much on prompt structure as on raw model intelligence.