Why LLMs get dumb (Context Windows Explained)

LLMs start “getting dumb” in long chats because their context window—the maximum amount of text (measured in tokens) the model can actively pay attention to—fills up and attention degrades. As the conversation grows, the model must keep track of more tokens and run heavier attention computations, which increases GPU load and slows responses. When the model can’t reliably focus on the relevant parts, accuracy drops and hallucinations become more likely.

The transcript uses a local-model demo in LM Studio to make the mechanism concrete. A model such as “GEMMA 3 4 B” is configured with a context window (e.g., 2048 tokens). After feeding it a statement about a book (“How to Take Smart Notes”), the conversation is extended with unrelated prompts (“a story about cows,” then a sequel, then a prequel). Even though the earlier detail was provided, the model later fails to recall the book—showing how earlier context can effectively fall out of usable attention once the window is overwhelmed. Increasing the context window (to 4096) restores the ability to retrieve the earlier information, but it doesn’t remove the underlying tradeoff.

Bigger context windows demand more compute and memory. The transcript notes that while some models advertise extremely large contexts—GPT-4o at 128,000 tokens, Claude 3.7 at 200,000, Google Gemini at 1 million, and even Meta’s Llama 4 Scout at 10 million—running those limits locally is constrained by hardware. In the demo, pushing toward very large contexts (around 120,000–131,000 tokens) maxes out video RAM (VRAM), causing the system to slow dramatically and become harder to interact with. The key point: “full advertised context” doesn’t automatically mean a local machine can handle it smoothly.

Even with large windows, attention can still fail. A cited research result (“Lost in the Middle”) describes a U-shaped accuracy pattern: LLMs tend to perform best on information at the beginning and end of a long input, while middle content suffers. The transcript frames this as the model “falling asleep” during long sequences—less reliable attention to the middle portions of the conversation.

To manage the problem, the transcript recommends practical tactics: start a new chat when the user makes a significant topic shift, rather than letting one thread sprawl. It also highlights tooling to reduce token waste and improve readability when ingesting web pages—converting HTML into clean Markdown before pasting.

Finally, it outlines local performance optimizations that can make long contexts more feasible: enabling Flash Attention to avoid building the full attention comparison matrix in memory; using KV cache optimizations to compress stored attention data; and optionally using paged cache to spill cache from GPU VRAM into system RAM (faster than disk, but still slower than staying on-GPU). The transcript closes with a caution: larger contexts increase the “attack surface,” making it easier for malicious content to hide in long inputs and potentially bypass safeguards. In short, long context is powerful, but it’s expensive, attention is imperfect, and longer chats can be riskier.