How might LLMs store facts | Deep Learning Chapter 7

Large language models don’t just “know” facts in a vague sense—those facts can be traced to specific internal computations, especially inside the multilayer perceptron (MLP) parts of transformer networks. A concrete toy example uses the prompt “Michael Jordan plays the sport of blank” to show how an MLP block could, in principle, turn a representation of “Michael Jordan” into an added direction corresponding to “basketball.” Even though a full mechanistic map of real-world fact storage remains unsolved, the evidence discussed points to a broad pattern: the most useful capacity for storing and transforming factual associations appears concentrated in the MLP sub-networks rather than the attention mechanism alone.

The explanation starts with how transformer vectors encode meaning in a high-dimensional space. Different directions in that space can represent different features. Under simplifying assumptions, one direction corresponds to the first name “Michael,” another nearly perpendicular direction corresponds to the last name “Jordan,” and a third direction corresponds to “basketball.” If an internal vector aligns with both “Michael” and “Jordan,” then the MLP’s computations can be arranged so that the “basketball” direction gets injected into the output.

Inside an MLP block, each token’s vector is processed independently in parallel. The core computation is two matrix multiplications with biases and a nonlinearity in between. First comes an “up projection” matrix: it linearly mixes the input vector into a larger hidden space. Then a ReLU activation clips negative values to zero, turning a fuzzy linear score into a more gate-like signal. With a carefully chosen bias, a hidden neuron can behave like an AND gate: it activates only when the input vector simultaneously indicates “Michael” and “Jordan,” while other partial matches (like “Michael” with a different last name) fail to activate it. Finally, a “down projection” matrix maps the activated hidden units back into the original embedding dimension. In the toy setup, one particular column in this down-projection matrix is aligned with the “basketball” direction, so activating the “Michael Jordan” neuron causes the output vector to gain the “basketball” component. The block then adds this result back to the residual stream.

The transcript also grounds the toy story in scale. For GPT-3, the up and down projection matrices each contribute roughly 1.2 billion parameters together, and 96 separate MLPs bring the total MLP parameter count to about 116 billion—around two-thirds of the model’s 175 billion parameters. Normalization-related parameters exist too, but they’re described as a small fraction.

A key twist comes from interpretability research: individual neurons may not correspond to single clean concepts like “Michael Jordan.” Instead, a “superposition” hypothesis suggests that many features share the same neurons by using nearly orthogonal directions in high-dimensional space. In high dimensions, it becomes possible to pack exponentially more nearly perpendicular feature directions than the raw dimensionality would suggest, which helps explain why models can scale well while remaining hard to interpret. In that view, features are distributed across combinations of neurons rather than appearing as a single, obvious activation; tools like sparse autoencoders are mentioned as a way to tease out these hidden features. The next major step promised is the training process—backpropagation, language-model loss functions, reinforcement learning with human feedback, and scaling laws.