Biology of LLMs - Part 1

Mechanistic interpretability is moving from “what concepts are stored where” toward “how those concepts get used to produce the next token.” The session lays out the background for Anthropic’s “Biology of LLMs” work—especially its focus on Claude 3.5 Haiku and a smaller, unnamed transformer—and explains why understanding modern transformer behavior is harder than earlier circuit work on CNNs.

The talk traces interpretability’s evolution. Early “distill circuits” research treated neural nets like electrical circuits: break models into smaller units, study individual components, and map how signals flow. That approach worked well for CNNs, where lower layers reliably detect simple features (edges, lines) and higher layers compose them into more complex ones (textures, shapes, even faces). Transformers, however, introduce far more nonlinearities and a tangled interaction between attention and MLP blocks. The session highlights a key obstacle: transformers stack dozens of layers, and each layer’s internal representations shift as information moves through attention heads and feed-forward networks.

A central theme in the preceding wave of transformer-circuit research is “superposition.” Instead of dedicating one direction in representation space to one concept, LLMs can pack many concepts into a limited-dimensional residual stream. When concepts are not orthogonal and appear together, they interfere—sometimes canceling out in dot-product probes. Earlier papers used compressed-sensing-style arguments and experiments to show that this interference can be tolerated when concepts are sparse in the training distribution: if two concepts rarely co-occur, the model can reuse directions without catastrophic collisions. The session uses a toy example in a 2D space to illustrate how increasing sparsity changes the geometry from clean orthogonal “compass directions” to a “pentagon” arrangement where multiple concepts coexist with controlled interference.

But knowing that concepts exist as directions (and that they can be recovered) still doesn’t answer the practical question: how do those directions drive generation? The session frames Anthropic’s next step as studying the MLPs—because attention mainly moves information across token positions, while MLPs operate on the current token’s residual stream. Attention can “pull” earlier facts into the present position, enabling the MLP to act on them; without that pull, MLPs can’t directly access earlier tokens. The residual stream itself is treated as the shared workspace: if attention and MLP outputs were zeroed, the model would fall back to a default next-token prediction, implying that meaningful information already lives in the stream and layers add, subtract, or nudge it.

The background also emphasizes how attention is analyzed in circuit frameworks: Q and K determine attention patterns (a probability distribution over prior token positions), while the values and output projections affect which subspaces get written to the residual stream. That separation matters because later interpretability work can focus on attention patterns without needing the raw Q/K magnitudes.

Finally, the session explains why Anthropic’s “biology” approach is different from earlier direction-finding work using sparse autoencoders. Prior work could recover millions of concept directions, but diagnosing how they get used across layers remained out of reach. The promised technique replaces the original model with progressively simpler surrogate models: first a sparsified replacement model, then a “local replacement model” tailored to a specific prompt. By hardcoding attention patterns and removing nonlinearities, the relationship between layers becomes linear enough to compute influence on the next token directly. The approach then validates hypotheses back on the original model, aiming to connect recovered concepts to concrete causal effects inside specific MLP layers—moving interpretability from geometry to mechanism.