Attention in transformers, step-by-step | Deep Learning Chapter 6

Attention in transformers is the mechanism that lets each token’s embedding absorb information from other tokens—turning context-free word vectors into context-rich representations that can accurately predict the next word. The core move is to compute, for every token position, a set of relevance scores against every other position, normalize those scores with a softmax, and then use them as weights for a weighted sum of “value” vectors. That weighted sum becomes the update to the token’s embedding, so meanings shift depending on surrounding words rather than staying stuck to a lookup-table identity.

The process starts with embeddings: each token maps to a high-dimensional vector that encodes both the token’s identity and its position. While the initial embedding for a word like “mole” is the same across different sentences, attention later determines which other tokens should influence it. In the examples, “mole” changes meaning across “American shrew mole,” “one mole of carbon dioxide,” and “take a biopsy of the mole,” because the attention mechanism routes information from the surrounding context into the “mole” embedding. A similar idea appears with “tower”: when preceded by “Eiffel,” the model can steer the representation toward the Eiffel Tower; when preceded by “miniature,” it can shift away from large, tall associations.

To compute these context-dependent updates, a single attention head builds three learned linear projections of the embeddings: queries (Q), keys (K), and values (V). Each token produces a query vector and a key vector in a smaller “query-key space” (for GPT-3, 128-dimensional), using learned matrices wq and wk. Relevance is measured by dot products between every query and every key, producing a grid of scores. Those scores are scaled by the square root of the key-query dimension for numerical stability, then normalized with a softmax (applied column-by-column) to form an attention pattern—effectively a set of probability-like weights indicating which source tokens matter for updating each target token.

Training adds an important constraint: masking prevents later tokens from influencing earlier ones. During next-token prediction, allowing future context to leak backward would make the task artificially easy, so entries corresponding to “future-to-past” influence are set to negative infinity before softmax, forcing their weights to become zero while keeping normalization intact.

Once the attention pattern is computed, the head updates embeddings using the value vectors. A learned value matrix maps each token embedding into a high-dimensional value vector; each token’s embedding update is a weighted sum of these value vectors, where weights come from the attention pattern. Conceptually, “fluffy” and “blue” contribute strongly to the update of “creature” in the phrase “a fluffy blue creature roamed the verdant forest,” shifting the noun embedding toward a more specific meaning.

Multi-headed attention runs many such heads in parallel, each with its own Q/K/V projections, so the model can learn multiple distinct context-update strategies at once. GPT-3 uses 96 attention heads per block, and the parameter count per block is enormous—around 600 million for multi-headed attention alone. Across 96 layers, attention parameters total just under 58 billion, about a third of the roughly 175 billion parameters in GPT-3, with the rest largely residing in the feed-forward (MLP) components between attention blocks.

Finally, attention’s success isn’t only about what it computes; it’s also about how efficiently it scales. The operations are highly parallelizable on GPUs, aligning with the broader deep-learning lesson that scale and throughput often translate into better performance. The result is a mechanism that turns context into actionable routing: tokens selectively pull meaning from other tokens, repeatedly, across layers, until high-level abstractions can emerge.