Transformers, the tech behind LLMs | Deep Learning Chapter 5

Transformer-based models—behind systems like ChatGPT—turn text into a stream of vectors, mix information across tokens with attention, and then convert the final hidden representation into a probability distribution over the next possible token. The practical breakthrough is that this “predict-the-next-token” loop can be run repeatedly: feed the model a prompt, sample from its predicted distribution for the next token, append that token, and repeat. What looks like simple next-word prediction becomes coherent multi-sentence generation when the model is large enough and trained well.

At a high level, the process starts by splitting input into tokens. For text, tokens often correspond to words, word fragments, or common character sequences; for multimodal variants, tokens can represent small image patches or audio fragments. Each token is mapped to a vector via an embedding matrix: a vocabulary-sized table where each row/column corresponds to a token and the vector dimension encodes learned semantic relationships. In GPT-3, the embedding dimension is 12,288 and the vocabulary is 50,257 tokens, implying roughly 617 million parameters just for embeddings. These vectors are not merely “word meanings.” They also carry positional and contextual information once the model processes them through layers.

The core mechanism that lets tokens influence one another is attention. The model repeatedly alternates between attention blocks and feed-forward (multi-layer perceptron) blocks. Attention is where vectors “talk” to each other: it updates token representations based on which other tokens in the current context are relevant. A key example is how the word “model” changes meaning depending on whether it appears in “machine learning model” versus “fashion model.” The feed-forward layers then transform each token representation in parallel, applying non-linear computation without direct token-to-token communication.

After enough layers, the model produces a final hidden vector for the context window. That vector is projected through an “unembedding” matrix (W_U) into a set of scores—one per vocabulary token—then normalized into probabilities using softmax. Softmax converts arbitrary real-valued scores into a valid probability distribution by exponentiating each score and dividing by the sum of all exponentiated values. Higher scores become more likely, but the distribution remains smooth enough to support sampling rather than always choosing the single most likely token.

Generation is then a loop. Given a prompt, the model predicts a distribution for the next token, samples from it, appends the sampled token to the prompt, and predicts again. This is why a model can appear to “write” stories: it’s repeatedly sampling from its own next-token predictions. Temperature controls how sharp the distribution is: lower temperature makes the model behave closer to greedy decoding (near-deterministic), while higher temperature increases randomness by flattening probabilities. The transcript notes that some APIs cap temperature (e.g., not above 2) to prevent outputs from degrading into nonsense.

Finally, the transcript grounds these ideas in the training paradigm of deep learning: large models learn weights by backpropagation, using vast datasets rather than hand-coded rules. GPT-3 is highlighted as an example of scale—175 billion parameters—organized into many matrix multiplications. The embedding and unembedding matrices, attention, and feed-forward transformations all reduce to weighted sums and non-linearities, making the transformer architecture a scalable, trainable engine for next-token prediction and, by extension, the fluent text generation seen across modern AI tools.