Positional Encoding in Transformers | Deep Learning | CampusX

Transformers need positional information because self-attention treats tokens as a set—great for parallel context building, but blind to word order. The core problem is that “river bank” and “bank river” can end up with the same attention inputs, even though they mean different things. Positional encoding fixes this by injecting a representation of each token’s position into the model so attention can distinguish not just *what* words are present, but *where* they occur.

The transcript starts by unpacking why self-attention is powerful: it generates context-aware embeddings dynamically, and it can compute interactions for all tokens in parallel. That parallelism is a major speed advantage over inherently sequential architectures like RNNs, especially for long documents. But the same parallel design creates the order problem: since all tokens are fed together, self-attention has no built-in mechanism to know whether “Nithish” came before “Kild” or vice versa. Without an order signal, the model would struggle with real NLP tasks where syntax and semantics depend heavily on position.

A naive fix—appending the token index as an extra number—fails for multiple reasons. First, the index grows unbounded with sequence length, which can destabilize training because neural networks often become numerically unstable with very large values (leading to issues like exploding/vanishing gradients). Second, normalization doesn’t solve the deeper issue: the same “position” would map to different values across different sequence lengths, making the encoding inconsistent across training samples. Third, using discrete indices harms learning because neural networks generally prefer smooth, continuous patterns. Finally, simple indexing captures absolute position but not relative distance well, since the model can’t infer how far apart two tokens are when the encoding is effectively a discrete lookup.

The transcript then motivates the standard Transformer solution: use sinusoidal positional encodings. By mapping each position through sine and cosine functions, the encoding stays bounded (values remain between -1 and 1), changes smoothly, and can represent relative offsets. A single sine wave still risks collisions because sine is periodic—different positions can produce the same value. The fix is to use multiple frequencies: pair sine and cosine at different “wavelengths,” turning each token’s position into a vector rather than a single scalar. Higher-frequency components vary quickly (helping distinguish nearby positions), while lower-frequency components vary slowly (supporting longer-range distinctions). This multi-frequency design reduces the chance that two different positions share the same encoding.

Finally, the transcript explains how positional encodings integrate with token embeddings in practice. For each token, a positional encoding vector is computed with the same dimensionality as the token embedding (denoted as d_model). Instead of concatenating (which would increase dimensionality and effectively double parameter/training cost), the approach adds the positional vector to the token embedding element-wise. The result is a combined representation that carries both word meaning and position information into the self-attention block.

The closing sections emphasize that the sinusoidal scheme isn’t just a heuristic: with the right mathematical structure, it supports linear relationships that allow the model to recover relative positional information. The overall takeaway is that positional encoding is the bridge between self-attention’s parallel set-based processing and the order-sensitive nature of language.