Self Attention Geometric Intuition | How to Visualize Self Attention | CampusX

Self-attention in Transformers can be visualized as a geometry-driven “pull” between word embeddings: each token’s new representation is a weighted combination of other tokens, where the weights come from similarity scores computed via dot products. That geometric intuition matters because it turns a dense block of matrix math into something you can picture—angles between vectors determine which words influence each other, and the softmax turns those similarities into attention weights.

The walkthrough starts with a quick recap of the standard self-attention pipeline on a two-word example like “money bank.” First, each word gets an embedding vector (e.g., e_money and e_bank). Then three learned projection matrices—W_Q, W_K, and W_V—transform these embeddings into query, key, and value vectors. For each word, the model computes similarity scores by taking dot products between queries and keys (Q·K), scales them by 1/√d_k to control magnitude, and applies softmax to convert scores into normalized weights. Finally, each token’s output vector is formed as a weighted sum of value vectors (the V vectors), using those attention weights.

The core addition in this session is the geometric lens. Embeddings are treated as vectors in a space (the transcript uses a simplified 2D visualization for intuition, even though real embeddings are high-dimensional). The projection step is described as a linear transformation: dotting a vector with a matrix “moves” it into new vector directions, producing Q, K, and V. Next, similarity becomes geometry: the dot product is tied to the angle between vectors—smaller angular distance yields a larger dot product, which yields larger attention after scaling and softmax.

Using the “money bank” pair, the intuition is that the “bank” token’s output vector (y_bank) shifts toward the “money” token’s embedding direction after attention is applied. In the transcript’s comparison, the original bank embedding sits farther away, but the self-attended bank representation ends up much closer to money. That’s framed as gravity-like behavior: the context word “money” pulls “bank” into a context-appropriate meaning.

The explanation extends this idea to counterfactual context. If the surrounding word were “river” instead of “money,” the model would pull “bank” toward the “river” context, producing a different contextual representation. The mechanism is described as context-aware because attention weights depend on the current sentence’s token relationships—learned patterns from the dataset decide which words should influence each other.

To make the geometry feel tangible, the transcript also references embedding visualizations using dimensionality reduction (PCA) to project high-dimensional word vectors into 2D/3D plots. In those plots, semantically related words cluster together, reinforcing the idea that embeddings encode meaning as spatial relationships. Overall, the takeaway is that self-attention is not just computation—it’s a context-driven re-composition of token meaning based on vector similarity and weighted aggregation.