Transformer Architecture | Part 1 Encoder Architecture | CampusX

Transformer encoder architecture is built from a repeating pattern: each encoder block takes token embeddings (augmented with positional information), runs multi-head self-attention, then passes the result through a two-layer feed-forward network, with residual connections and layer normalization wrapping both sub-steps. The practical payoff is that the model can convert a sequence like “How are you” into context-aware representations—while keeping tensor shapes consistent (512-dimensional vectors per token) so the same block can be stacked six times.

The walkthrough starts by reframing the famous transformer diagram into something easier to reason about: a transformer is split into an encoder side and a decoder side, and each side contains multiple identical blocks. In the original “Attention Is All You Need” setup, the encoder uses six encoder blocks and the decoder uses six decoder blocks. Because the blocks are architecturally identical, understanding one encoder block unlocks the rest; the only difference across blocks is that each has its own trainable parameters that get updated during backpropagation.

Before any encoder block runs, the input sentence goes through an “input block” with three steps. First comes tokenization: the sentence is split into tokens (the example assumes word-level tokenization, producing tokens like “How”, “are”, “you”). Second is embedding: each token becomes a 512-dimensional vector via an embedding layer, since models operate on numbers rather than raw text. Third is positional encoding: because embeddings alone don’t tell the model which word comes first or last, positional vectors are generated for each token position (also 512-dimensional) and added to the token embeddings. After this, the encoder receives a sequence of 512-dimensional vectors.

Inside an encoder block, multi-head self-attention is applied first. The key motivation is contextualization: the same word embedding can represent different meanings depending on surrounding words (e.g., “bank” in different contexts). Self-attention lets each token’s representation change based on other tokens in the sequence, producing context-aware outputs (denoted as z1, z2, z3 for the example tokens). Multi-head attention repeats this idea across multiple attention “views,” yielding richer, more diverse contextual signals.

Next, residual connections and layer normalization stabilize the computation. The block adds the attention output back to the original input vectors (a skip path that bypasses the attention transformation) and then normalizes each token vector using layer norm (mean/standard deviation over the 512 dimensions, plus learned parameters gamma and beta). This normalization is presented as a training-stability mechanism: without it, attention outputs can drift into uncontrolled numeric ranges.

Then the feed-forward network runs. It’s a two-layer MLP applied position-wise: the first layer expands from 512 to 2048 units with ReLU nonlinearity, and the second layer projects back from 2048 to 512 with a linear activation. The transcript emphasizes the shape flow: after the first layer, token representations become 3×2048 (for three tokens), then return to 3×512 after the second layer. The residual connection and layer normalization wrap this feed-forward step as well.

Finally, the encoder block output (still 3×512 in the example) becomes the input to the next encoder block. This repeats six times, producing the encoder’s final token representations that are then handed off to the decoder. The transcript closes by addressing common “why” questions: residual connections are linked to stable training and preserving useful features; feed-forward layers are tied to introducing nonlinearity and representational capacity; and stacking multiple encoder blocks is justified by the need for stronger representation power to capture human language patterns.