Attention Mechanism in 1 video | Seq2Seq Networks | Encoder Decoder Architecture

Attention-based encoder–decoder models fix two core weaknesses of the classic LSTM Seq2Seq setup: they stop forcing a single, static sentence summary to carry everything, and they let the decoder dynamically “look up” the most relevant parts of the source while generating each target word. The practical result is more stable translation quality as sentence length grows, along with interpretable attention weights that reveal which source words matter for each generated output.

In the baseline encoder–decoder architecture, the encoder reads the entire input sentence step by step and compresses it into one fixed vector (a summary or representation). The decoder then generates the output sentence step by step using that same static vector. This design breaks down for longer sentences. A quick thought experiment—reading a long sentence, closing the eyes, and translating from memory—captures the intuition: humans don’t rely on a single all-purpose memory snapshot for long text. Instead, they focus on a moving window of context. In the Seq2Seq model, the encoder’s fixed summary becomes overloaded when inputs exceed roughly 25 words, because the decoder must infer too much from too little.

The second weakness appears on the decoder side. At each decoding time step, the model really needs only certain source tokens (or nearby context) to produce the next target word. For example, generating “light” should depend on the source region that contains the relevant concept, not the entire sentence. Yet the vanilla architecture feeds the same static representation at every time step, so the decoder can’t selectively emphasize the right encoder states. The transcript frames this as a mismatch: static representations are convenient, but translation decisions are inherently time-dependent.

The attention mechanism introduces a dynamic alternative. Rather than passing one fixed vector, the decoder receives a context vector c_i at each time step i. That context vector is computed as a weighted combination of encoder hidden states h_j (one per encoder time step). The weights—called attention weights or alignment scores (α)—are different for each decoder time step, effectively telling the model which encoder positions to prioritize when producing the next word.

Concretely, the transcript defines encoder hidden states h1…h4 and decoder states s1…s4, along with the decoder input y1…y3. At each decoder step i, attention produces α_i1…α_i4, then forms c_i = Σ_j α_ij h_j. Those α values are computed by an alignment function that takes the encoder hidden state h_j and the decoder’s previous hidden state s_{i−1} (denoted as s_prev or similar in the explanation). Rather than hand-designing this alignment function, the approach uses a small feed-forward neural network to approximate it, leveraging the universal function approximation idea and training via backpropagation through time.

Empirical motivation is provided via reported BLEU-score behavior: most models’ BLEU drops sharply once sentence length exceeds about 30 words, while the attention-based model stays comparatively stable. Another reported experiment plots attention weights (α) as a grid for English-to-French translation, showing that specific source words—like agreement-related tokens—receive the highest attention when generating corresponding target words. Finally, the transcript notes that the original paper used bidirectional LSTMs in the encoder, improving context by incorporating both past and future information, while leaving the attention alignment mechanism structurally the same.