Attention Is All You Need

Summary The Transformer is a pioneering neural network architecture that abandons recurrent and convolutional layers in favor of a mechanism called self-attention to process sequences. By allowing for significantly higher parallelization during training, it achieves state-of-the-art results in machine translation and parsing with drastically reduced training times.

Q: How does the Transformer architecture differ from traditional RNN and CNN models in handling sequence data? A: Unlike Recurrent Neural Networks (RNNs) that process tokens sequentially—creating a bottleneck that prevents parallelization—the Transformer uses self-attention to relate all positions in a sequence simultaneously. While Convolutional Neural Networks (CNNs) can parallelize, they require multiple layers to connect distant signals; the Transformer reduces this to a constant number of operations, facilitating the learning of long-range dependencies.

Q: What is the specific function and mathematical implementation of "Scaled Dot-Product Attention"? A: It maps a query (Q) and a set of key-value (K, V) pairs to an output. The model computes the dot products of the query with all keys, scales them by the square root of the dimension of the keys (1/√dk) to prevent gradient vanishing in the softmax layer, and applies a softmax function to determine the weights assigned to the values. The formula is Attention(Q, K, V) = softmax(QKᵀ/√dk)V.

Q: What is "Multi-Head Attention" and why is it beneficial? A: Instead of a single attention function, the model performs attention in parallel across multiple "heads" (8 in the base model). Each head projects the queries, keys, and values into different learned linear subspaces. This allows the model to simultaneously attend to information from different representation subspaces at different positions, which would otherwise be lost due to averaging in a single attention head.

Q: How does the model account for the order of words without using recurrence? A: The Transformer employs "Positional Encodings" added to the input embeddings at the bottom of the encoder and decoder stacks. These encodings use sine and cosine functions of different frequencies to inject information about the relative or absolute position of tokens. This ensures the model can utilize the sequence order despite the inherently non-sequential nature of the attention mechanism.

Q: Describe the internal structure of the Encoder and Decoder stacks. A: Both consist of N=6 identical layers. Each encoder layer has two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network. The decoder adds a third sub-layer that performs multi-head attention over the encoder's output. Both use residual connections around sub-layers followed by layer normalization to facilitate deep signal flow.

Review Questions

1. Why is the scaling factor (1/√dk) necessary in the attention mechanism for larger dimensions?
2. In what way does the decoder's self-attention layer differ from the encoder's to maintain the auto-regressive property?
3. What were the primary performance advantages of the Transformer (big) model compared to previous state-of-the-art models like GNMT or ConvS2S?