Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

This paper asks whether adding an explicit visual attention mechanism to neural image caption generation improves both caption quality and interpretability. The problem matters because captioning requires more than mapping an image to a single fixed embedding: it must decide which parts of the image are relevant at each word position and express relationships in natural language. Prior “show and tell” style models typically encode an image once (e.g., using the top layer of a CNN) and then decode a sentence with an RNN, which can force the model to compress all visual information into a static vector. That compression can lose fine-grained details needed for richer captions, especially in cluttered scenes.

The authors’ contribution is an encoder–decoder captioning framework with attention over spatial CNN features. They propose two attention variants under a shared architecture: (1) a deterministic “soft” attention mechanism that computes a weighted average of image annotation vectors using attention weights ${\alpha }_{t,i}\breve{0}007$ at each decoding step, enabling end-to-end training with standard backpropagation; and (2) a stochastic “hard” attention mechanism that samples a single spatial location at each time step (a latent variable), trained by maximizing a variational lower bound using Monte Carlo gradient estimates, which is equivalent to REINFORCE-style learning.

Methodologically, the encoder extracts a set of spatial “annotation vectors” from a convolutional feature map. Concretely, they use Oxford VGG features pretrained on ImageNet without fine-tuning, taking the 14$\times$14$\times$512 feature map before max pooling. Flattening yields $L=196$ spatial locations, each represented by a $D=512$-dimensional vector ${\mathbf{a}}_i\in {\mathbb{R}}^D$. The decoder is an LSTM that generates a caption word-by-word. At each time step $t$, the LSTM produces a hidden state ${\mathbf{h}}_{t-1}$, which conditions an attention network $f_{\text{att}}$ to score each spatial location: $e_{t,i}=f_{\text{att}}({\mathbf{a}}_i,{\mathbf{h}}_{t-1})$, followed by a softmax to obtain normalized attention weights ${\alpha }_{t,i}=\frac{\exp (e_{t,i})}{{\sum }_{k=1}^L\exp (e_{t,k})}$. The context vector ${\hat{\mathbf{z}}}_t$ is then computed either as a weighted sum (soft attention) or as a sampled feature vector (hard attention). The output layer combines the previous word embedding, the LSTM hidden state, and the context vector to produce next-word probabilities.

For soft attention, the context vector is deterministic: ${\hat{\mathbf{z}}}_t={\sum }_{i=1}^L{\alpha }_{t,i}{\mathbf{a}}_i$. Training minimizes a penalized negative log-likelihood: $L_d=-\log P(\mathbf{y}|\mathbf{x})+\lambda {\sum }_i^L(1-{\sum }_t^C{\alpha }_{t,i}{)}^2$. The additional term encourages “doubly stochastic” behavior so that attention mass covers all image regions across the generated sequence (i.e., ${\sum }_t{\alpha }_{t,i}\approx 1$ for each location). They also introduce a gating scalar ${\beta }_t$ (computed from ${\mathbf{h}}_{t-1}$) that scales the context vector, $\varphi (\{{\mathbf{a}}_i\},\{{\alpha }_i\})=\beta {\sum }_i{\alpha }_i{\mathbf{a}}_i$, to emphasize salient objects.

For hard attention, they treat the attended location as a one-hot latent variable $s_t$ drawn from a multinoulli distribution parameterized by ${\alpha }_{t,i}$. The context becomes a random variable ${\hat{\mathbf{z}}}_t={\sum }_i{s}_{t,i}{\mathbf{a}}_i$. They optimize a variational lower bound $L_s$ on $\log p(\mathbf{y}|\mathbf{a})$, approximating gradients via Monte Carlo sampling of $s_t$. To reduce variance, they use a moving-average baseline $b_k=0.9b_{k-1}+0.1\log p(\mathbf{y}|{\tilde{s}}_k,\mathbf{a})$, add an entropy regularizer on the attention distribution, and use a hybrid strategy where with probability 0.5 they replace the sampled location with its expected value $\alpha$. They report that this hard attention formulation is equivalent to REINFORCE where the reward is proportional to the log-likelihood of the target sentence under the sampled attention trajectory.

The experiments evaluate caption quality on Flickr8k (8,000 images), Flickr30k (30,000 images), and MS COCO (82,783 images). The datasets provide multiple reference captions per image (5 for Flickr; COCO has variable counts, and they discard extra references for consistency). They use a fixed vocabulary size of 10,000. They report BLEU-1 through BLEU-4 (without brevity penalty) and METEOR, and compare against prior methods under comparable feature extraction settings (noting that some baselines use AlexNet while their main comparisons use GoogLeNet/Oxford VGG features).

The key quantitative results are summarized in their Table 1. On Flickr8k, their soft-attention model achieves BLEU-1 = 63, BLEU-2 = 41, BLEU-3 = 27, BLEU-4 = 17.7, and METEOR = 27.7 (the table also lists hard-attention values of BLEU-1 = 65.6, BLEU-2 = 42.4, BLEU-3 = 29.9, BLEU-4 = 19.5, and METEOR = 31.4). On Flickr30k, soft attention yields BLEU-1 = 66.3, BLEU-2 = 42.3, BLEU-3 = 27.7, BLEU-4 = 18.3, and METEOR = 17.1, while hard attention yields BLEU-1 = 66.9, BLEU-2 = 43.9, BLEU-3 = 28.8, BLEU-4 = 19.1, and METEOR = 19.9. On MS COCO, the reported soft-attention results include BLEU-1 = 64.2, BLEU-2 = 45.1, BLEU-3 = 30.4, BLEU-4 = 20.3, and METEOR = 20.41; hard attention reports BLEU-1 = 66.6, BLEU-2 = 46.1, BLEU-3 = 32.9, BLEU-4 = 23.04, and METEOR = 25.0 (the table shows multiple baseline rows and some missing metrics).

Overall, the authors claim state-of-the-art performance on all three datasets and emphasize that their approach uses a single model (not an ensemble) while still outperforming prior systems. They also note that they improve METEOR on MS COCO, attributing this partly to their regularization and the use of lower-level convolutional features that preserve more information than a single high-level vector.

Beyond numbers, the paper provides qualitative evidence by visualizing attention weights over the 14$\times$14 spatial grid (upsampled and smoothed). They argue that the learned alignments correspond well to human intuition and that the hard/soft attention mechanisms can attend to non-object salient regions, unlike some prior attention methods that rely on object detectors to define candidate alignment targets. They further suggest that attention visualizations can help diagnose errors by revealing where the model “looked” when generating a word.

Limitations are not exhaustively quantified in the provided text, but several constraints are apparent from the methodology. First, the encoder is fixed to pretrained VGG features without fine-tuning, which may limit ceiling performance relative to fully end-to-end vision-language training. Second, hard attention relies on Monte Carlo sampling and variance-reduction tricks, which can be less stable and more expensive than soft attention; the paper does not provide a detailed compute comparison beyond noting training time for soft attention on COCO (less than 3 days on an NVIDIA Titan Black GPU). Third, evaluation relies on BLEU and METEOR, which are known to have shortcomings for semantic faithfulness; the authors acknowledge BLEU criticism by including METEOR.

Practically, the results matter for anyone building image captioning systems or other vision-language models that require word-by-word grounding. The work provides a clear recipe for implementing attention over spatial CNN features, demonstrates that attention improves caption metrics on major benchmarks, and offers interpretability through attention maps. Researchers and practitioners should care because the approach is modular (encoder-decoder with attention) and can be adapted to other tasks where dynamic selection of visual evidence is needed, such as visual question answering, referring expression generation, and multimodal translation.

In sum, the paper shows that learning to attend over spatial features—via differentiable soft attention or variationally trained hard attention—substantially improves neural image caption generation quality and yields interpretable alignments, establishing attention as a core component of modern captioning systems.