Lecture 3: Recurrent Neural Networks (Full Stack Deep Learning - Spring 2021)

Recurrent neural networks (RNNs) were built to handle sequence data efficiently by reusing the same weights across time and carrying information forward in a hidden state—an approach meant to exploit patterns that repeat over a sequence rather than treating every position as unrelated. The lecture frames sequence modeling as a set of problem types—time-series forecasting, translation, speech, image captioning, and question answering—then argues that feedforward networks struggle with variable-length sequences and become data-inefficient when they must learn position-specific mappings for repeated patterns.

In the core RNN setup, each time step takes the current input x_t and the previous hidden state h_{t-1}, computes a new hidden state h_t, and produces an output y_t. The hidden-state update is described as two matrix multiplications—one for h_{t-1} and one for x_t—summed and passed through an activation function (the basic form uses tanh). This weight sharing across time replaces the “single massive matrix” approach and makes the architecture naturally suited to sequences of varying length. For tasks where only one output is needed at the end (many-to-one), the last hidden state is fed into a classifier; for tasks where an output sequence must be generated from a single input (one-to-many), an encoder-decoder pattern is used: an encoder compresses the input into an initial state, and a decoder generates outputs step-by-step.

The lecture then zooms in on why vanilla RNNs break down on long sequences: backpropagation through time multiplies gradients across many steps, and with common activations like sigmoid and tanh, derivatives shrink toward zero, leading to vanishing gradients. The result is that long-term dependencies—like remembering a character name introduced early in a long sentence—get lost. LSTMs (Long Short-Term Memory networks) address this by adding a separate cell state plus gating mechanisms. A forget gate decides what to remove from the cell state, an input gate controls what new information to add, and the output computation uses a gated version of the cell state to produce the next hidden state. The lecture notes that GRUs and other variants exist, and cites empirical findings suggesting regular LSTMs are hard to beat across datasets, while GRUs can sometimes train more easily or perform slightly better.

Machine translation serves as a case study for how RNN encoder-decoder systems were made practical. The baseline idea—encode the source sentence with an RNN and initialize the decoder with the final encoder state—creates an information bottleneck for long sentences. Improvements include stacking LSTM layers (with residual connections to make deeper stacks trainable), adding attention so the decoder can access a weighted summary of all encoder hidden states instead of only the last one, and using bidirectionality so the encoder reads the sentence in both forward and backward directions. Attention is described via relevance weights over encoder states, producing attention maps that show which source words influence each target word.

Finally, the lecture introduces CTC loss (Connectionist Temporal Classification) for sequence tasks with misalignment, using handwriting recognition as the motivating example. CTC allows the model to emit characters or a special blank (epsilon) token at each time step, then merges repeated characters and removes blanks to produce the final transcription—handling cases where input and output lengths differ. The lecture closes by weighing RNN trade-offs: flexible architectures and strong historical performance versus slower, less parallelizable training, which helped open the door for transformers. It previews non-recurrent alternatives such as WaveNet-style convolutional sequence models, using causal and dilated convolutions to capture long-range context while enabling parallel training, at the cost of more expensive inference.