Lecture 2A: Convolutional Neural Networks (Full Stack Deep Learning - Spring 2021)

Convolutional neural networks gained their edge in computer vision by replacing the “flatten an image and learn a giant matrix” approach with a sliding, weight-sharing operation that scales far better as images get larger. Fully connected networks treat every pixel as a distinct input feature, so the number of learnable weights grows rapidly with image size: a 32×32 grayscale image flattens to 1024 values, but moving to 64×64 multiplies the input dimensionality by four, and 128×128 multiplies it by sixteen—ballooning the parameter count. Convnets also address a second weakness: fully connected models are not naturally invariant to translations. If an object shifts a few pixels, the fully connected layer effectively “looks at” different pixel positions, so the model’s outputs can change unless translation robustness is engineered via augmentation.

A convolutional filter fixes both issues by operating on local patches and reusing the same learned weights across the image. Instead of multiplying a whole-image vector by a massive matrix, a conv layer extracts a small window—commonly 5×5—flattens it, and computes a dot product with a learned weight vector to produce one output value for that patch. Sliding the window across height and width yields an output map with one value per patch location. Historically, carefully chosen convolution kernels could perform interpretable image processing like blurring; modern convnets learn those weights directly from data.

The core operation extends naturally to color images and multiple feature maps. For RGB inputs shaped 32×32×3, a 5×5×3 patch contains 75 values, so each filter learns 75 weights. Applying multiple filters produces multiple channels in the output tensor: for example, 10 filters can turn a 28×28×3 input into a 28×28×10 output. Crucially, the output has the same three-dimensional tensor structure as the input, enabling stacking: one convolutional layer’s output can feed the next. Because each convolution is linear, convnets typically insert a nonlinearity (often ReLU) after each convolution to increase expressiveness.

Two practical knobs control how the filter moves and how tensor sizes change. Stride determines the step size of the sliding window; larger strides skip positions and downsample the feature map. Padding adds a border around the input (often zeros) so the filter can still be applied near edges; “same” padding is designed to keep output spatial dimensions equal to input, while “valid” uses no padding. Output sizes follow standard arithmetic based on input size, filter size, stride, and padding.

Beyond basic convolution, the lecture highlights operations that shape what the network can “see” and how it compresses information. Stacking convolutions increases the receptive field: two 3×3 layers can match the spatial coverage of a single 5×5, often with better empirical performance due to added nonlinearity. Dilated convolutions expand receptive field without increasing parameters by skipping pixels inside the kernel footprint. To reduce spatial dimensions, pooling—especially 2×2 max pooling—summarizes local regions by taking max (or sometimes average). A 1×1 convolution reduces channel depth while mixing information across channels at each spatial location.

As a baseline architecture, the lecture describes the classic LeNet-style pattern: repeated blocks of convolution + nonlinearity, followed by pooling, then fully connected layers and a softmax output. The discussion also clarifies training pragmatics, such as placing activations between fully connected layers but avoiding a nonlinearity after the final fully connected layer when the loss function expects raw logits.