Lecture 1: Deep Learning Fundamentals (Full Stack Deep Learning - Spring 2021)

Deep learning fundamentals hinge on a simple but powerful idea: neural networks are flexible function approximators whose weights can be trained by minimizing a loss function using gradient-based optimization. That combination—universal approximation plus practical training via gradient descent and back propagation—explains why neural networks can tackle everything from image recognition to language modeling, and why modern compute (especially GPUs) made the approach scalable.

The lecture starts with the biological metaphor of neurons, then translates it into the perceptron: inputs are weighted (w) and shifted by a bias (b), summed, and passed through an activation function that decides whether the neuron “fires.” Classic activations like the sigmoid squash outputs into a 0–1 range, while hyperbolic tangent offers a similar smooth nonlinearity. The modern workhorse is the rectified linear unit (ReLU), defined as max(0, x), which outputs zero for negative inputs and passes positive values through. ReLU’s gradient is simple (1 when x>0, otherwise 0), and its adoption is credited as a key ingredient in the deep learning revival around 2013.

Neural networks become “networks” by stacking perceptrons into layers: an input layer, one or more hidden layers, and an output layer. Each layer’s weights and biases determine how the model transforms inputs into predictions. Theoretical results support the intuition that sufficiently wide two-layer networks (one hidden layer) can approximate essentially any function—this is the universal approximation theorem. The intuition offered is that many hidden units can act like a collection of “basis” components that combine to reproduce complex shapes, reminiscent of how Fourier-style representations can model complicated signals.

From there, the lecture maps neural networks onto major machine learning problem types. Supervised learning trains on labeled pairs (x, y), such as images mapped to categories (cat/not cat) or audio mapped to spoken content. Unsupervised learning uses unlabeled data (x only) to discover structure—examples include next-character prediction for language modeling, learning word relationships via vector representations, and learning compact image representations through compression and reconstruction. Reinforcement learning trains an agent to choose actions in an environment, receiving feedback as rewards; the state changes after each action, and the goal is to learn strategies that maximize long-term outcomes.

Training is framed as empirical risk minimization: choose model parameters to minimize a loss function. For regression, squared error is a common loss; for classification, cross entropy loss is typical because outputs correspond to discrete categories. Optimization proceeds with gradient descent: update each parameter by moving opposite the gradient of the loss, scaled by a learning rate (α). Because full-dataset updates are expensive, the lecture emphasizes stochastic and mini-batch variants that trade noisier updates for faster progress. Back propagation then provides an efficient way to compute gradients through layered computations using the chain rule, typically handled automatically by tools like PyTorch or TensorFlow.

Finally, architectural choices and compute infrastructure matter. Convolutional networks encode locality for vision, recurrent networks capture sequence structure, and techniques like skip connections help gradient flow in deeper models. The deep learning boom is also tied to better GPU support: Nvidia CUDA enabled GPUs—originally built for graphics—to accelerate the matrix multiplications that dominate neural network workloads, making large-scale training practical.