Lecture 1: Introduction to Deep Learning - Full Stack Deep Learning - March 2019

Deep learning’s breakthrough in 2012 wasn’t just a better model—it replaced hand-crafted image features with learned representations, turning “what to look for” into something the system figures out from data. The core shift was moving from engineered pipelines (like edge histograms fed into support vector machines) to training large neural networks end-to-end, where millions of parameters encode the decision logic. Instead of designing invariances such as “edges stay put under lighting changes,” researchers learned those invariances directly by optimizing weights using labeled examples.

The lecture uses the ImageNet era to show why this mattered. In 2010 and 2011, traditional computer vision approaches improved slowly, with error rates around 30% for the winning system. In 2012, a neural network with roughly 60 million parameters produced a major jump and then accelerated progress in subsequent years. The explanation offered is practical as much as theoretical: neural networks can approximate complex functions, but they need large datasets to determine which function to learn. With enough labeled images (ImageNet’s scale is described as over a million), training becomes feasible, and the learned weights act like a program discovered through data rather than written by hand.

That same “learn input-to-output mappings” idea extends beyond image classification. Neural networks can be trained to generate captions from images using datasets that pair pictures with text, and multiple research groups reportedly achieved similar results quickly once the data and expertise were in place. The lecture then pushes the argument further: captioning can be seen as pattern matching, but question answering about an image is closer to understanding. A dataset pairing images, questions, and answers demonstrates this capability with examples like identifying vegetables and counting school buses.

Applications also moved from lab benchmarks into domains where humans historically required extensive training. In medicine, the lecture points to neural networks matching long-established diagnostic workflows in areas like radiology and cancer screening, and notes that by April 2018 an FDA-approved neural network could support decisions for diabetes-related retinal referrals. Autonomous driving is framed as another major frontier, with many companies pursuing the market, though outcomes remain uncertain.

Why did deep learning suddenly surge after decades of earlier neural network work? Three drivers are highlighted: abundant data, more compute, and training methods that made large models practical. Backpropagation and convolutional architectures existed earlier, but the lecture argues that scaling data and compute changed what was achievable. Training runs are described as expensive—one strong example took about six days on state-of-the-art GPUs at the time—and progress depends on repeated experiments across hyperparameters and architectures.

The session closes by addressing skepticism around robustness, especially adversarial examples. The lecture’s framing is that random noise rarely breaks models, but carefully optimized small perturbations can flip predictions because high-dimensional spaces contain directions that change outputs with minimal pixel changes. Defenses include adversarial training (adding adversarially generated examples into the training set) and other robustness techniques, with progress ongoing rather than “solved.” It also notes that adversarial perturbations can be generated automatically by optimizing pixels to maximize a target class, effectively running the same gradient-based machinery used in training—just applied to the input instead of the weights.