Lecture 8: Troubleshooting Deep Neural Networks - Full Stack Deep Learning - March 2019

Troubleshooting deep neural networks is hard not because training is mysterious, but because the same drop in performance can come from many different failures—often with bugs that fail silently. The core message is that top practitioners treat debugging like a decision tree: start with the simplest, most controllable setup, verify it works end-to-end, then gradually add complexity while using validation signals to decide whether to improve the model, the data, or the training setup.

The lecture frames why debugging dominates real-world deep learning work. Even advanced teams can spend most of their time on “invisible” issues: implementation mistakes, unstable numerics, mismatched tensor shapes, and subtle preprocessing errors. A concrete example comes from a training pipeline where file ordering was nondeterministic—using Python glob without sorting mixed up features and labels, causing training to fail while looking superficially plausible. Hyperparameters can also derail learning entirely: learning rate choices that are too small or too large can prevent convergence, and certain initialization schemes can be essential for networks like ResNet to train at all. Model choice matters too; copying an architecture that worked on ImageNet doesn’t guarantee it will work on very different domains such as self-driving car imagery. Finally, data construction is a frequent culprit: insufficient data, class imbalance, noisy labels, and distribution shift between training and testing can all produce the same symptom—worse-than-expected accuracy.

Because so many causes can look identical from the outside, the lecture argues for a pessimistic mindset and a disciplined workflow. The recommended strategy is a loop: begin with the simplest possible architecture and dataset, implement and debug it, evaluate performance, then make a targeted decision—improve the model, improve the data, or tune hyperparameters—before ramping up complexity. “Start simple” isn’t just philosophical; it’s operationalized through steps like overfitting a single batch. If a model can’t drive training error near zero on a tiny slice of data, the problem is likely a bug in the pipeline (shapes, casting, loss inputs, preprocessing) rather than a lack of model capacity.

The lecture then lays out practical defaults for early debugging: use Adam with a learning rate of 3e-4, start with ReLU for fully connected or cloned networks and LSTM with tanh (noting typical initialization choices), avoid regularization and batch normalization at first because they can introduce additional failure modes, and normalize inputs by subtracting mean and dividing by variance (while avoiding common mistakes like dividing by 255 twice). It also recommends simplifying the dataset—fewer classes, smaller images, or synthetic data—so the team can validate the training loop quickly.

Once training runs and a single batch can be overfit, the next step is comparing against known results: official implementations from paper authors are best, otherwise benchmark datasets like MNIST can reveal pipeline bugs, and random GitHub reimplementations should be treated cautiously because many contain serious errors. After that, performance decisions should be guided by bias-variance decomposition: training error indicates underfitting, the gap between training and validation indicates variance/overfitting to training, and validation-to-test gaps reveal distribution shift or overfitting to the validation set. The order of fixes follows that decomposition—address underfitting first, then overfitting, then distribution shift, and finally validation overfitting.

Hyperparameter tuning is treated as an empirical, often manual process (“graduate student descent”), with random search and coarse-to-fine strategies often outperforming naive grid search. Bayesian optimization is presented as a later-stage option once the codebase and experimentation pipeline are mature. The overall takeaway is a method for turning an overwhelming debugging space into a sequence of verifiable checkpoints, so complexity is added only after earlier assumptions are proven safe.