Lecture 4: Infrastructure and Tooling - Full Stack Deep Learning - March 2019

Deep learning success depends less on model architecture than on building an end-to-end system that can ingest data, train reliably, deploy safely, and keep improving after launch. The “ideal” loop starts with labeled data and ends with an accurate prediction service, but real-world work is dominated by infrastructure tasks: aggregating and cleaning data, labeling and versioning it, writing and debugging training code, running many experiments to tune parameters, storing results, testing, deploying to production, and then monitoring predictions in the wild. When models fail—often because user data drifts—teams need feedback to relabel or review low-confidence cases and feed that back into training, creating a data flywheel that steadily improves the system.

That operational reality is why machine learning can accumulate technical debt quickly. Google’s framing likens ML to a “high-interest credit card” of technical debt: shipping a working model is only the beginning, while scaling requires refactoring, better tooling, and careful handling of data pipelines, training jobs, analysis, serving, and monitoring. A striking implication follows from the codebase breakdown at scale: the actual neural network code is a small slice of the total system. Most engineering effort goes into where data comes from, how features are extracted across services, how training jobs are allocated to machines, how models are analyzed, how predictions are served, and how production behavior is monitored.

From there, the lecture lays out a tooling and infrastructure landscape for a full ML codebase, organized by layers. At the bottom are data storage and retrieval, then data workflows for processing and labeling, and dataset versioning to handle continuously changing data. Above that sit development tooling (editors, notebooks, deep learning frameworks), resource management (scheduling jobs across GPUs and machines), experiment management (tracking runs, parameters, and metrics), and distributed training (multiple GPUs for one model). Deployment adds continuous integration and testing so upgrades—like moving between TensorFlow versions—don’t silently break models, plus web/mobile serving constraints and model interchange formats for running trained models in different runtimes.

The lecture also compares major deep learning frameworks along two axes: ease of development and scalability/production readiness. Caffe is optimized for production but requires more low-level work in C++. TensorFlow targets production and deployment across commodity servers and mobile, but its computational graph abstraction can make debugging harder. Keras improves development ergonomics as a front-end to TensorFlow. PyTorch is praised for development speed because models are written in Python and executed directly, making debugging and research iteration easier; it can still be productionized by compiling to optimized graphs and using Caffe2 for execution. The practical recommendation: use TensorFlow with Keras or PyTorch unless there’s a clear reason to deviate.

Finally, the lecture turns to hardware and cloud strategy. NVIDIA dominates for now, with architectures moving from Kepler/Maxwell through Pascal/Volta and toward newer generations; cloud providers offer different GPU families and configurations. The decision between on-prem and cloud hinges on constraints like GPU availability, cost, data locality, and privacy. Cloud can be cheaper for short bursts and large-scale experimentation, especially when spot/preemptible instances reduce cost, but on-prem can win when workloads are steady and predictable. Resource management tooling ranges from simple scripts and spreadsheets to Slurm, Docker, and Kubernetes, with specialized ML layers like Kubeflow and RiseML to make large experiment fleets manageable.

The throughline is that teams increasingly converge on a standardized ML workflow: prepare data, train and evaluate, tune, deploy, serve predictions, and monitor model performance—then repeat. Big “all-in-one” platforms such as Google Cloud ML Engine, Amazon SageMaker, and open-source stacks like Kubeflow aim to package these steps, while startups fill gaps in experiment tracking, hyperparameter optimization, and model lifecycle management.