Lecture 02: Development Infrastructure & Tooling (FSDL 2022)

Machine learning development runs on a “data flywheel,” but getting from an idea to a reliable system at scale depends on disciplined software engineering—especially around tooling, reproducibility, and training infrastructure. The core workflow starts long before model code: teams must aggregate and clean data, label it, choose or train an architecture and pre-trained weights, iterate on model code through debugging and experiments, then deploy and monitor predictions. After deployment, user activity generates fresh data that must be fed back into the dataset, closing the loop.

That reality splits development into three practical layers: data and preparation (yellow), model development (the middle), and deployment (green). The lecture argues that most friction happens in the middle—where code, experiments, and environments must stay consistent while models evolve. Python is treated as the default language for ML because its ecosystem dominates scientific and data computing. From there, the tooling stack matters: editors like VS Code (recommended in the course) or PyCharm, notebook environments such as Jupyter/JupyterLab, and notebook workflows that enable fast feedback.

Notebooks are praised as a “first draft” environment because they provide immediate output and encourage rapid iteration. But they also create problems: limited refactoring support, weak versioning of cell outputs, out-of-order execution artifacts, and poor fit for unit testing. The lecture points to nbdev as a way to unify documentation, code, and tests inside notebooks, while still leveraging VS Code’s notebook support, remote editing via SSH, and debugging features like breakpoints.

For interactive sharing, Streamlit is highlighted as a way to turn Python scripts into lightweight web apps with widgets and efficient reruns. The lecture then shifts to environment management, emphasizing that deep learning setups are fragile: CUDA versions, Python versions, and library versions (e.g., PyTorch, NumPy) must align. A reproducible approach uses environment.yaml plus conda for Python/CUDA, then pip-tools to resolve compatible package versions and lock them so experiments can be recreated later. Makefiles can streamline common commands.

On the framework side, the lecture frames the choice as ecosystem and engineering fit. PyTorch is positioned as the full-stack default: it’s widely used in research and industry, supports CPU/GPU/TPU/mobile via an optimized execution graph, and has a strong distributed training ecosystem. PyTorch Lightning is recommended for structuring model code, optimizers, training loops, evaluation, and data loaders—making it easier to switch between single- and multi-device training with minimal code changes. Alternatives are acknowledged: TensorFlow (including Keras for layer composition), JAX for research and general vectorization/auto-differentiation, and meta-frameworks like Flax/Haiku.

Finally, scaling and compute management are treated as part of development infrastructure rather than an afterthought. Distributed training ranges from data parallelism (replicating the model across GPUs and averaging gradients) to sharded approaches for models that don’t fit on one GPU. Techniques like ZeRO-style optimizer/model sharding and Fully Sharded Data Parallel (FSDP) can cut memory usage dramatically, enabling much larger batch sizes and parameter counts. The lecture also surveys compute choices—NVIDIA GPUs, TPUs on GCP—and stresses that cost comparisons must consider time-to-train, not just hourly rates. For experiment and model management, it highlights tools such as TensorBoard, MLflow, Weights & Biases, and hyperparameter sweeps (e.g., Hyperband), plus all-in-one platforms that combine training, tracking, and deployment—while deferring deeper data and deployment topics to later weeks.