Frameworks & Distributed Training (5) - Infrastructure & Tooling - Full Stack Deep Learning

Deep learning frameworks have shifted from “fast in production, painful in development” toward a convergence where developers write in Python with immediate feedback, while the system still produces optimized execution graphs for deployment. Early ecosystems split along a two-axis tradeoff: Caffe (written in C++ at Berkeley) delivered high performance and straightforward GPU/CUDA execution, but adding new layer types or wiring unusual computation graphs required C++ work and manual backward implementations. TensorFlow (2015) improved hardware portability and production readiness, yet its original static-graph style required describing computation indirectly, making iteration and debugging harder.

Keras (built on top of TensorFlow and other libraries like Theano) made model building more ergonomic by letting developers declare models with less graph “meta-programming,” but the core runtime still reflected TensorFlow’s graph-first approach. PyTorch (2017) flipped the development experience by executing the forward pass directly in Python, enabling standard debugging workflows such as breakpoints inside model code. That development-first design raised production concerns—until TensorFlow added eager execution and PyTorch introduced TorchScript, an optimized execution graph that can run efficiently on constrained systems, including mobile.

In practice, most new projects start with PyTorch, with TensorFlow 2.0 (released recently relative to the discussion) serving as an upgrade path for older TensorFlow 1.x models. The training ecosystem also matters: fastai provides training utilities—learning-rate finding and out-of-the-box model components—so teams can begin with strong defaults and diverge only when needed. Research and hiring signals reinforce the momentum: an academic analysis (“the gradient”) found PyTorch mentions rising sharply in major machine learning conferences (from near zero to close to 80% over a few years), while TensorFlow and Keras still lead job posts but lag research workflows.

The discussion then pivots to distributed training, where a single training run uses multiple GPUs or multiple machines to reduce iteration time—especially when datasets or models are too large for quick single-device passes. Two strategies dominate. Data parallelism replicates the model on each GPU and splits each batch across devices; after backprop, gradients synchronize so weights stay tied. For convolutional networks, speedups are close to linear at small scales (roughly 2 GPUs ~1.8–2x, 4 GPUs ~3–3.5x), though not perfectly linear.

Model parallelism becomes necessary only when the model cannot fit in one GPU’s memory, partitioning the model graph so different weights live on different devices. It’s more complex and generally not worth it unless memory constraints force the issue.

Framework support makes data parallel relatively straightforward: TensorFlow’s Distributed Mirrored Strategy and PyTorch’s data-parallel wrapping can automatically split batches and synchronize weights. Multi-machine distribution is harder because it introduces parameter servers and inter-process coordination. That’s where tools like Ray (for stateful distributed computing in Python) and Horovod (Uber’s approach using MPI instead of TensorFlow’s parameter-server machinery) enter as practical alternatives. The Q&A also highlights a recurring bottleneck: data loading. Best practice is to lean on framework-native input pipelines—TFRecord and TensorFlow’s TF data API in TensorFlow, or PyTorch’s DataLoader—then tune prefetching and disk/cloud-to-GPU transfer timing to keep GPUs fed. Overall, the infrastructure goal is to make scaling easier than it used to be, with distributed training increasingly feasible as frameworks and tooling mature—setting up a later comparison with managed platforms like SageMaker.