Resource Management (4) - Infrastructure & Tooling - Full Stack Deep Learning

Resource management in deep learning is about making shared compute usable: multiple people need to launch experiments quickly, with dependencies handled and the right GPUs and CPUs allocated—without fighting over hardware. The spectrum runs from low-tech coordination (spreadsheets reserving machines) to full systems built for machine learning clusters. A more practical middle step is automation on a single machine: a short script can allocate free GPUs to incoming jobs so experiments start with the correct resources and minimal manual bookkeeping.

For larger teams and multi-user environments, workload managers like SLURM provide the standard approach. Users declare requirements—such as “two GPUs, eight CPUs, and 12 GB of RAM”—then submit a job. The scheduler queues it, waits until the requested resources are available, locks them down to prevent contention, and runs the job. That model removes the need for manual reservation and makes it easy to run repeatable batches of experiments.

Container tooling then addresses the dependency problem. Docker packages an entire software stack into a lightweight unit, avoiding the heavier overhead of full virtual machines. Kubernetes takes this further by orchestrating many Docker containers across a cluster, allocating compute to containers as resources become available. In the course’s compute setup, Kubernetes runs a JupyterHub environment (JupyterHub with JupyterLab) across shared GPUs and CPUs, enabling multiple users to work concurrently on the same underlying hardware.

On top of Kubernetes, machine-learning-specific platforms aim to handle common training patterns. Kubeflow (an open-source project associated with Google, with contributions now largely coming from outside Google) includes modules for launching and managing Jupyter notebooks with specified GPU counts—so a user can request a “two-GPU notebook” without knowing which physical machine will host it. Kubeflow also supports multi-step workflows, where preprocessing and data preparation can be CPU-heavy while later training is GPU-heavy. For example, downloading and cropping a large image dataset may not benefit from GPUs initially, but the subsequent concatenation and training do. The platform’s goal is to allocate different resources at different stages rather than overprovisioning everything for the entire pipeline.

The discussion also notes related workflow tooling such as Polyaxon, described as open source with enterprise features, and suggested to be more actively developed than Kubeflow. Finally, a practical question arises about cost and utilization: in the cloud, teams can request 50 GPUs for a day and stop paying afterward, but on-prem hardware can sit idle. The conversation floats the idea of sharing or contributing idle on-prem GPUs to a peer-to-peer style “cloud,” potentially combining ownership with compensation—though no concrete solution is provided. Overall, Kubernetes is positioned as the dominant container orchestration layer for packing workloads onto shared infrastructure, with higher-level ML platforms like Kubeflow handling the workflow and user experience on top of it.