Lab 04: Experiment Management (FSDL 2022)

Experiment management is the difference between “useful training output” and “lost knowledge.” During model training, metrics like loss and validation accuracy stream to the command line, but without structured logging those numbers get overwritten, vanish when the notebook restarts, and make it impossible to reconstruct which hyperparameters, git state, or timestamps produced a given result. The lab frames this as a core engineering need: preserve metrics over time and attach the surrounding context—code version, system details, and other metadata—so experiments can be compared, debugged, and improved.

TensorBoard is introduced as the traditional baseline for tracking training curves. It runs as a separate service and visualizes logged metrics as charts over time, which works well for inspecting a single run. But once multiple experiments enter the picture, the workflow becomes harder: comparing across runs and managing the lifecycle of the logging service adds friction, and the setup can distract many ML engineers from building models and products.

Weights & Biases (W&B) is presented as the recommended upgrade for experiment management, largely because it integrates smoothly with popular deep learning stacks (including PyTorch Lightning, Keras, and TensorBoard) and improves both the logging experience and the interface for reviewing results. The integration can be added to an experiment script with only a handful of lines, and the lab walks through what those additions unlock in the W&B web UI while training is still running. Metrics update live, and the run page is organized into tabs that separate concerns: an Overview tab for run metadata (start time, duration, OS/Python version, git repository state, command-line arguments, and final metrics), a System tab for CPU/GPU utilization and memory (useful for catching performance regressions), a Model tab enabled by a single watch call, a Logs tab capturing terminal output, and a Files tab that includes environment snapshots and diffs against version control.

W&B also adds higher-value artifacts and reproducibility features. The Artifacts tab stores generated binaries such as model checkpoints and input/output media from the text recognizer, versioned like directories with lineage tracking. The lab highlights that these artifacts can be tagged (for example, “best” checkpoints via PyTorch Lightning callbacks), and that W&B can track which runs created or consumed which artifacts—critical for debugging and for tracing model behavior back to data and code.

Beyond browsing, the lab emphasizes programmatic access via the W&B API, enabling workflows like pulling logged tables into pandas, analyzing augmented data samples, and traversing the run→artifact→run graph to diagnose issues in data processing pipelines. Because raw logs can become overwhelming, W&B Reports are introduced as a way to package results into shareable, structured dashboards and PR-friendly summaries, linking metrics back to git changes.

Finally, the lab scales from single experiments to large projects and hyperparameter optimization. It demonstrates filtering and custom charts across many runs (including derived metrics like generalization gap) and shows how W&B can orchestrate hyperparameter sweeps using a YAML-configured controller with lightweight agents. The exercises encourage contributing to a massive sweep, exploring the W&B SDK for manual logging, searching for better hyperparameters for a line CNN transformer, and extending metrics logging using torchmetrics—turning experiment tracking into a repeatable, collaborative development loop.