Lecture 8: Data Management (Full Stack Deep Learning - Spring 2021)

Data management is where most deep learning projects quietly win or fail: getting messy, distributed inputs into a GPU-ready training pipeline—and keeping that pipeline reliable over time—often dominates engineering effort more than model design or GPU time. The lecture frames data work as both a performance lever and a complexity trap. Cleaning, moving, labeling, and versioning data can consume the majority of a machine learning team’s time, and the “data flow” between systems is frequently the hardest part.

A central message is that adding data (or augmenting existing data) is often the most cost-effective way to improve model performance. Instead of chasing new architectures or running exhaustive hyperparameter searches, teams should look for ways to expand the dataset, then use augmentation as baseline “table stakes.” Augmentation is treated as a practical engineering loop: CPU-side workers generate transformed samples (cropping, masking, pixelation, rotations for images; temporal cropping and speed changes for video; noise injection and masking for sequences) while the GPU trains in parallel on the augmented stream.

The lecture then breaks down where data comes from and why proprietary datasets usually matter. Deep learning often relies on proprietary sources—images, text, logs, and database records—because public datasets rarely provide lasting competitive advantage. Even when public data helps as a starting point, real differentiation typically comes from unique labels, domain-specific data collection, and product-driven feedback. The “data flywheel” concept highlights how deploying an early model can improve future training by turning user interactions into cleaner or more informative data.

To reduce labeling costs, the lecture emphasizes semi-supervised learning: reformulate tasks so the data supervises itself (predict future words, reconstruct sentence segments, or learn whether two sentences belong together). It cites Facebook AI Research’s SEER model as an example of training on a billion random unlabeled images to reach state-of-the-art performance on ImageNet top-1, using released loss functions and training libraries.

Next comes the storage and systems layer. The lecture distinguishes file systems (fastest, foundational), object storage like Amazon S3 (API-based, durable, versionable, good for cloud workflows), and databases for structured, repeatedly accessed metadata. It contrasts OLTP databases (fast because data is effectively held in memory with disk persistence) with OLAP-style data warehouses for analysis, and introduces ETL as the classic extract-transform-load pipeline. Data lakes and “lakehouse” approaches (e.g., Delta Lake within Databricks’ vision) aim to store raw and processed data together—structured, semi-structured, and unstructured—then transform it later for training or analytics.

Finally, the lecture treats orchestration, labeling, and versioning as the glue that makes data pipelines dependable. Workflow managers like Airflow define DAGs of dependent tasks across SQL, Python, and other steps; newer alternatives include Prefect, dbt, and Dask/Beam-style processing. Labeling is framed as a quality problem: clear guidelines, training annotators, and choosing between hiring, crowdsourcing, or specialized labeling companies. Tools such as Label Studio, plus approaches like weak supervision (Snorkel), help scale annotation while controlling quality.

The closing sections stress versioning and privacy. Without data versioning, deployed models can’t be reliably reproduced or rolled back because models are partly “code” and partly “data.” The lecture outlines a progression from snapshotting to Git LFS-based dataset manifests, then to specialized tools like DVC and Dolt. It ends with privacy concerns—especially in healthcare—pointing to federated learning, differential privacy, and encrypted-data training as active research areas. The overall takeaway: deep learning success depends on engineering data pipelines that are fast, simple enough to maintain, and rigorous enough to reproduce.