Versioning Data for Machine Learning

Data versioning is treated as a ladder of increasing sophistication—because deployed machine learning models depend on both code and the exact data they were trained on, and teams eventually need to roll back when things break. At the simplest level, data sits on a file system or in S3 without versioning; training on Tuesday uses whatever data exists then, and training on Friday uses whatever exists then. The trouble shows up during deployment: if the deployed model fails and the team needs to revert, the lack of data history makes it hard to reproduce the prior training set, even though the model itself is versioned.

A more reliable approach is to snapshot the dataset and store it alongside the code version. In practice, that means packaging the full training data at a given time, storing it somewhere durable, and recording a pointer to that snapshot when training and deploying. With this method, a deployment can reference both the Git commit for the code and a specific data snapshot date (or identifier), enabling a rollback to a known-good combination. The workflow still has friction—manual snapshotting can feel “super hacking”—which motivates a better pattern: version data as a mix of assets and code.

Level 2 centers on keeping large raw assets in object storage while versioning the *references* and metadata in Git. For image processing or speech recognition, the actual audio files or images can live in S3 under unique IDs. The training dataset then becomes a structured list of those IDs plus labels and other metadata stored in JSON or similar formats that Git can track. As long as the underlying S3 objects aren’t deleted, checking out an earlier Git revision effectively reconstructs the dataset “as of” that point in time. Because JSON manifests can grow huge, the transcript flags Git LFS (Git Large File Storage) as a way to handle large files similarly to code.

This scheme raises a future-proofing question: what happens when systems evolve—database schemas change, or data gets deleted? The answer depends on application constraints. In education-focused use cases with strict privacy rules, deletion requests must be honored, so retaining old data for future training isn’t allowed; in those environments, the “right place” to enforce correctness is upstream via contracts, tests, and alerts, or by using S3 versioning so files aren’t removed—only new versions are added. That shifts the burden from downstream rollback to earlier safeguards.

Level 3 moves beyond Git-based reference tracking into specialized data versioning systems. The transcript names DBC, Pachyderm, and Quill, with DBC highlighted as especially promising. DBC automates the linkage between data additions and stored artifacts in S3, and it records transformations (for example, converting an XML dataset into a training-ready CSV) so the pipeline and provenance can be retrieved later. The takeaway is pragmatic: try Level 2 first, then evaluate Level 3 if the operational overhead or provenance needs justify it.