Versioning (5) - Data Management - Full Stack Deep Learning

Versioning in machine learning isn’t just about saving model code—it’s about making the trained artifact reproducible by tracking the exact data used to train it. Without data versioning, teams eventually lose the ability to recreate a previously working model: data changes after deployment, but the system only remembers the code hash, not the dataset state that produced the model. The result is a slow drift into irreproducible performance, where retraining may no longer recover the earlier “known-good” model because the original training inputs can’t be reconstructed.

A practical way to think about versioning is in four layers. At the lowest level is “unversioned” data—keeping datasets only in a file system, S3, or a database and training on whatever happens to be there later. The next level adds a snapshot: store a hash or snapshot of the training data alongside the code version so the deployed artifact can be tied to a specific dataset state. But doing this as a crude “snapshot everything” approach can become hacky, especially if it doesn’t capture the full code bundle or if it’s too heavy to manage.

Level two is more systematic: version data as a combination of assets and code. For example, speech datasets might store WAV files in S3 under unique IDs, while labels live in a JSON mapping those IDs to annotations. Those label files can be tracked in version control, but they may balloon into multi-gigabyte blobs when there are millions of labeled rows. Tools like Git Large File Storage (Git LFS) address this by keeping large files out of the Git repository: commits store hashes and pointers while the actual content is uploaded to S3. An additional optimization, lazy data loading, can prevent downloading entire large repositories until the data is actually needed.

In this setup, the dataset version becomes the hash of the underlying raw data and labels—so any change to labels, added rows, or modified waveform files produces a new dataset hash and therefore a new dataset “version.” Adding a timestamp can make it easier to correlate versions with training runs.

Level three points to specialized machine-learning data versioning systems. The transcript flags a key caution: these tools should be adopted only after understanding the specific problem they solve, because some quickly expand into managing infrastructure and pipelines. Examples mentioned include DVC, which version-controls both data and transformation steps via pipelines (useful in tabular/CSV-style workflows), and Pachyderm, which emphasizes language-agnostic data versioning plus automated triggering, parallelism, and resource management. Another option highlighted is Dolt from Liquid Data, which focuses on database versioning: it provides Git-like branching and merging for databases, supports conflict detection and resolution, and enables comparisons of performance across different dataset versions by running models against each version.

The core takeaway is straightforward: reliable deployment requires treating training data as a first-class, versioned dependency—ideally with tooling that can reproduce both the dataset state and the transformations that produced the training inputs.