Storage (4) - Data Management - Full Stack Deep Learning

Storage choices determine how data moves, how fast it can be read, and how safely it can be reused across training and production. The core takeaway is a practical division of labor: file systems handle foundational “files,” object storage wraps files as addressable objects with API-style operations and built-in versioning/redundancy, databases store structured records for fast querying, and data lakes aggregate semi-structured or log-like data for later transformation.

At the base layer, file systems treat the fundamental unit as a file—text or binary—typically not versioned at the file-system level and easy to overwrite or delete. They work well when access patterns are simple, such as reading a whole dataset from a single disk (which can be fast but limits parallelism because the data must pass through one physical location). Network file systems extend this by letting multiple machines share the same file namespace, and distributed file systems (including Hadoop-style setups) allow many machines to access files without needing to know where the bytes physically live.

Object storage changes the performance and management model by presenting an API over storage rather than a traditional filesystem. Instead of thinking in terms of “where the file is,” the system treats the fundamental unit as an object—often binary, but sometimes text—and can add features like versioning (new writes create additional versions rather than overwriting) and redundancy (replicating across multiple disks). That redundancy enables parallel reads: many “get” requests can be served from different disks at once. The tradeoff is that the extra abstraction layer can make object storage slower than raw local disk access, and it must handle concurrent readers. Amazon S3 is the canonical example, with similar patterns available via other cloud providers.

Databases sit on top of this with a different mental model: data is treated as if it lives in RAM for speed, while the database engine ensures durability by persisting changes to disk so nothing is lost on shutdown or power failure. The fundamental unit is a row with a unique identifier, columns for values, and references to other rows (including across tables). Databases are meant for structured, query-heavy data—not binary blobs. A common pattern is to store images or other binaries in object storage and keep metadata in the database: labels, dimensions, uploader identity, and the object path/ID. Logs are treated differently: they’re often stored for later investigation or metric computation, not as the primary operational dataset.

For the database layer, PostgreSQL is presented as a pragmatic default because it supports both SQL and JSON documents, letting teams handle “schema on read” needs without abandoning structured querying. SQL is framed as the right interface for structured data, and avoiding it risks reinventing query logic poorly.

Data lakes aggregate data from multiple sources—often logs and event streams—using “schema on read.” Raw data lands in the lake, then downstream processes impose structure later, transforming it into databases, packaged training files, or other artifacts. When training time arrives, minimizing distance between data and GPUs matters: the needed subset is copied to local or network storage near the compute so training doesn’t stall on slow transfers.

Finally, “what goes where” is summarized as: binaries to object storage (for versioning and parallel access), metadata/labels to databases, and raw event/log data to data lakes, with feature extraction or aggregation steps producing the training-ready datasets. For deeper study, the discussion points to Designing Data-Intensive Applications as a first-principles guide to databases, logs, and related tradeoffs.