Data Lake in Machine Learning

The Problem with Separate Storage Silos

Before data lakes, organizations maintained a fragmented landscape of storage systems: relational databases for structured data, HDFS for batch analytics, blob storage for unstructured files, and specialized systems for each use case. An ML engineer who needed to join user clickstream logs with product catalog data and customer support tickets would have to navigate three different systems, each with its own access patterns, security models, and query interfaces.

This fragmentation created a data gravity problem: moving data between systems was expensive, slow, and error-prone. Teams ended up maintaining redundant copies, leading to inconsistency. The cost of storage was dominated not by the bytes themselves -- cloud object storage is cheap at $0.023/GB/month (~INR 1.93/GB/month) on S3 Standard -- but by the operational overhead of managing dozens of point-to-point data pipelines.

Two Trends That Made Data Lakes Essential

Trend 1: The volume and variety of ML data exploded. Modern ML systems don't just consume clean tabular data. They ingest clickstreams, sensor readings, images, audio, video, log files, and text corpora. A single recommendation model at a company like Flipkart might need to join structured purchase history with semi-structured browsing events and unstructured product reviews. No single database format can efficiently store all of these.

Trend 2: Cloud object storage became absurdly cheap and durable. Services like Amazon S3, Google Cloud Storage, and Azure Data Lake Storage Gen2 offer 99.999999999% (11 nines) durability at fractions of a penny per gigabyte. This made it economically viable to store everything -- even data whose value isn't yet clear -- and apply schema when reading rather than writing.

From Data Swamp to Lakehouse

The first generation of data lakes (circa 2010-2016) earned a reputation as data swamps -- vast, disorganized repositories where data went to die. The problem wasn't the storage layer; it was the lack of management primitives. Without ACID transactions, concurrent writes could corrupt data. Without schema enforcement, datasets drifted into incompatible formats. Without cataloging, nobody could find anything.

The answer came from three open-source projects that emerged between 2016 and 2019: Apache Hudi (Uber, 2016), Delta Lake (Databricks, 2019), and Apache Iceberg (Netflix, 2017). Each added a metadata layer on top of object storage that provides ACID transactions, schema evolution, time travel, and efficient metadata operations. This created the lakehouse architecture -- a system that combines the low-cost, flexible storage of a data lake with the reliability, performance, and governance features of a data warehouse.

Key Takeaway: Data lakes exist because ML systems need a single, cost-effective storage layer that handles all data types, supports concurrent reads and writes with transactional guarantees, and scales to petabytes without breaking the bank. The lakehouse evolution solved the governance and reliability problems that plagued first-generation data lakes.

The Core Promise

Think of a data lake as a universal filing cabinet for your entire organization's data. Unlike a traditional database (which is more like a rigid spreadsheet with predefined columns), a data lake accepts anything -- CSVs, Parquet files, JSON blobs, images, audio recordings, model checkpoints -- and stores it all on the same infinitely expandable shelf (cloud object storage). You decide how to interpret the data when you read it, not when you write it.

The key insight is the separation of storage and compute. Your data sits in cheap, durable object storage. When you need to process it -- whether that's running a Spark job for feature engineering, a Trino query for analytics, or a PyTorch DataLoader for model training -- you spin up compute resources on demand. When the job finishes, the compute disappears but your data remains. You pay for storage 24/7 (cheap) and compute only when needed (expensive but intermittent).

What a Data Lake Does NOT Do

A common misconception: a data lake is NOT a query engine. It doesn't run SQL queries itself -- that's the job of engines like Spark SQL, Trino, Athena, or BigQuery that sit on top. The data lake provides the storage substrate, metadata management, and transactional guarantees. The query engine provides the compute.

Another trap: a data lake is NOT a substitute for a feature store. While you can compute features from data lake tables, a feature store adds point-in-time correctness, low-latency serving, and feature reuse that raw lake tables don't provide.

A Useful Mental Model: The Medallion Architecture

The most widely adopted pattern for organizing a data lake is the medallion architecture (also called multi-hop or bronze-silver-gold):

• Bronze (raw): Data lands here exactly as ingested -- no transformations, no deduplication. This is your source of truth and your insurance policy against upstream changes.
• Silver (cleaned): Deduplicated, validated, and joined. Schema is enforced. This is where most data exploration happens.
• Gold (curated): Business-level aggregates, ML-ready feature tables, and reporting datasets. This is what downstream consumers -- ML training jobs, dashboards, APIs -- actually use.

Each layer applies progressively more structure, and the open table format (Delta, Iceberg, or Hudi) ensures that every transition is atomic and auditable.

Expert Note: The single biggest determinant of data lake success is not the technology -- it's the organizational discipline to maintain clear ownership, naming conventions, and data quality standards at each medallion layer. Technology can enforce schemas; it can't enforce culture.

Formal Model of a Data Lake

A data lake can be formally described as a tuple $\mathcal{L} = (\mathcal{S}, \mathcal{M}, \mathcal{T}, \mathcal{C})$ where:

• $\mathcal{S}$ is the storage layer -- a cloud object store providing key-value access to immutable data files (Parquet, ORC, Avro, etc.)
• $\mathcal{M}$ is the metadata layer -- a transaction log and catalog that tracks table schemas, partition layouts, file manifests, and snapshot history
• $\mathcal{T}$ is the table format -- the specification (Delta Lake, Iceberg, or Hudi) defining how metadata and data files are organized to provide ACID semantics
• $\mathcal{C}$ is the catalog service -- the namespace registry (Hive Metastore, AWS Glue, Unity Catalog) that maps logical table names to physical metadata locations

ACID Guarantees

Modern data lake table formats provide full ACID transactions:

• Atomicity: Each write operation (INSERT, UPDATE, DELETE, MERGE) either fully commits or is fully rolled back. Implemented via optimistic concurrency control -- writes produce new data files, and a metadata commit atomically publishes the new snapshot.
• Consistency: Schema enforcement and check constraints prevent invalid data from being committed.
• Isolation: Concurrent readers see a consistent snapshot (snapshot isolation). Writers use conflict detection on commit.
• Durability: Both data files and transaction logs are persisted to the object store with its underlying durability guarantees.

Cost Model

The total cost of a data lake deployment can be modeled as:

$C_{total} = C_{storage} + C_{transactions} + C_{compute} + C_{egress}$

Where:

• $C_{storage} = V \times p_s \times t$ -- volume $V$ (GB) times price per GB per month $p_s$ times months $t$
• $C_{transactions} = (n_{read} \times p_r + n_{write} \times p_w)$ -- number of API calls times price per 1000 requests
• $C_{compute}$ -- cost of query engines and ETL jobs processing the data
• $C_{egress}$ -- cross-region or internet data transfer costs

For example, storing 100 TB on S3 Standard costs approximately:

$C_{storage} = 100{,}000 \text{ GB} \times \$0.023/\text{GB/month} = \$2{,}300/\text{month} \approx \text{INR } 1.93 \text{ lakh/month}$

Partition Pruning Efficiency

Partition pruning is critical for query performance. Given a table with $N$ total data files distributed across $P$ partitions, a query with a selective partition predicate reads only:

$\text{Files scanned} = \frac{N}{P} \times |P_{match}|$

where $|P_{match}|$ is the number of partitions matching the predicate. With well-chosen partition keys (e.g., date), $|P_{match}| \ll P$, reducing I/O by orders of magnitude. Apache Iceberg's hidden partitioning further improves this by automatically deriving partition values from column transforms (year, month, day, hour, bucket, truncate) without requiring users to manage partition columns explicitly.

A data lake is a centralized storage repository built on cloud object storage (S3, GCS, ADLS) that holds structured, semi-structured, and unstructured data in open formats. It serves as the foundational storage layer for ML systems, providing the raw data from which features are engineered, models are trained, and predictions are monitored.

The evolution from first-generation data lakes (prone to becoming 'data swamps') to the modern lakehouse architecture was driven by three open table formats: Delta Lake (Databricks), Apache Iceberg (Netflix), and Apache Hudi (Uber). These formats add ACID transactions, schema evolution, time travel, and efficient metadata operations to raw object storage, combining data lake flexibility with data warehouse reliability. The medallion architecture (Bronze/Silver/Gold) provides organizational structure, while data catalogs (Unity Catalog, OpenMetadata, AWS Glue) ensure discoverability and governance.

For ML teams, the data lake's value proposition is clear: 3-5x cheaper storage than data warehouses, direct integration with ML frameworks (PyArrow, Spark, PyTorch), data versioning for reproducible training, and schema-on-read flexibility for evolving feature sets. The key operational challenges -- small file management, compaction, partition optimization, and governance -- are well understood and solvable with disciplined engineering practices. Whether you're building a 1 TB data lake for a Bengaluru startup or a 150 PB lake like Uber, the architectural principles are the same: separate storage from compute, choose an open table format, organize with medallion layers, and invest in governance from day one.