The Data Warehouse: the historical standard
The Data Warehouse emerged in the 1980s to meet a simple need: centralizing structured data from multiple operational systems and making it available for reporting and analysis. Bill Inmon laid the theoretical foundations; Ralph Kimball popularized dimensional modeling (star schema, snowflake schema).
A Data Warehouse is optimized for OLAP (Online Analytical Processing) analytical queries. It stores structured data, carefully modeled into fact and dimension tables. Its main strength: excellent query performance thanks to indexes, partitions, and columnar optimizations.
Modern cloud Data Warehouses
Modern architectures have migrated on-premise DWHs (Oracle, Teradata, IBM DB2) to the cloud. Snowflake, Google BigQuery and Amazon Redshift dominate this market. They separate storage from compute (elastic scaling), offer standard SQL, and integrate natively with BI tools (Tableau, Looker, Power BI). Snowflake was valued at $120 billion at its 2020 IPO, reflecting the market enthusiasm.
Cloud DWHs also introduced key innovations: automatic clustering (BigQuery partitioning, Snowflake micro-partitions), result caching, serverless query execution (BigQuery on-demand), and native semi-structured data support (Snowflake VARIANT column for JSON).
Kimball vs Inmon: two modeling philosophies
Bill Inmon advocated building a normalized enterprise data warehouse (3NF) first, then deriving data marts from it. This 'top-down' approach ensures consistency but requires significant upfront investment and long implementation cycles -- typically 18-24 months before the first report is delivered.
Ralph Kimball's approach is 'bottom-up': start with specific data marts (Sales, Finance, Marketing) built on dimensional models, then integrate them through a shared bus of conformed dimensions. A star schema places the fact table (measures: revenue, quantity) at the center, surrounded by dimension tables (time, product, geography). This model is simpler to query and faster to deliver.
In practice, most modern teams use a hybrid approach: Inmon-style 3NF in the integration layer ('raw' or 'staging'), then Kimball-style star schemas in the presentation layer ('marts'). dbt codifies this pattern with its staging/intermediate/mart layer convention.
The star schema (fact tables at the center, dimensions around them) remains the most common model in analytical DWHs. It optimizes joins for BI queries. The snowflake schema further normalizes dimensions (e.g. splitting a Date dimension into Year, Quarter, Month tables) but makes queries more complex and rarely improves performance in columnar engines.
The Data Lake: store everything, transform later
The Data Lake emerged around 2010 with the popularization of Hadoop and HDFS, then established itself in the cloud with Amazon S3, Azure Data Lake Storage, and Google Cloud Storage. The core idea: store all data in its native format (structured, semi-structured, unstructured) at very low cost, with no schema imposed upfront.
The Data Lake applies the 'schema on read' principle: the schema is only defined when reading and analyzing the data. This is the opposite of the Data Warehouse (schema on write). This flexibility allows ingesting application logs, JSON files, images, Kafka streams, without prior modeling.
The danger of the Data Swamp
Without governance, a Data Lake quickly becomes a Data Swamp: terabytes of undocumented files, without lineage, without verified quality, that become practically unusable. This is the major risk of poorly governed Data Lake initiatives. Teams spend more time searching for and validating data than actually analyzing it.
Common failure patterns: no data catalog (no one knows what files exist or what they mean), no quality checks at ingestion (garbage in, garbage out), no access control (sensitive data mixed with public data), and no retention policy (indefinite accumulation of stale, redundant files).
The medallion architecture: preventing the Data Swamp
The medallion architecture -- popularized by Databricks -- organizes the Data Lake into three concentric layers that enforce progressive data quality. Bronze (raw): data is ingested as-is from source systems, immutable, timestamped. No transformations, no corrections. This is the single source of truth for raw events. Typically stored in Parquet or Avro with partitioning by ingestion date.
Silver (cleansed): Bronze data is cleansed, deduplicated, and conformed to a common schema. Business rules are applied: null handling, type normalization, known duplicates removed. Data at this layer is trusted and ready for exploratory analysis. Gold (business-ready): Silver data is aggregated into business domain models -- typically Kimball-style star schemas or wide aggregate tables. This layer feeds BI tools and dashboards directly.
The medallion pattern maps naturally to dbt layers: Bronze = sources/seeds, Silver = staging + intermediate models, Gold = mart models. This alignment between the storage architecture and the transformation framework simplifies team organization and quality responsibility.
A view widely echoed across the data industry in the late 2010s was that a very large share of Big Data projects failed to create measurable business value. One of the main causes: Data Lakes turning into Data Swamps due to a lack of governance, cataloging, and quality control. The medallion architecture and a data catalog (Apache Atlas, DataHub, Alation) are non-negotiable investments.
The Lakehouse: DWH performance on Lake flexibility
The Lakehouse emerged around 2020 to resolve the limitations of both previous architectures. The concept, formalized by Databricks, combines the flexibility and low cost of the Data Lake with the performance, reliability, and analytical capabilities of the Data Warehouse.
Technically, the Lakehouse relies on open table formats (Delta Lake, Apache Iceberg, Apache Hudi) that add ACID layers, versioning, schema enforcement, and query optimization directly on object storage (S3, ADLS, GCS). You write Parquet or ORC files to an S3 bucket and query them with near-DWH performance thanks to file statistics and embedded indexes.
File formats: Parquet, ORC and Avro
Before choosing a table format (Delta Lake, Iceberg, Hudi), you choose a physical file format. Parquet is the dominant columnar format: it stores data column by column (not row by row), enabling query engines to read only the columns needed. This reduces I/O by 60-90% for analytical queries that touch 3-5 columns out of 50. Parquet also achieves excellent compression ratios (SNAPPY, ZSTD) because similar values in a column compress much better than mixed row data.
ORC (Optimized Row Columnar) is Parquet's historical competitor, more common in Hive and Presto/Trino ecosystems. It offers slightly better compression for certain workloads and has built-in ACID support for Apache Hive. Avro is a row-based format -- ideal for streaming (Kafka messages, Debezium CDC events) because it handles schema evolution elegantly via its JSON schema registry. In practice: Parquet for analytics and DWH layers, Avro for event ingestion (Bronze layer), ORC for Hive-centric architectures.
Delta Lake, Apache Iceberg and Apache Hudi
Delta Lake (created by Databricks, open-source since 2019) is the most widely adopted table format in Spark architectures. It maintains a transaction log (_delta_log/) that records every write operation (insert, update, delete, schema change). This log enables ACID transactions, time travel (querying past states), schema enforcement, and automatic compaction (OPTIMIZE command, Z-ORDER clustering).
Apache Iceberg, initiated by Netflix, is preferred in AWS ecosystems (Athena, EMR, Glue) and is gaining ground in Snowflake (Iceberg Tables) and BigQuery (BigLake). Iceberg's key advantage: partition evolution without rewriting data, hidden partitioning (the query engine chooses partitions), and row-level deletes without rewriting entire Parquet files. Apache Hudi, created by Uber, excels in frequent upsert scenarios (CDC - Change Data Capture) with its MERGE ON READ mode that uses lightweight delta files for updates.
Lakehouse tools: Databricks, Amazon Lake Formation, Azure Synapse
Databricks Data Intelligence Platform is the market reference. It combines Spark, Delta Lake, an ML layer (MLflow), and a high-performance SQL engine (Photon). On AWS, Lake Formation with S3 + Glue + Athena offers a serverless alternative. Azure Synapse Analytics integrates the Lakehouse into the Microsoft ecosystem. Google BigLake enables querying S3 or ADLS data from BigQuery.
The streaming Lakehouse is an emerging pattern: Kafka or Kinesis streams are written directly to Delta Lake or Iceberg tables via Spark Structured Streaming or Apache Flink. This enables near-real-time analytics (seconds to minutes latency) without a separate streaming storage layer, reducing architectural complexity significantly compared to Lambda architecture.
The term 'Lakehouse' was formalized in the paper 'Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics' published by Databricks in 2021 (VLDB 2021). The paper demonstrates that Delta Lake on S3 achieves 90% of the TPC-DS performance of a dedicated Data Warehouse at 1/3 the storage cost.
Armbrust et al., VLDB 2021Data Lake vs Data Warehouse vs Lakehouse: key dimensions
Here are the five dimensions on which these three architectures differ fundamentally.
Schema: on write vs on read
The Data Warehouse enforces a strict schema at write time (schema on write): data must conform to the model before loading. The Data Lake defines the schema only at read time (schema on read). The Lakehouse enables both: optional schema enforcement for critical tables (Delta Lake CONSTRAINT clause, Iceberg schema enforcement), schema on read for exploration zones.
Schema evolution is a practical advantage of open table formats: adding a column to a Delta Lake table is instant (ALTER TABLE ADD COLUMN) and backward compatible -- old Parquet files simply return NULL for the new column. Renaming or dropping a column requires updating the transaction log but never touches existing files.
ACID guarantees: where they come from
Traditional Data Warehouses have always supported ACID transactions natively (PostgreSQL-style locking). Pure Data Lakes on S3 or HDFS have none: two concurrent writes can corrupt a partition, and a failed job leaves partial data. This is why ACID in a Data Lake requires a table format (Delta Lake, Iceberg, Hudi) -- without it, you have no atomicity, no consistency guarantees.
Delta Lake achieves atomicity via the transaction log: a write either commits fully (a new JSON entry in _delta_log/) or is rolled back by deleting uncommitted Parquet files. Isolation is ensured via optimistic concurrency control (OCC): each writer reads the current log version and conflicts are detected at commit time. This is why Delta Lake operations scale well under concurrent writes from multiple Spark jobs.
Supported data types
The DWH is limited to structured data (SQL tables). The Data Lake accepts everything: structured, semi-structured (JSON, Parquet, Avro), unstructured (images, videos, logs). The Lakehouse adds native ML/AI workflow support on raw data, enabling models to be fed directly from the Lake without an intermediate export step.
Cost structure
Storage in a cloud DWH (Snowflake, BigQuery) costs approximately 20 to 40 times more than object storage (S3, ADLS). Data Lakes and Lakehouses use object storage, hence very low storage costs. In return, ad hoc queries on a well-optimized DWH remain faster and less compute-intensive.
The 'storage separation' trend has blurred this distinction: Snowflake's Iceberg Tables and BigQuery's BigLake allow storing data on S3 while querying it with the DWH engine. You pay object storage rates for data and cloud DWH rates only for compute -- the best of both worlds economically.
A naive S3 Data Lake has no ACID guarantees. If a Spark job fails mid-write, you get partial Parquet files in the partition. If two jobs write to the same partition simultaneously, you get file corruption. This is why Delta Lake, Iceberg, and Hudi exist: they add the transaction log layer that makes object storage behave like a database.
Lambda and Kappa: combining batch and streaming
Analytics use cases increasingly require low-latency data alongside historical batch data. Two architectural patterns address this: Lambda and Kappa. Understanding their trade-offs is essential for designing modern data platforms.
Lambda architecture: two parallel layers
Proposed by Nathan Marz (2011), Lambda architecture maintains two parallel processing layers. The batch layer (Spark, Hadoop) processes all historical data periodically (hourly, daily) and produces accurate, comprehensive views. The speed layer (Kafka Streams, Flink, Spark Streaming) processes real-time events as they arrive and produces approximate near-real-time views to compensate for the batch layer's latency.
A serving layer (Cassandra, Druid, Elasticsearch) merges batch views and speed views to answer queries. The problem: maintaining two codebases that implement the same business logic in different frameworks is expensive. Lambda creates operational complexity and version drift between the batch and streaming implementations -- a bug fixed in one layer must be fixed in the other.
Kappa architecture: streaming as the single source
Jay Kreps (Kafka co-creator) proposed Kappa architecture in 2014: replace the batch layer entirely with a streaming layer. Since Kafka retains messages for configurable periods (days, weeks, months), historical reprocessing is just a matter of replaying from offset 0 with an updated consumer. One codebase, one runtime, no synchronization between batch and streaming.
Kappa works well when: event retention in Kafka covers the required historical window, the streaming framework (Flink, Spark Structured Streaming) handles your throughput at acceptable cost, and your business logic is naturally expressible as streaming transformations. The streaming Lakehouse (Kafka + Delta Lake + Spark) is a modern Kappa implementation where Delta Lake provides the durable storage and time travel that Kafka alone cannot offer for very long retention windows.
The streaming Lakehouse: the convergence
The streaming Lakehouse is the pragmatic synthesis for 2026: real-time events from Kafka or Kinesis are written directly to Delta Lake or Iceberg tables via Spark Structured Streaming or Apache Flink. The medallion architecture still applies -- Bronze receives raw events in micro-batches (seconds to minutes), Silver applies streaming transformations (deduplication, enrichment), Gold produces aggregate tables refreshed in near-real-time.
This pattern eliminates the Lambda duality: the same Delta Lake tables serve both streaming analytics (queried by Databricks SQL or Trino) and batch ML training (Spark or PyTorch DataLoader reading Parquet files). The result is a single platform that handles historical analysis, real-time dashboards, and ML training without maintaining separate storage systems.
Lambda architecture is increasingly considered legacy. The operational cost of maintaining two parallel codebases rarely justifies the marginal latency advantage over a well-tuned streaming Lakehouse. Kappa + Delta Lake is the recommended default for new architectures combining batch and streaming requirements.
How to choose your storage architecture
The choice is not made in theory but based on your concrete needs: volume, data types, usage patterns (BI, ML, streaming), team skills, and budget.
Choosing a pure Data Warehouse
Recommended if: your needs are 100% BI/reporting on structured data, your team is SQL-first without Spark skills, you have strict SLAs on query times (sub-second dashboards), and your data volume fits comfortably within managed DWH storage economics (typically under 10 TB). Snowflake and BigQuery are excellent for this profile. BigQuery's serverless model (pay per query) is particularly suited to variable workloads.
Choosing a pure Data Lake
Recommended if: you store massive volumes of unstructured data (logs, images, audio, video) that don't fit in a SQL schema, your primary use cases are Machine Learning on raw data (training large models from S3-stored files), and you have a platform team capable of managing governance, cataloging, and access control. Without that team, default to the Lakehouse pattern instead.
Choosing the Lakehouse
The Lakehouse is the default choice for new architectures in 2026. Recommended if you want to combine BI and ML on the same platform, benefit from low-cost object storage with DWH-like performance, and preserve raw data with robust governance via open formats. Start with S3 + Delta Lake + dbt + Databricks SQL or Athena -- this stack covers 90% of use cases at manageable cost and operational complexity.
The Lakehouse also future-proofs your architecture: as AI workloads grow, having raw data in Parquet on object storage means your ML engineers can access training data without an ETL export step. This is increasingly the deciding factor for data teams building both analytical and AI capabilities.
For teams under 5 data engineers or datasets under 1 TB, a pure cloud DWH (BigQuery, Snowflake) is almost always the right choice. Lakehouse architecture introduces operational complexity (table format management, compaction jobs, Spark cluster sizing) that outweighs its benefits at small scale. Start simple and migrate when you hit real scaling constraints.
Anchoring these concepts with spaced repetition
The distinction between Data Lake, Data Warehouse, and Lakehouse is a frequent topic in Data Engineering and Data Architecture interviews. The concepts are clear on the surface but quickly intertwine in practice. Memia's data architecture flashcards let you anchor these distinctions in long-term memory.
Key concepts worth drilling: ACID guarantees and why they require Delta Lake on S3, time travel syntax in Delta Lake, the medallion architecture layers (Bronze/Silver/Gold) and what quality level each represents, Kimball vs Inmon trade-offs, and when to choose Parquet vs Avro. Spaced repetition ensures these distinctions remain sharp over months, not just the week before an interview.
The most useful comparisons to master: schema on write vs on read, ACID in a Data Lake (impossible without Delta Lake/Iceberg/Hudi), Time Travel in Delta Lake (VERSION AS OF / TIMESTAMP AS OF), medallion architecture layers, difference between partitioning and clustering in BigQuery, Parquet vs Avro use cases.
Explore the Data & AI cluster
Frequently asked questions about Data Lake, Data Warehouse and Lakehouse
What is the difference between a Data Lake and a Data Warehouse?
The Data Warehouse stores only structured data with a schema defined upfront (schema on write), optimized for SQL analytical queries. The Data Lake stores any type of data (structured, semi-structured, unstructured) in its native format without prior schema (schema on read), at very low cost on object storage (S3, ADLS, GCS).
What is a Lakehouse?
A Lakehouse combines the advantages of a Data Lake (cheap object storage, flexibility, all data types) and a Data Warehouse (ACID transactions, SQL performance, schema enforcement). It relies on open table formats like Delta Lake, Apache Iceberg, or Apache Hudi that add transaction management on top of Parquet files stored in object storage.
What is a Data Swamp?
A Data Swamp is a Data Lake that has become unusable due to lack of governance: undocumented data, without lineage, without verified quality, without cataloging. Teams no longer know what it contains or how to use it. It is the main risk of a Data Lake without a metadata management strategy. The medallion architecture (Bronze/Silver/Gold) is the primary defense against it.
What is the medallion architecture?
The medallion architecture organizes a Data Lake or Lakehouse into three quality layers. Bronze: raw data ingested as-is from sources, immutable. Silver: cleansed, deduplicated, conformed data with business rules applied. Gold: aggregated, business-ready data that feeds BI tools. Each layer represents a higher level of data trust and refinement. It maps directly to dbt's staging/intermediate/mart layer convention.
Delta Lake or Apache Iceberg: which to choose?
Delta Lake is preferred in Databricks and Spark ecosystems. Apache Iceberg is more widely adopted in AWS ecosystems (Athena, EMR, Glue) and benefits from growing support from Snowflake and BigQuery. Both offer ACID, time travel and schema evolution -- the choice depends primarily on your cloud provider and existing tools. Hudi is a third option optimized for frequent upserts (CDC use cases).
Is Snowflake a Data Warehouse or a Lakehouse?
Snowflake started as a pure cloud Data Warehouse. It is evolving toward the Lakehouse with Iceberg Tables (S3 storage + Snowflake compute), but remains primarily positioned as a high-performance analytical DWH. BigQuery is in a similar position with BigLake enabling queries over S3 or ADLS data.
What is time travel in Delta Lake?
Time travel is the ability to query the past state of a Delta Lake table. For example: SELECT * FROM my_table VERSION AS OF 10 or SELECT * FROM my_table TIMESTAMP AS OF '2026-01-01'. This is made possible by the transaction log that Delta Lake maintains for each write. It enables auditing, debugging (comparing before/after a bad transformation), and data recovery without backups.
What is the difference between Lambda and Kappa architecture?
Lambda maintains two parallel layers: a batch layer (Spark, Hadoop) for accurate historical views and a speed layer (Kafka Streams, Flink) for real-time approximate views. Kappa eliminates the batch layer: all processing runs through a single streaming pipeline, with historical reprocessing done by replaying Kafka from offset 0. Kappa reduces operational complexity but requires Kafka retention to cover the required historical window.
Why use Parquet and not CSV or JSON?
CSV and JSON are row-based formats: analytical queries that read 3 columns out of 50 must still scan every column. Parquet is columnar: only the needed columns are read, reducing I/O by 60-90%. Parquet also compresses much better (similar values in a column compress efficiently) and stores column statistics (min/max) that allow query engines to skip irrelevant row groups entirely. For analytical workloads, Parquet is 10-100x more efficient than CSV.
What is the difference between Kimball and Inmon modeling?
Inmon's approach is 'top-down': build a normalized 3NF enterprise DWH first, then derive department-specific data marts. Ensures consistency but requires 18-24 months before first delivery. Kimball's approach is 'bottom-up': build dimensional star schema marts department by department, then integrate via shared conformed dimensions. Faster to deliver value. Modern teams often use a hybrid: Inmon-style normalization in the staging layer, Kimball-style star schemas in the presentation layer.
Next article: Data Governance - definition and implementation