Data Lakehouse Architecture: The Best of Data Lakes and Warehouses

In This Guide

• 1. The Problem: Data Lakes vs Data Warehouses
• 2. What Is a Data Lakehouse?
• 3. Open Table Formats — Delta Lake, Iceberg, Hudi
• 4. Lakehouse Architecture in Practice
• 5. Query Engines — How You Actually Query a Lakehouse
• 6. When to Use (and Not Use) a Lakehouse
• 7. Frequently Asked Questions

Traditional Data Lake:
    S3 bucket → Parquet files (no guarantees, partial writes, schema drift)

Lakehouse:
    S3 bucket → Parquet files + Transaction Log (Delta Lake / Iceberg / Hudi)
                                    │
                                    ├── ACID commits (atomic writes)
                                    ├── Schema enforcement & evolution
                                    ├── Time travel (query past snapshots)
                                    ├── Partition & file-level statistics
                                    └── UPDATE, DELETE, MERGE operations

Same storage. Same files. Just smarter metadata.

Delta Lake — PySpark Example

# Write data as a Delta table (ACID-guaranteed)
df = spark.read.json("s3://raw-data/orders/2026-02-06/")

df.write \
    .format("delta") \
    .mode("append") \
    .partitionBy("order_date") \
    .save("s3://lakehouse/orders/")

# MERGE (upsert) — update existing, insert new
from delta.tables import DeltaTable

target = DeltaTable.forPath(spark, "s3://lakehouse/orders/")
source = spark.read.json("s3://raw-data/orders/latest/")

target.alias("t") \
    .merge(source.alias("s"), "t.order_id = s.order_id") \
    .whenMatchedUpdate(set={"status": "s.status", "updated_at": "s.updated_at"}) \
    .whenNotMatchedInsertAll() \
    .execute()

# Time travel — query yesterday's version
df_yesterday = spark.read \
    .format("delta") \
    .option("timestampAsOf", "2026-02-05") \
    .load("s3://lakehouse/orders/")

# Schema enforcement — this fails if schema doesn't match
df_bad = spark.createDataFrame([{"wrong_column": "value"}])
df_bad.write.format("delta").mode("append").save("s3://lakehouse/orders/")
# AnalysisException: schema mismatch detected

Apache Iceberg — SQL Example

-- Create an Iceberg table (works with Spark, Trino, Flink, Dremio)
CREATE TABLE lakehouse.orders (
    order_id    STRING,
    customer_id STRING,
    amount      DECIMAL(10,2),
    status      STRING,
    order_date  DATE
) USING iceberg
PARTITIONED BY (days(order_date));  -- Hidden partitioning!

-- Users don't need to know about partitions
SELECT * FROM lakehouse.orders WHERE order_date = '2026-02-06';
-- Iceberg automatically prunes to the right partition

-- Partition evolution — change partitioning without rewriting data
ALTER TABLE lakehouse.orders ADD PARTITION FIELD months(order_date);

-- Time travel
SELECT * FROM lakehouse.orders VERSION AS OF 12345;  -- snapshot ID
SELECT * FROM lakehouse.orders TIMESTAMP AS OF '2026-02-05T10:00:00';

-- Expire old snapshots (cleanup)
CALL lakehouse.system.expire_snapshots('orders', TIMESTAMP '2026-01-01');

Medallion Architecture (Bronze → Silver → Gold)

Data Sources (APIs, DBs, Events)
    │
    ▼
┌─────────────────────────────────────────────────────┐
│  BRONZE (Raw)                                        │
│  s3://lakehouse/bronze/orders/                       │
│  • Raw data as-is from source                        │
│  • Append-only, no transformations                   │
│  • Schema-on-read, keep everything                   │
│  • Retention: forever (cheap storage)                │
└─────────────────────┬───────────────────────────────┘
                      │  dbt / Spark ETL
                      ▼
┌─────────────────────────────────────────────────────┐
│  SILVER (Cleaned)                                    │
│  s3://lakehouse/silver/orders/                       │
│  • Deduplicated, validated, typed                    │
│  • Schema enforced, nulls handled                    │
│  • Slowly changing dimensions applied                │
│  • Retention: 2-5 years                              │
└─────────────────────┬───────────────────────────────┘
                      │  dbt models / aggregations
                      ▼
┌─────────────────────────────────────────────────────┐
│  GOLD (Business-Ready)                               │
│  s3://lakehouse/gold/monthly_revenue/                │
│  • Aggregated, business-logic applied                │
│  • Ready for BI dashboards and reports               │
│  • Optimized for query performance (Z-ordered)       │
│  • Retention: as needed                              │
└─────────────────────────────────────────────────────┘

Related Articles