Data Lake vs Data Warehouse — Architecture Comparison

In this tutorial, you'll learn about Data Lake vs Data Warehouse. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

A data lake stores raw data in native format using object storage and schema-on-read, while a data warehouse stores structured, processed data optimized for analytical queries using schema-on-write — each serving different use cases and user personas.

What You'll Learn

This tutorial covers the complete comparison: data lake vs warehouse fundamentals, schema-on-read vs schema-on-write, the lakehouse architecture with Delta Lake and Apache Iceberg, when to choose each, medallion architecture, and how to build hybrid data platforms on AWS and Azure.

Why It Matters

Choosing the wrong storage architecture wastes money and slows down analytics. Raw data in a warehouse costs too much and breaks schemas. Structured data in a lake frustrates analysts expecting fast SQL. Understanding the trade-offs helps you design cost-effective, performant data platforms. DodaTech's Durga Antivirus Pro uses a data lake for raw threat samples and a warehouse for structured detection analytics.

Real-World Use

Netflix stores raw streaming logs (petabytes/day) in S3 Data Lakes for ML model training, then processes into a warehouse for business KPIs. Uber's data lake on HDFS stores 100PB+ of raw trip data; their warehouse on Presto/Spark serves analytics. Capital One runs a lakehouse on AWS with Delta Lake for both Data Science and regulatory reporting.

flowchart TD
  Q[What are you doing with the data?]
  Q --> A[Analysis / Reporting]
  Q --> B[ML / Exploration]
  Q --> C[Real-time / Streaming]
  A --> W[Data Warehouse]
  B --> L[Data Lake]
  C --> L
  subgraph Hybrid
    H[Lakehouse: Both]
  end

Data Lake vs Data Warehouse

Think of a data lake like a giant warehouse where you store every box as-is. You don't sort, label, or organize boxes when they arrive. When someone needs something, they search and extract at that moment.

A data warehouse is like a library — everything is cataloged, organized on shelves, and easy to find. But preparing items for the library takes time and you can't add things that don't fit.

Schema-on-Read vs Schema-on-Write

Schema-on-Write (Data Warehouse)

You define the schema before loading data. If a column is added in the source, you must alter the table before loading.

# Schema-on-write: Table must exist before loading
# CREATE TABLE sales (id INT, amount DECIMAL, date DATE);
# INSERT INTO sales VALUES (1, 100.50, '2026-06-01');
# A new column 'region' would require: ALTER TABLE sales ADD COLUMN region TEXT;

Schema-on-Read (Data Lake)

You store raw data as-is (JSON, Parquet, CSV). The schema is applied when you query it.

# Schema-on-read: Store raw, interpret at query time
# Spark query:
# df = spark.read.json("s3://datalake/events/2026/06/01/")
# df.createOrReplaceTempView("events")
# spark.sql("SELECT event_type, COUNT(*) FROM events GROUP BY event_type").show()
# Schema is inferred from the data itself at read time.

Simulating a Data Lake

# datalake.py — Store raw data with schema-on-read
import json, os
from datetime import datetime

class DataLake:
    def __init__(self, base_path="datalake"):
        self.base_path = base_path
        os.makedirs(f"{base_path}/raw", exist_ok=True)

    def ingest_raw(self, source, data):
        partition = datetime.now().strftime("%Y/%m/%d")
        path = f"{self.base_path}/raw/{source}/{partition}"
        os.makedirs(path, exist_ok=True)
        filename = f"{datetime.now().strftime('%H%M%S')}.json"
        with open(f"{path}/{filename}", 'w') as f:
            json.dump(data, f)
        print(f"[INGEST] Raw data -> {path}/{filename}")
        return f"{path}/{filename}"

    def read_raw(self, path):
        with open(path) as f:
            raw = json.load(f)
        schema = {"event": str, "user_id": str, "value": float}
        validated = {}
        for field, expected_type in schema.items():
            value = raw.get(field)
            try:
                validated[field] = expected_type(value) if value is not None else None
            except (ValueError, TypeError):
                validated[field] = None
        print(f"[READ] Schema-on-read: {validated}")
        return validated

lake = DataLake()
path = lake.ingest_raw("clickstream", {
    "event": "page_view", "user_id": "user_12345", "value": 1.0,
    "browser": "Chrome", "ip": "192.168.1.1",
})
lake.read_raw(path)

print("\n=== Exploring Raw Data Lake ===")
for root, dirs, files in os.walk(lake.base_path):
    level = root.replace(lake.base_path, '').count(os.sep)
    print(f"{'  ' * level}{os.path.basename(root)}/")
    for f in files[:3]:
        size = os.path.getsize(os.path.join(root, f))
        print(f"{'  ' * (level + 1)}{f} ({size} bytes)")

Expected output:

[INGEST] Raw data -> datalake/raw/clickstream/2026/06/21/100000.json
[READ] Schema-on-read: {'event': 'page_view', 'user_id': 'user_12345', 'value': 1.0}

=== Exploring Raw Data Lake ===
datalake/
  raw/
    clickstream/
      2026/
        06/
          21/
            100000.json (194 bytes)

Lakehouse Architecture

The lakehouse combines the flexibility of Data Lakes with the reliability and performance of data warehouses — ACID transactions, schema enforcement, and performance optimizations on object storage.

Key Technologies

-- Delta Lake: time travel to query previous versions
SELECT * FROM events TIMESTAMP AS OF '2026-01-15'
WHERE event_type = 'purchase';

-- Iceberg: partition evolution without rewriting
ALTER TABLE orders ADD PARTITION FIELD month(order_date);

Medallion Architecture

flowchart TB
  RAW[Bronze
Raw Data] --> SILVER[Silver
Cleaned & Validated]
  SILVER --> GOLD[Gold
Aggregated & Curated]
  GOLD --> BI[BI Dashboards]
  GOLD --> ML[ML Pipelines]
  GOLD --> ADHOC[Ad-hoc Analytics]
  style RAW fill:#ff6b6b,color:#fff
  style SILVER fill:#feca57,color:#333
  style GOLD fill:#48dbfb,color:#333

# medallion.py — Bronze → Silver → Gold transformation
import json, hashlib

raw_events = [
    {"event": "login", "user": "alice", "ts": "2026-06-15T08:00:00"},
    {"event": "purchase", "user": "alice", "ts": "2026-06-15T08:30:00", "amount": 49.99},
    {"event": "login", "user": "bob", "ts": "2026-06-15T09:00:00"},
    {"event": "error", "user": None, "ts": "2026-06-15T09:15:00"},
    {"event": "purchase", "user": "bob", "ts": "2026-06-15T09:30:00", "amount": 199.99},
]

def bronze_zone(events):
    bronze = []
    for e in events:
        bronze.append({
            **e, "_ingested_at": "2026-06-15T10:00:00",
            "_row_hash": hashlib.md5(json.dumps(e, sort_keys=True).encode()).hexdigest()[:8],
        })
    return bronze

def silver_zone(bronze):
    seen = set()
    silver = []
    for row in bronze:
        if row["_row_hash"] in seen or row["user"] is None:
            continue
        seen.add(row["_row_hash"])
        silver.append(row)
    return silver

def gold_zone(silver):
    user_metrics = {}
    for row in silver:
        user = row["user"]
        user_metrics.setdefault(user, {"logins": 0, "purchases": 0, "total_spent": 0.0})
        if row["event"] == "login":
            user_metrics[user]["logins"] += 1
        elif row["event"] == "purchase":
            user_metrics[user]["purchases"] += 1
            user_metrics[user]["total_spent"] += row["amount"]
    return user_metrics

b = bronze_zone(raw_events)
s = silver_zone(b)
g = gold_zone(s)

print(f"Bronze: {len(b)} events, Silver: {len(s)} events")
for user, metrics in g.items():
    print(f"  {user}: {metrics['logins']} logins, {metrics['purchases']} purchases, ${metrics['total_spent']:.2f}")

Expected output:

Bronze: 5 events, Silver: 4 events
  alice: 1 logins, 1 purchases, $49.99
  bob: 1 logins, 1 purchases, $199.99

When to Use Each

Common Mistakes

1. Creating a "Data Swamp"

A data lake with no organization, no metadata, and no governance becomes a data swamp. Implement Partitioning, cataloging, and naming conventions from day one.

2. Not Managing Permissions

Raw data may contain PII, financial details, or credentials. Apply access controls at the storage level (IAM policies, bucket policies).

3. Ignoring Small Files Problem

Storing millions of tiny CSV files kills query performance. Coalesce small files into larger Parquet files (100MB-1GB) using Spark.

4. Writing Without Schema Validation

With no schema-on-write, bad data can silently enter the lake. Use Great Expectations to catch corrupted records.

5. No Data Retention Policies

Raw data accumulates fast. Without lifecycle policies, storage costs explode. Move cold data to cheaper tiers.

6. Using Only Raw Zone

A lake with only raw data is hard to use. Implement a medallion architecture: Bronze (raw), Silver (cleaned), Gold (aggregated).

Practice Questions

1. What is schema-on-read and how is it different from schema-on-write? Schema-on-read applies structure when data is queried, not when it's stored. Schema-on-write defines structure before loading. Data Lakes use schema-on-read; warehouses use schema-on-write.

2. What is a data lakehouse? A lakehouse combines data lake flexibility with warehouse reliability by adding ACID transactions, schema enforcement, and performance optimizations on top of object storage.

3. When would you choose a data lake over a data warehouse? When data types are diverse (text, images, JSON), schemas are unknown/unstable, storage cost is a priority, or data scientists need raw data for ML exploration.

4. What is the medallion architecture? A layered approach: Bronze (raw ingested data), Silver (cleaned/deduplicated), Gold (aggregated, business-ready). Each layer increases quality and reduces volume.

5. Challenge: Design a data Strategy for a healthcare company collecting patient vitals from IoT sensors, lab results as PDFs, appointment logs, and insurance claims.

Mini Project: Hybrid Lakehouse Simulator

class LakehouseSimulator:
    def __init__(self):
        self.bronze = []
        self.silver = []
        self.gold = {}

    def ingest(self, records):
        for r in records:
            self.bronze.append({**r, "ingested": True})
        print(f"Bronze: {len(self.bronze)} records")

    def clean(self):
        self.silver = [r for r in self.bronze if r.get("user") and r.get("amount", 1) > 0]
        print(f"Silver: {len(self.silver)} records (cleaned)")

    def aggregate(self):
        for r in self.silver:
            user = r["user"]
            self.gold.setdefault(user, {"total": 0, "count": 0})
            self.gold[user]["total"] += r.get("amount", 0)
            self.gold[user]["count"] += 1
        print(f"Gold: {len(self.gold)} users aggregated")

lakehouse = LakehouseSimulator()
lakehouse.ingest([
    {"user": "alice", "amount": 100, "page": "/home"},
    {"user": "bob", "amount": 50, "page": "/about"},
    {"user": None, "amount": 200, "page": "/error"},
])
lakehouse.clean()
lakehouse.aggregate()
for user, stats in lakehouse.gold.items():
    print(f"  {user}: {stats['count']} events, ${stats['total']:.2f}")

FAQ

Related Concepts

What's Next

You now understand the data lake vs warehouse trade-offs! Next, explore Data Lakehouse architecture with Delta Lake, and learn how Apache Spark processes data at scale across both lakes and warehouses.

• Practice daily — Set up Bronze/Silver/Gold folders for your personal data
• Build a project — Use AWS S3 or MinIO to create a small data lake
• Explore related topics — Check out Apache Iceberg for open table formats

Remember: every expert was once a beginner. Keep coding!