Blob Store Design — S3, Azure Blob & GCS Architecture Guide

In this tutorial, you'll learn about Blob Store Design. We cover key concepts, practical examples, and best practices.

Blob storage (object storage) stores unstructured data as objects in a flat namespace accessed via HTTP APIs, providing massive scalability, durability, and low cost — the foundation for cloud storage services like S3, Azure Blob Storage, and Google Cloud Storage.

Why Blob Store Design Matters

The world generates 120+ zettabytes of data annually. Traditional file systems can't scale to petabyte-scale unstructured data. Object storage solves this with a flat namespace, HTTP-based access, and 99.999999999% (11 nines) durability through replication. S3 stores over 100 trillion objects and serves millions of requests per second. DodaZIP's cloud backup feature stores compressed archives in S3-compatible storage, leveraging its multi-region replication for disaster recovery.

Plain-Language Explanation

Think of a library where instead of organizing books on shelves (hierarchical file system), every book gets a unique barcode, and all books are stored in a giant warehouse. To find a book, you give the barcode to a robot that retrieves it from any shelf. Books never have the same barcode. You can't rename a book — you just store a new version with the same barcode. This is object storage: each object has a unique key, lives in a flat namespace, and is accessed via a simple API.

graph TD
    Client --> LB[Load Balancer]
    LB --> GW[API Gateway
S3-Compatible REST API]
    GW --> MD[MetaData Service
Object -> Location Mapping]
    MD --> PD[Partition Dispatcher]
    PD --> SP1[Storage Partition 1]
    PD --> SP2[Storage Partition 2]
    PD --> SP3[Storage Partition 3]
    PD --> SPN[Storage Partition N]
    SP1 --> REP1[Replica 1]
    SP1 --> REP2[Replica 2]
    SP1 --> REP3[Replica 3]
    GW --> AUTH[Authentication
& Authorization]
    style GW fill:#3498db,color:#fff
    style MD fill:#e67e22,color:#fff
    style PD fill:#27ae60,color:#fff

MinIO Implementation

MinIO is an open-source S3-compatible object store:

from minio import Minio
from minio.error import S3Error

client = Minio(
    "play.min.io:9000",
    access_key="minioadmin",
    secret_key="minioadmin",
    secure=True,
)

bucket_name = "my-bucket"

if not client.bucket_exists(bucket_name):
    client.make_bucket(bucket_name)
    print(f"Created bucket: {bucket_name}")

client.put_object(
    bucket_name,
    "photos/vacation/beach.jpg",
    io.BytesIO(b"fake image data"),
    length=15,
    content_type="image/jpeg",
)
print("Object uploaded")

objects = client.list_objects(bucket_name, prefix="photos/")
for obj in objects:
    print(f"{obj.object_name} - {obj.size} bytes")

Expected output:

Created bucket: my-bucket
Object uploaded
photos/vacation/beach.jpg - 15 bytes

Partitioning Strategy

Blob stores partition data by object key prefix:

import hashlib

class BlobPartitioner:
    def __init__(self, num_partitions: int = 256):
        self.num_partitions = num_partitions

    def get_partition(self, object_key: str) -> int:
        hash_val = int(hashlib.md5(object_key.encode()).hexdigest(), 16)
        return hash_val % self.num_partitions

    def should_split(self, partition_loads: dict) -> bool:
        avg = sum(partition_loads.values()) / len(partition_loads)
        max_load = max(partition_loads.values())
        return max_load > avg * 1.5

partitioner = BlobPartitioner(16)
keys = ["photos/beach.jpg", "docs/report.pdf", "videos/demo.mp4"]
for key in keys:
    print(f"{key} -> partition {partitioner.get_partition(key)}")

Versioning and Consistency

S3 provides read-after-write consistency for PUT of new objects and eventual consistency for overwrites:

class BlobVersioning:
    def __init__(self, store):
        self.store = store

    def put_object(self, key: str, data: bytes, version_id: str):
        self.store[f"{key}/versions/{version_id}"] = data
        self.store[f"{key}/current"] = version_id

    def get_object(self, key: str, version_id: str = None) -> bytes:
        if version_id:
            return self.store.get(f"{key}/versions/{version_id}")
        current = self.store.get(f"{key}/current")
        return self.store.get(f"{key}/versions/{current}")

    def list_versions(self, key: str) -> list:
        return [k.split("/")[-1] for k in self.store if k.startswith(f"{key}/versions/")]

Common Mistakes

1. Hot partition: Objects with the same popular prefix hash to the same partition, creating a hotspot. Add random prefixes to distribute load.

2. Not using multipart upload: Files over 100MB should use multipart upload for parallel throughput and retry resilience.

3. Ignoring request rate limits: S3 has a 3,500 PUT/POST/DELETE and 5,500 GET requests per second per prefix. Spread load across prefixes.

4. No lifecycle policies: Old object versions and incomplete multipart uploads accumulate cost. Configure lifecycle rules to expire or transition to cold storage.

5. Over-fetching in listings: Listing objects with prefix "" on a bucket with billions of objects is slow. Use delimited prefixes as a directory structure.

Practice Questions

1. How does blob storage differ from block storage? Blob storage stores objects via HTTP API with a flat namespace. Block storage presents as a mounted volume with a filesystem hierarchy. Blob storage scales to exabytes; block storage is limited by volume size.

2. What are the 11 nines of durability? 99.999999999% durability means losing one object out of 100 billion in a year. Achieved through erasure coding or multi-AZ replication.

3. How does S3 achieve strong consistency? S3 provides read-after-write consistency for PUTs of new objects and strong consistency for all operations since December 2020. Overwrites are now also strongly consistent.

4. What is erasure coding? Data is split into data and parity shards distributed across nodes. Any N of M shards can reconstruct the original data, providing durability with less overhead than full replication.

5. When should you use blob storage vs a database? Blob storage for large unstructured data (images, videos, backups). Databases for structured data requiring queries, joins, and transactions.

Mini Project

Build an S3-compatible blob server using Flask:

from flask import Flask, request, Response
import hashlib, os

app = Flask(__name__)
BASE = "./blobstore"
os.makedirs(BASE, exist_ok=True)

@app.put("/<bucket>/<path:key>")
def put_object(bucket, key):
    os.makedirs(f"{BASE}/{bucket}", exist_ok=True)
    path = f"{BASE}/{bucket}/{key.replace('/', '_')}"
    with open(path, "wb") as f:
        f.write(request.data)
    return {"etag": hashlib.md5(request.data).hexdigest()}, 201

@app.get("/<bucket>/<path:key>")
def get_object(bucket, key):
    path = f"{BASE}/{bucket}/{key.replace('/', '_')}"
    if not os.path.exists(path):
        return {"error": "Not found"}, 404
    with open(path, "rb") as f:
        return Response(f.read(), mimetype="application/octet-stream")

@app.delete("/<bucket>/<path:key>")
def delete_object(bucket, key):
    path = f"{BASE}/{bucket}/{key.replace('/', '_')}"
    if os.path.exists(path):
        os.remove(path)
    return "", 204

@app.get("/<bucket>")
def list_objects(bucket):
    prefix = request.args.get("prefix", "")
    files = []
    for f in os.listdir(f"{BASE}/{bucket}"):
        if f.startswith(prefix.replace("/", "_")):
            files.append(f.replace("_", "/"))
    return {"objects": files}

if __name__ == "__main__":
    app.run(port=9000)

Cross-References

• Key-Value Store Design
• Database Sharding
• Caching
• CDN
• Distributed Caching