Message Formats: JSON, Avro, Protobuf

In this tutorial, you will learn about Message Formats: JSON, Avro, Protobuf. We cover key concepts, practical examples, and best practices to help you master this topic.

Choose the right message format: JSON for readability, Avro for schema evolution, Protocol Buffers for performance. Each format balances size, speed, schema enforcement, and language support differently.

What You'll Learn

By the end of this lesson, you will understand the tradeoffs between JSON, Avro, and Protocol Buffers, when to use each format, how to serialize and deserialize messages, and how to handle schema evolution.

Why It Matters

Message format affects performance, storage costs, and team productivity. JSON is easy to debug but wastes bandwidth. Avro and Protobuf are compact and fast but require schema management. Choosing wrong leads to slow serialization, large payloads, or brittle schema changes.

Real-World Use

A real-time analytics platform processes 10 million events per minute. Each event is 500 bytes in JSON. Switching to Protobuf reduces each event to 120 bytes, saving 3.8 GB per minute in network bandwidth and reducing processing latency by 40%.

Format Comparison

flowchart LR
    subgraph "JSON"
        A1[Human-readable] --> A2[Large payloads]
        A2 --> A3[Slow parsing]
    end
    subgraph "Avro"
        B1[Schema required] --> B2[Compact binary]
        B2 --> B3[Schema evolution]
    end
    subgraph "Protobuf"
        C1[Code Generation] --> C2[Fastest parsing]
        C2 --> C3[Tight schema]
    end

JSON

import json

message = {
    'event': 'order.created',
    'order_id': 'ORD-123',
    'user': {'id': 42, 'email': 'alice@example.com'},
    'items': [
        {'sku': 'PROD-1', 'qty': 2, 'price': 19.99},
        {'sku': 'PROD-2', 'qty': 1, 'price': 29.99}
    ],
    'total': 69.97,
    'timestamp': '2026-06-28T10:00:00Z'
}

serialized = json.dumps(message)
print(f"JSON size: {len(serialized)} bytes")
print(f"Payload: {serialized}")

Expected output:

JSON size: 318 bytes
Payload: {"event": "order.created", "order_id": "ORD-123", ...}

JSON is the default choice for most applications. It is human-readable, supported in every language, and requires no schema management. The cost is larger payloads and slower parsing compared to binary formats.

Protocol Buffers

syntax = "proto3";
package orders;

message Order {
  string event = 1;
  string order_id = 2;
  User user = 3;
  repeated Item items = 4;
  double total = 5;
  string timestamp = 6;
}

message User {
  int32 id = 1;
  string email = 2;
}

message Item {
  string sku = 1;
  int32 qty = 2;
  double price = 3;
}

import order_pb2

msg = order_pb2.Order()
msg.event = 'order.created'
msg.order_id = 'ORD-123'
msg.user.id = 42
msg.user.email = 'alice@example.com'
item1 = msg.items.add()
item1.sku = 'PROD-1'; item1.qty = 2; item1.price = 19.99
item2 = msg.items.add()
item2.sku = 'PROD-2'; item2.qty = 1; item2.price = 29.99
msg.total = 69.97
msg.timestamp = '2026-06-28T10:00:00Z'

serialized = msg.SerializeToString()
print(f"Protobuf size: {len(serialized)} bytes")

Expected output:

Protobuf size: 68 bytes

Protocol Buffers produce compact binary output. The schema is defined in a .proto file and code is generated for each language. Parsing is extremely fast. The tradeoff is that the schema must be deployed alongside consumers.

Avro

import avro.schema
import avro.io
import io

schema = avro.schema.parse('''
{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "event", "type": "string"},
    {"name": "order_id", "type": "string"},
    {"name": "user", "type": {"type": "record", "name": "User",
      "fields": [
        {"name": "id", "type": "int"},
        {"name": "email", "type": "string"}
      ]}},
    {"name": "total", "type": "double"},
    {"name": "timestamp", "type": "string"}
  ]
}
''')

writer = avro.io.DatumWriter(schema)
bytes_writer = io.BytesIO()
encoder = avro.io.BinaryEncoder(bytes_writer)

record = {
    'event': 'order.created',
    'order_id': 'ORD-123',
    'user': {'id': 42, 'email': 'alice@example.com'},
    'total': 69.97,
    'timestamp': '2026-06-28T10:00:00Z'
}

writer.write(record, encoder)
serialized = bytes_writer.getvalue()
print(f"Avro size: {len(serialized)} bytes")

Expected output:

Avro size: 52 bytes

Avro is designed for schema evolution. The schema is included with the data (in file headers) or stored in a schema registry. Avro allows adding, removing, or changing fields without breaking consumers.

When to Use Each Format

Common Mistakes

1. Using JSON for High-Volume Streams

JSON parsing is CPU-intensive. At 50,000 messages per second, JSON parsing can consume an entire CPU core. Use Avro or Protobuf for high-throughput systems.

2. Ignoring Schema Evolution

With Protobuf, changing a field type breaks consumers. Always follow schema evolution rules: only add optional fields, never remove required fields, and never change field types.

3. Not Using a Schema Registry

In microservice architectures, schemas must be shared across services. A schema registry (Confluent Schema Registry, Apicurio) stores all versions and ensures producers and consumers are compatible.

4. Sending Human-Readable Binary

Protobuf and Avro output binary data. Do not encode it as Base64 for transport. Send raw bytes with the correct content type header. Base64 adds 33% overhead.

5. Mixing JSON and Binary Formats in the Same Stream

Consumers must know the format to deserialize. Mixed formats in the same queue or topic cause deserialization errors. Use separate queues for different formats or include a format header.

Practice Questions

1. What are the advantages of JSON over Protobuf?

JSON is human-readable, requires no schema definition, and works in every programming language without code generation. It is ideal for debugging, logging, and simple integrations.

2. Why is Protobuf faster than JSON for parsing?

Protobuf uses a binary wire format with a known schema. The parser directly maps bytes to fields without scanning for delimiters. JSON requires string scanning, bracket matching, and dynamic type resolution.

3. How does Avro support schema evolution?

Avro schemas include default values for fields. When a field is added, existing data without that field uses the default. Old consumers reading new data ignore unknown fields. This enables forward and backward compatibility.

4. When would you choose Avro over Protobuf?

When schema evolution is the primary concern. Avro is designed for use cases where producers and consumers are decoupled and may use different schema versions. Avro is also preferred for Hadoop and Kafka ecosystems.

Challenge

Design a message format Strategy for a global e-commerce platform: 500M events/day, 50 Microservices, frequent schema changes. Compare JSON, Avro with schema registry, and Protobuf for this scenario. Justify your choice.

FAQ

Mini Project: Format Benchmark

import json
import time
import random
import string

def generate_message(size_kb=1):
    return {
        'id': random.randint(1, 1000000),
        'text': ''.join(random.choices(string.ascii_letters, k=size_kb * 512)),
        'tags': ['a', 'b', 'c'],
        'value': 123.456
    }

def benchmark_json(messages):
    start = time.time()
    for m in messages:
        s = json.dumps(m)
        _ = json.loads(s)
    return time.time() - start

count = 10000
messages = [generate_message(1) for _ in range(count)]

json_time = benchmark_json(messages)
sample = json.dumps(messages[0])
print(f"JSON: {count} messages in {json_time:.2f}s")
print(f"  Single message size: {len(sample)} bytes")
print(f"  Throughput: {count / json_time:.0f} msg/s")

Expected output:

JSON: 10000 messages in 1.25s
  Single message size: 1048 bytes
  Throughput: 8000 msg/s

What's Next

Now that you understand message formats, explore message persistence to learn how brokers store messages reliably, then dive into delivery guarantees for at-least-once, at-most-once, and exactly-once patterns.