AsyncAPI with Kafka — Complete Guide
In this tutorial, you will learn about AsyncAPI with Kafka. We cover key concepts, practical examples, and best practices to help you master this topic.
Apache Kafka is one of the most popular event streaming platforms. AsyncAPI provides comprehensive support for documenting Kafka topics, including partitions, Replication, schema registry, and consumer group configurations.
What You'll Learn
- Kafka topic configuration in AsyncAPI
- Partition and replication settings
- Schema registry integration
- Consumer group documentation
- Kafka-specific message features
Why It Matters
Kafka configurations are complex and often undocumented. AsyncAPI brings clarity to Kafka topics, making them discoverable and automatable through specification-driven tooling.
Real-World Use
A data engineering team documents all 200+ Kafka topics using AsyncAPI. Each topic spec includes partition count, retention policy, schema registry references, and owning team. This enables automated topic provisioning and self-service documentation.
Flow Chart
flowchart LR
A[Kafka AsyncAPI] --> B[Server Bindings]
A --> C[Topic Bindings]
A --> D[Message Bindings]
B --> E[Schema Registry URL]
C --> F[Partitions/Replicas]
C --> G[Retention Policy]
D --> H[Key Schema]
D --> I[Value Schema]
D --> J[Schema ID Location]
Code Examples
Example 1: Complete Kafka Topic Documentation
asyncapi: '2.6.0'
info:
title: Kafka Order Events
version: '2.0.0'
servers:
production:
url: kafka://kafka.example.com:9092
protocol: kafka
description: Production Kafka cluster
bindings:
kafka:
schemaRegistryUrl: https://schema-registry.example.com
schemaRegistryVendor: confluent
channels:
order.created:
description: Order creation events
bindings:
kafka:
topic: order.created
partitions: 6
replicas: 3
topicConfiguration:
cleanup.policy: delete
retention.ms: 604800000
retention.bytes: 1073741824
delete.retention.ms: 86400000
max.message.bytes: 1048588
compression.type: producer
publish:
operationId: publishOrderCreated
bindings:
kafka:
acks: all
compression.type: snappy
enable.idempotence: true
max.in.flight.requests.per.connection: 5
message:
name: OrderCreated
bindings:
kafka:
key:
type: string
description: Order ID
schemaIdLocation: header
schemaIdPayloadEncoding: confluent
payload:
$ref: '#/components/schemas/OrderCreatedEvent'
Expected output: Complete Kafka topic documentation with server bindings, topic configuration, and message key/schema settings.
Example 2: Consumer Group Documentation
servers:
production:
url: kafka://kafka.example.com:9092
protocol: kafka
channels:
order.created:
subscribe:
operationId: processOrderCreated
message:
payload:
$ref: '#/components/schemas/OrderCreated'
bindings:
kafka:
groupId: order-processor-group
clientId: order-processor-v1
session.timeout.ms: 45000
heartbeat.interval.ms: 3000
max.poll.interval.ms: 300000
auto.offset.reset: latest
enable.auto.commit: false
order.fulfilled:
subscribe:
operationId: processOrderFulfilled
message:
payload:
$ref: '#/components/schemas/OrderFulfilled'
bindings:
kafka:
groupId: order-fulfillment-group
clientId: fulfillment-service
session.timeout.ms: 60000
Expected output: Consumer groups with different configurations, including offset management and timeout settings.
Example 3: Kafka with Avro and Schema Registry
channels:
user.events:
bindings:
kafka:
topic: user.events
partitions: 3
publish:
message:
schemaFormat: 'application/vnd.apache.avro+json'
bindings:
kafka:
key:
type: string
schemaIdLocation: header
schemaIdPayloadEncoding: confluent
payload:
type: record
name: UserEvent
namespace: com.example.events
fields:
- name: userId
type: string
- name: eventType
type:
type: enum
name: UserEventType
symbols:
- LOGIN
- LOGOUT
- PROFILE_UPDATE
- name: timestamp
type:
type: long
logicalType: timestamp-millis
- name: metadata
type:
type: record
name: EventMetadata
fields:
- name: ipAddress
type: string
- name: userAgent
type: [string, null]
default: null
subscribe:
message:
schemaFormat: 'application/vnd.apache.avro+json'
payload:
type: record
name: UserCommand
namespace: com.example.commands
fields:
- name: commandId
type: string
- name: userId
type: string
- name: action
type: string
Expected output: Kafka topic using Avro schemas registered with Confluent Schema Registry.
Common Mistakes
| Mistake | Explanation |
|---|---|
| Not specifying partition count | Partitions affect parallelism; always document the expected partition count |
| Forgetting key schema | Kafka keys determine Partitioning; document key schemas explicitly |
| Mixing schema formats | Use one schema format consistently (Avro, JSON Schema, Protobuf) per topic |
| Ignoring consumer group configs | Consumer group settings like auto.offset.reset affect behavior significantly |
| Not documenting retention policies | Retention policy affects data availability; consumers must know how far back they can read |
Practice Questions
- How do you configure Kafka topic partitions in AsyncAPI?
- What is the role of the schema registry in Kafka AsyncAPI?
- How do you document consumer group configurations?
- What Kafka-specific message bindings are available?
- How do you handle multiple schema versions in a Kafka topic?
Challenge
Create a comprehensive AsyncAPI specification for a Kafka-based event platform with multiple topics, Avro schemas, schema registry integration, consumer groups, and detailed topic configuration including retention, compression, and cleanup policies.
FAQ
Mini Project
Design a complete Kafka event platform for an e-commerce system using AsyncAPI. Include topics for orders, payments, inventory, and shipping. Use Avro schemas with Confluent Schema Registry, configure appropriate retention and compaction policies, and document consumer groups for each microservice.
What's Next
Learn how to use AsyncAPI with MQTT
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro