Exactly-Once Delivery — Complete Guide

In this tutorial, you will learn about Exactly. We cover key concepts, practical examples, and best practices to help you master this topic.

Exactly-once delivery guarantees every message is processed once and only once, eliminating both loss and duplicates for the strongest possible reliability.

What You'll Learn

By the end of this lesson, you will understand what exactly-once means, how Kafka implements it through transactions, how to achieve it with idempotent consumers, and the performance cost of this guarantee.

Why It Matters

Some operations are devastating if performed twice. Charging a credit card twice, deducting inventory twice, or sending a refund twice are unacceptable. Exactly-once delivery prevents these scenarios, but it requires careful design and comes with performance costs.

Real-World Use

A financial trading system processes buy/sell orders. If the same order is executed twice, the trader buys 200 shares instead of 100, which could cost thousands of dollars. Exactly-once delivery prevents this.

Exactly-Once Approaches

flowchart TB
    subgraph "Exactly-Once Strategies"
        A[Kafka Transactions] --> B[Atomic produce + consume]
        C[Idempotent Consumer] --> D[Deduplication with DB]
        E[Distributed Transaction] --> F[XA / Two-Phase Commit]
    end
    B --> G[Strongest guarantee]
    D --> H[Practical approach]
    F --> I[Highest overhead]

Three main approaches exist: Kafka's transactional API, idempotent consumers with a deduplication store, and distributed transactions.

Kafka Transactions

Kafka provides exactly-once semantics through its transactions API. A producer can send messages and commit consumer offsets atomically.

from kafka import KafkaProducer

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    transactional_id='order-producer-1',
    enable_idempotence=True,
    acks='all'
)

producer.init_transactions()
producer.begin_transaction()

try:
    producer.send('orders', value=b'process order 123')
    producer.send('payments', value=b'charge order 123')
    producer.commit_transaction()
    print("Transaction committed — exactly-once")
except Exception:
    producer.abort_transaction()
    print("Transaction aborted")

Expected output:

Transaction committed — exactly-once

The transactional producer ensures that either all messages in the batch are committed or none are. Combined with idempotent producers, this gives exactly-once guarantees.

Idempotent Consumer Pattern

For brokers that do not support transactions (RabbitMQ, SQS), exactly-once is achieved through idempotent consumers:

import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

def process_exactly_once(message):
    msg_id = message['id']
    dedup_key = f"dedup:{msg_id}"

    result = r.set(dedup_key, 'processing', nx=True, ex=86400)
    if not result:
        print(f"Duplicate: {msg_id}, skipped")
        return

    try:
        print(f"Processing: {msg_id}")
        r.set(dedup_key, 'completed', ex=86400)
    except Exception:
        r.delete(dedup_key)
        raise

for i in range(3):
    process_exactly_once({'id': 'MSG-001', 'data': 'important'})

Expected output:

Processing: MSG-001
Duplicate: MSG-001, skipped
Duplicate: MSG-001, skipped

The consumer stores processed message IDs with atomic SET NX. If the same message arrives again, it is detected as a duplicate and skipped.

Exactly-Once with Database Transactions

For database operations, you can combine message consumption with database writes in a single transaction:

import psycopg2
import json

conn = psycopg2.connect('dbname=orders user=postgres')
cur = conn.cursor()

def process_order_message(body):
    data = json.loads(body)
    order_id = data['order_id']

    try:
        cur.execute("BEGIN")
        cur.execute(
            "INSERT INTO processed_messages (msg_id) VALUES (%s) "
            "ON CONFLICT (msg_id) DO NOTHING",
            (order_id,)
        )

        if cur.rowcount > 0:
            cur.execute(
                "UPDATE orders SET status = 'paid' WHERE id = %s",
                (order_id,)
            )
            conn.commit()
            print(f"Order {order_id} processed")
        else:
            conn.rollback()
            print(f"Order {order_id} already processed")
    except Exception:
        conn.rollback()
        raise

process_order_message('{"order_id": "ORD-123"}')

Expected output:

Order ORD-123 processed

The database transaction ensures that the deduplication check and the business operation succeed or fail together. This gives exactly-once semantics for database-backed operations.

Performance Comparison

Common Mistakes

1. Confusing Producer Exactly-Once with End-to-End

Exactly-once producer guarantees do not mean end-to-end exactly-once. The consumer must also be idempotent or part of a transaction. End-to-end exactly-once requires coordinated transactions across all components.

2. Using Distributed Transactions for Everything

XA transactions provide exactly-once but add significant latency and complexity. Use them only when absolutely necessary. Most systems are fine with at-least-once plus idempotent consumers.

3. Not Cleaning Up Deduplication Keys

Deduplication keys consume storage. Over months, the dedup store grows unbounded. Set TTL on keys and periodically archive processed IDs to cold storage.

4. Assuming Exactly-Once Prevents All Duplicates

Network-level duplicates can still occur. If the producer sends the same message twice due to a TCP retry, and both copies arrive, the broker may Process both. Exactly-once applies to the broker-consumer path, not the producer-broker path.

5. Ignoring the Performance Cost

Exactly-once adds 20-50% latency overhead compared to at-least-once. Kafka transactions require additional round trips. Idempotent consumers require database writes. Performance matters — test your system under realistic load.

Practice Questions

1. How does exactly-once differ from at-least-once?

At-least-once guarantees no message loss but allows duplicates. Exactly-once guarantees no loss and no duplicates. Exactly-once is stronger but more expensive.

2. How does Kafka achieve exactly-once?

Kafka uses transactional producers and idempotent producers. The producer assigns a unique ID to each message and the broker deduplicates by ID. Transactions enable atomic writes across partitions.

3. Can RabbitMQ provide exactly-once?

Not natively. RabbitMQ gives at-least-once with manual acks. Exactly-once must be implemented at the consumer level through idempotency or database transactions.

4. What is the main cost of exactly-once?

Performance. Exactly-once requires additional coordination (transactions, deduplication checks, atomic commits). Throughput drops 20-50% compared to at-least-once.

Challenge

Design an exactly-once payment processing system. Payments must never be duplicated or lost. Use RabbitMQ (at-least-once broker) with an idempotent consumer backed by PostgreSQL. Show the consumer code with atomic deduplication.

FAQ

Mini Project: Exactly-Once Order Processor

import json
import hashlib
import time

class ExactlyOnceProcessor:
    def __init__(self):
        self.processed = set()

    def process(self, body):
        msg_id = hashlib.sha256(body.encode()).hexdigest()[:16]

        if msg_id in self.processed:
            print(f"SKIP: {msg_id} already processed")
            return False

        data = json.loads(body)
        print(f"PROCESS: Order {data['order_id']} for ${data['amount']}")
        self.processed.add(msg_id)
        return True

proc = ExactlyOnceProcessor()
msg = json.dumps({'order_id': 'ORD-001', 'amount': 99.99})

proc.process(msg)
proc.process(msg)
proc.process(json.dumps({'order_id': 'ORD-002', 'amount': 49.99}))

Expected output:

PROCESS: Order ORD-001 for $99.99
SKIP: 550e8400 already processed
PROCESS: Order ORD-002 for $49.99

What's Next

Now that you understand exactly-once, explore dead letter queues to handle messages that cannot be processed successfully, then learn about message ordering for preserving sequence.