Real-Time Stream Processing — Kafka, Flink and Event-Driven Architecture

In this tutorial, you'll learn about Real. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Real-time Stream Processing enables organizations to act on data within milliseconds of its creation — detecting fraud, personalizing content, monitoring infrastructure, and responding to business events as they happen rather than hours later in batch jobs.

What You'll Learn

You'll master event-time vs processing-time semantics, stateful Stream Processing with Flink, exactly-once delivery guarantees, windowing strategies (tumbling, sliding, session) for late-arriving data, Kafka as an event store, and production deployment patterns for low-latency streaming pipelines.

Why It Matters

Batch processing introduces hours of latency between data creation and insight. Real-time streaming reduces this to milliseconds, enabling fraud detection during a Transaction, personalized recommendations during a session, and infrastructure alerts before a cascading failure. At DodaTech, streaming patterns power the real-time threat detection pipeline in Durga Antivirus Pro, processing 10 million security events per second with sub-100ms latency.

Real-World Use

A ride-hailing company processes 25 million trips daily. A batch pipeline could calculate surge pricing every 6 hours. A streaming pipeline recalculates every 30 seconds, adjusting prices in real time as demand spikes near a concert venue. The difference is millions of dollars in revenue and thousands of drivers routed to high-demand areas.

Stream Processing Architecture

flowchart LR
    Producers[Event Producers] -->|Events| Kafka[Apache Kafka]
    Kafka --> Flink[Flink Stream Processor]
    Kafka --> Connect[Kafka Connect]
    Kafka --> KS[KSQL DB]
    Flink --> State[State Backend - RocksDB]
    Flink --> Sink1[(PostgreSQL - Enriched)]
    Flink --> Sink2[(Elasticsearch - Index)]
    Flink --> Sink3[Alert Topic]
    Sink2 --> Dashboard[Real-Time Dashboard]
    Sink3 --> Pager[Alerting System]
    Connect --> Sink4[(S3 - Archive)]

Event-Time vs Processing-Time Semantics

# Conceptual example — event time vs processing time
from dataclasses import dataclass
from datetime import datetime
import time

@dataclass
class Event:
    event_id: str
    event_time: datetime  # When the event actually happened
    processing_time: datetime  # When the system received it
    data: dict

events = [
    Event("e1", datetime(2026, 6, 22, 10, 30, 5), datetime(2026, 6, 22, 10, 30, 6), {"amount": 100}),
    Event("e2", datetime(2026, 6, 22, 10, 30, 3), datetime(2026, 6, 22, 10, 30, 8), {"amount": 200}),
    Event("e3", datetime(2026, 6, 22, 10, 30, 4), datetime(2026, 6, 22, 10, 30, 10), {"amount": 150}),
]
# Events arrived out of order (e1 arrived before e2, but e2 has earlier event_time)
# Event-time processing correctly groups e2 (10:30:03) in the 10:30:00 window
# Processing-time processing would incorrectly group e2 in the 10:30:05 window

Event-time processing groups events by when they actually occurred, not when the system received them. This is critical for out-of-order data from mobile devices, IoT sensors, and web clients with unreliable connections.

Flink Stateful Stream Processing

// Flink streaming job for real-time fraud detection
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.configuration.Configuration;

public class FraudDetectionJob {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        DataStream<Transaction> transactions = env
            .addSource(new KafkaSource<>("transactions", new TransactionDeserializer()))
            .keyBy(t -> t.userId);

        DataStream<Alert> alerts = transactions
            .process(new FraudDetector())
            .name("fraud-detection");

        alerts.addSink(new KafkaSink<>("fraud-alerts", new AlertSerializer()));

        env.execute("Real-Time Fraud Detection");
    }
}

class FraudDetector extends KeyedProcessFunction<String, Transaction, Alert> {
    private ValueState<Double> lastAmountState;
    private ValueState<Long> lastTimestampState;

    @Override
    public void open(Configuration parameters) {
        lastAmountState = getRuntimeContext().getState(
            new ValueStateDescriptor<>("last-amount", Double.class)
        );
        lastTimestampState = getRuntimeContext().getState(
            new ValueStateDescriptor<>("last-timestamp", Long.class)
        );
    }

    @Override
    public void processElement(Transaction t, Context ctx, Collector<Alert> out) throws Exception {
        Double lastAmount = lastAmountState.value();
        Long lastTimestamp = lastTimestampState.value();

        if (lastAmount != null && lastTimestamp != null) {
            long timeDiff = t.timestamp - lastTimestamp;
            double amountRatio = t.amount / lastAmount;

            if (timeDiff < 60000 && amountRatio > 5.0) {
                out.collect(new Alert(t.userId, t.transactionId,
                    String.format("Suspicious transaction: %.2f within %dms of previous %.2f",
                        t.amount, timeDiff, lastAmount)));
            }
        }

        lastAmountState.update(t.amount);
        lastTimestampState.update(t.timestamp);
    }
}

Expected behavior: The Flink job processes each Transaction, maintaining per-user state about their last Transaction amount and timestamp. If a user makes a Transaction more than 5x their previous amount within 60 seconds, an alert is emitted. State is stored in RocksDB for resilience and can hold billions of user sessions.

Windowing Strategies for Late Data

Streaming systems must handle late-arriving data. Flink provides configurable allowed lateness and side outputs for truly late events.

DataStream<SensorReading> sensorData = env
    .addSource(new KafkaSource<>("sensors", new SensorDeserializer()))
    .keyBy(s -> s.sensorId)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .allowedLateness(Time.minutes(2))
    .sideOutputLateData(lateOutputTag)
    .aggregate(new AverageAggregate());

// Main output: windows computed with 2-minute allowed lateness
sensorData.addSink(new CassandraSink<>("sensor_hourly_avg"));

// Side output: events that arrived more than 2 minutes late
DataStream<SensorReading> lateData = sensorData.getSideOutput(lateOutputTag);
lateData.addSink(new KafkaSink<>("late-sensor-data", new SensorSerializer()));

Exactly-Once Semantics

Kafka and Flink together provide end-to-end exactly-once processing through a combination of features.

// Enable exactly-once semantics in Flink
env.enableCheckpointing(10000);  // Checkpoint every 10 seconds
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(5000);
env.getCheckpointConfig().setCheckpointTimeout(60000);
env.getCheckpointConfig().setTolerableCheckpointFailureNumber(3);

// Kafka producer with exactly-once semantics
kafkaProducer.setProperty("enable.idempotence", "true");
kafkaProducer.setProperty("transactional.id", "fraud-detection-producer");
kafkaProducer.setProperty("acks", "all");

The exactly-once pipeline works as follows:

1. Flink checkpoints state and source offsets atomically
2. Kafka producer writes with idempotent and transactional semantics
3. On failure, Flink restores from the last successful checkpoint
4. Source reads resume from the checkpointed offset (no duplicate ingestion)
5. Sinks use transactional writes that commit on checkpoint completion

Common Errors

1. Processing-Time Windows for Out-of-Order Data

Using processing time instead of event time causes incorrect results when events arrive late. Always use event-time processing and configure allowed lateness based on your data arrival characteristics.

2. No Watermark Configuration

Without watermarks, Flink cannot determine when all events for a window have arrived. Configure watermarks based on expected event arrival delay. Set allowed lateness to handle outliers.

3. Ignoring Backpressure

When a downstream sink slows, backpressure propagates upstream through the entire pipeline. Monitor Kafka consumer lag, Flink task backlog, and sink throughput. Add auto-scaling or buffer sizing as mitigation.

4. Single-Partition Kafka Topics

Using one Kafka partition for high-throughput streams limits parallelism. Flink can only Process one partition per parallel subtask. Use at least as many partitions as the desired parallelism.

5. Large State Without RocksDB

Storing gigabytes of state in Flink's JVM heap causes GC pauses and out-of-memory errors. Switch to RocksDB state backend for large state. Monitor state size and configure incremental checkpoints.

6. Missing Dead Letter Queue

Events that fail processing should be sent to a dead letter queue, not silently dropped. Configure error handling with side outputs or a dedicated error topic.

7. No Schema Registry

Using raw JSON without schema validation leads to deserialization errors when producers update their event format. Use Confluent Schema Registry with Avro or Protobuf to enforce schema compatibility.

Practice Questions

1. What is the difference between event-time and processing-time processing?

Event-time processes events based on when they actually occurred (extracted from the event payload). Processing-time processes events based on when the system received them. Event-time is required for out-of-order data and accurate windowing.

2. How does Flink achieve exactly-once semantics?

Flink uses distributed checkpoints that save operator state and Kafka offsets atomically. On failure, the pipeline restores from the last checkpoint. Kafka's idempotent producer and transactional writes prevent duplicate output.

3. What is a watermark and why is it important?

A watermark is a heuristic that tracks the progress of event time. It tells Flink "no events with event-time before this timestamp will arrive." Watermarks allow the system to close Windows and trigger computations even with out-of-order data.

4. How do you handle late-arriving data after the watermark?

Configure allowedLateness to specify how long to wait for late events. Events that arrive after the allowed lateness window can be captured in a side output for separate processing or logging.

5. Challenge: Design a real-time sessionization system. Process clickstream events from Kafka, group them into user sessions (30-minute inactivity gap), compute per-session metrics (pages viewed, time spent, conversion), and output enriched session events to Elasticsearch for real-time analytics dashboarding.

Mini Project: Real-Time Streaming Dashboard

Build a real-time streaming analytics system:

1. Produce simulated events to a Kafka topic (user clicks, purchases, page views)
2. Flink streaming job that computes: rolling window counts, top 10 products, conversion funnel steps
3. Store results in Elasticsearch with hourly index rotation
4. Kibana or custom dashboard showing real-time metrics (QPS, active users, conversion rate)
5. Alert rule: if error rate exceeds 5 percent in a 1-minute window, send notification
6. Late data handling: events delayed more than 30 seconds go to a separate "late" index

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.