Real-Time Data Processing: Kafka, Flink, and Stream Architecture

In This Guide

• 1. Why Real-Time? Batch vs Stream Processing
• 2. Apache Kafka — The Event Backbone
• 3. Stream Processing Engines — Flink, Spark, ksqlDB
• 4. Stream Processing Patterns
• 5. Exactly-Once Semantics — The Hardest Problem
• 6. Reference Architectures
• 7. Operational Considerations
• 8. Frequently Asked Questions

Kafka Architecture — Key Concepts

Producer (Order Service)
    │
    ▼
┌──────────────────────────────────┐
│  Topic: "orders"                  │
│  ┌──────────┬──────────┬────────┐│
│  │Partition 0│Partition 1│Partition 2││
│  │ msg1,msg4 │ msg2,msg5 │ msg3,msg6 ││
│  └──────────┴──────────┴────────┘│
│  Retention: 7 days (configurable) │
└──────────────────────────────────┘
    │              │              │
    ▼              ▼              ▼
Consumer Group A        Consumer Group B
(Fraud Detection)       (Analytics Pipeline)
 ├─ Consumer 1 ←─ P0    ├─ Consumer 1 ←─ P0, P1
 ├─ Consumer 2 ←─ P1    └─ Consumer 2 ←─ P2
 └─ Consumer 3 ←─ P2

Key insight: Each consumer group tracks its own offset independently.
Group A can be real-time; Group B can be behind by hours — both are fine.

Kafka Producer — Java Example

// Producer with idempotent writes (prevents duplicates on retry)
Properties props = new Properties();
props.put("bootstrap.servers", "kafka-1:9092,kafka-2:9092,kafka-3:9092");
props.put("enable.idempotence", "true");       // Exactly-once producer
props.put("acks", "all");                       // Wait for all replicas
props.put("key.serializer", StringSerializer.class);
props.put("value.serializer", JsonSerializer.class);

KafkaProducer<String, OrderEvent> producer = new KafkaProducer<>(props);

// Key determines partition — same customer_id always goes to same partition
// This guarantees ordering per customer
OrderEvent event = new OrderEvent("ord-12345", "customer-789", "PLACED", items);
producer.send(new ProducerRecord<>("orders", event.customerId(), event),
    (metadata, exception) -> {
        if (exception != null) log.error("Failed to send", exception);
        else log.info("Sent to partition {} offset {}", metadata.partition(), metadata.offset());
    });

// Python equivalent (confluent-kafka)
from confluent_kafka import Producer

producer = Producer({'bootstrap.servers': 'kafka-1:9092',
                     'enable.idempotence': True})
producer.produce('orders', key='customer-789',
                 value=json.dumps(order_event).encode())

Apache Flink — Fraud Detection Example

// Flink DataStream API — detect rapid transactions from same card
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

DataStream<Transaction> transactions = env
    .addSource(new FlinkKafkaConsumer<>("transactions",
        new TransactionDeserializer(), kafkaProps));

// Pattern: 3+ transactions within 60 seconds from same card
Pattern<Transaction, ?> fraudPattern = Pattern.<Transaction>begin("first")
    .where(new SimpleCondition<Transaction>() {
        public boolean filter(Transaction t) { return t.getAmount() > 100; }
    })
    .followedBy("second").where(/* same card */)
    .followedBy("third").where(/* same card */)
    .within(Time.seconds(60));

PatternStream<Transaction> patternStream = CEP.pattern(
    transactions.keyBy(Transaction::getCardId), fraudPattern);

patternStream.select(new PatternSelectFunction<Transaction, FraudAlert>() {
    public FraudAlert select(Map<String, List<Transaction>> pattern) {
        return new FraudAlert(pattern.get("first").get(0).getCardId(),
            "3 transactions > $100 within 60 seconds");
    }
}).addSink(new FlinkKafkaProducer<>("fraud-alerts", ...));

env.execute("Fraud Detection Pipeline");

ksqlDB — Stream Processing with SQL

-- Create a stream from a Kafka topic
CREATE STREAM orders (
    order_id VARCHAR KEY,
    customer_id VARCHAR,
    amount DECIMAL(10,2),
    status VARCHAR,
    created_at TIMESTAMP
) WITH (
    kafka_topic = 'orders',
    value_format = 'JSON',
    timestamp = 'created_at'
);

-- Real-time aggregation: revenue per customer in sliding 1-hour window
CREATE TABLE customer_hourly_spend AS
SELECT customer_id,
       SUM(amount) AS total_spent,
       COUNT(*) AS order_count
FROM orders
WINDOW TUMBLING (SIZE 1 HOUR)
GROUP BY customer_id
HAVING SUM(amount) > 1000
EMIT CHANGES;

-- This automatically produces to a Kafka topic
-- Downstream services consume it for real-time alerts

CDC with Debezium — Database to Kafka

// Debezium connector config — captures every INSERT/UPDATE/DELETE from PostgreSQL
{
    "name": "orders-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "database.hostname": "orders-db.internal",
        "database.port": "5432",
        "database.dbname": "orders",
        "database.user": "debezium",
        "table.include.list": "public.orders,public.order_items",
        "topic.prefix": "cdc",
        "slot.name": "debezium_orders",
        "plugin.name": "pgoutput"
    }
}

// Result: Every change in orders table → Kafka topic "cdc.public.orders"
// Event payload includes: before (old values), after (new values), operation (c/u/d)
// Downstream consumers get real-time database change events

                    ┌─── Batch Layer (Spark) ──→ Batch Views ───┐
Raw Events ──→ Kafka ─┤                                           ├──→ Serving Layer
                    └─── Speed Layer (Flink) ──→ Real-Time Views ─┘

Pros: Accurate (batch corrects streaming approximations)
Cons: Maintaining two codepaths, double the complexity

Raw Events ──→ Kafka ──→ Stream Processor (Flink) ──→ Serving Layer

Reprocessing: Reset consumer offset → replay from Kafka
Pros: Single codebase, simpler ops
Cons: Kafka retention must be long enough for full reprocessing

Our recommendation: Start with Kappa. Lambda only when you genuinely need batch-level accuracy that streaming can't provide (rare in 2026 with Flink's exactly-once guarantees).

Order Service ──→ "order.placed" ──→ Kafka ──┬──→ Payment Service
                                              ├──→ Inventory Service
                                              ├──→ Notification Service
                                              └──→ Analytics Pipeline

Each service:
- Consumes events it cares about
- Produces events for downstream services
- Owns its own database (no shared state)
- Can be independently scaled and deployed

Related Articles