Ordering Guarantee #

In distributed systems, preserving the chronological order of events (event ordering) is one of the most complex yet crucial challenges. Imagine a banking system where a deposit transaction of Rp1,000,000 is written after a withdrawal transaction of Rp500,000, even though the withdrawal happened later. If these message orders get reversed, the system might reject the withdrawal because it detects insufficient balance, leading to transaction failure and user dissatisfaction. Apache Kafka is designed to solve this problem by providing very strong ordering guarantees, but those guarantees have architectural limits we must understand well to avoid bugs in production.

Partition-Level vs Topic-Level Ordering Guarantees #

One of the most common misconceptions among software architects new to Kafka is assuming Kafka guarantees message order at the topic level overall (topic-level ordering). In reality, Kafka only guarantees message order at the partition level (partition-level ordering).

Why Is Topic-Level Ordering an Anti-Pattern? #

To understand why Kafka limits ordering guarantees to the partition level, we must look back at Kafka’s main purpose: massive horizontal scalability with millions of messages per second throughput.

If Kafka had to guarantee topic-level ordering globally, all messages entering that topic would have to be written to a single log file sequentially. That would force us to use only one partition for the topic. Limiting a topic to one partition destroys Kafka’s horizontal scalability:

  1. Broker Bottleneck: Only one broker (the partition leader) can process write and read operations for that topic.
  2. Consumer Scaling Limit: We can’t scale consumers using a consumer group, because at most one consumer instance can read from that single partition at a time.
  3. Throughput Limit: System throughput is immediately capped by the disk I/O and network capabilities of a single broker machine.

Therefore, Kafka introduced workload division through partitions, where each partition is an independent append-only log file with its own internal order.

Key-Based Ordering Mechanism (Message Key) #

For partition-level ordering guarantees to be useful in solving real business problems, we must ensure that chronologically related events always land in the same partition. This is where the Message Key role becomes crucial.

When publishing a message, producers can include a key (for example, user_id, transaction_id, or order_id). By default, Kafka producers use the Murmur2 hashing algorithm to map that key to a specific partition number using the following formula:

$$\text{Partition} = \text{abs}(\text{MurmurHash2}(\text{Key})) \pmod{\text{Number of Partitions}}$$

With this mechanism, all events with the same key value are guaranteed to always be sent to the same partition, and thus their order is absolutely preserved within that partition.

flowchart TD
    subgraph ProducerClient["Producer (Client)"]
        direction TB
        E1["Event 1 (Key: User_A)"]
        E2["Event 2 (Key: User_B)"]
        E3["Event 3 (Key: User_A)"]
    end

    subgraph KafkaBroker["Kafka Broker (Topic: user-actions)"]
        direction TB
        subgraph P0["Partition 0"]
            P0_L1["Event 1 (Key: User_A)"]
            P0_L2["Event 3 (Key: User_A)"]
        end
        subgraph P1["Partition 1"]
            P1_L1["Event 2 (Key: User_B)"]
        end
    end

    E1 -->|"Hash(User_A) -> P0"| P0
    E2 -->|"Hash(User_B) -> P1"| P1
    E3 -->|"Hash(User_A) -> P0"| P0

    style P0 stroke:#0288d1,stroke-width:2px
    style P1 stroke:#2e7d32,stroke-width:2px

[!WARNING] Impact of Partition Expansion: If we add more partitions to a running topic, the modulo calculation result of the key hash changes. As a result, new messages with the same key will likely be mapped to a different partition than previous messages. This breaks the historical data ordering guarantee for that key. If ordering guarantees are crucial, avoid dynamic partition expansion, or use a custom partitioner that doesn’t directly depend on the total partition count.

Producer-Side Guarantees: Handling Out-of-Order from Retry #

Guaranteeing order on the broker alone isn’t enough. Ordering problems often happen during data’s journey from producer to broker. In unstable distributed network environments, data transmission failures are common.

Out-of-Order Risk with Standard Retry #

By default, Kafka producers are configured to send messages asynchronously for maximum throughput. Producers can send several batch requests simultaneously before receiving acknowledgements from the broker. The parameter controlling this parallel request count is max.in.flight.requests.per.connection.

Let’s break down the failure scenario without ordering protection:

  1. The producer sends Batch A (Message 1 & 2) and Batch B (Message 3 & 4) in parallel. max.in.flight.requests.per.connection is 5.
  2. The broker receives Batch B first and successfully writes it to disk.
  3. Batch A’s transmission experiences a temporary network failure (for example, a lost packet).
  4. The broker sends an error message to the producer for Batch A.
  5. Because the producer is configured with retries > 0, the producer retries sending Batch A.
  6. The broker successfully receives the resent Batch A and writes it to disk.

At the end of this scenario, the messages on the broker are written in the order: Message 3, Message 4, Message 1, Message 2. The original chronological order is completely destroyed!

Classic Solution vs Modern Solution #

In the past, the only way to guarantee producer order during failures was to limit the parallel request count to one:

# Classic Solution (ANTI-PATTERN for High Throughput)
max.in.flight.requests.per.connection=1
retries=2147483647

With this configuration, the producer never sends the next request before the previous request gets a successful ACK. Although this guarantees 100% ordering, it drastically reduces throughput because it turns the send process fully synchronous (one by one).

Since Kafka 0.11, a far more elegant solution was introduced: the Idempotent Producer.

# Modern Solution (Highly Recommended)
enable.idempotence=true
max.in.flight.requests.per.connection=5
retries=2147483647

How the Idempotent Producer Works #

When enable.idempotence is set to true, the broker allocates a unique ID called Producer ID (PID) for every new producer during the initialization phase. Additionally, every message sent by the producer gets a Sequence Number that increases monotonically for the specific destination partition.

The broker tracks the last successfully written Sequence Number for each PID and partition pair. When the broker receives a new message, it verifies its Sequence Number:

  • If Incoming SeqNum = Last Written SeqNum + 1, the broker accepts the message.
  • If Incoming SeqNum <= Last Written SeqNum, the broker detects the message as a duplicate and immediately discards it without writing to the log, but still sends a successful ACK to the producer (so the producer knows the message is safe).
  • If Incoming SeqNum > Last Written SeqNum + 1, the broker detects a data gap (out-of-sequence), rejects the message with the OutOfOrderSequenceException error, and forces the producer to reconcile.

With this rejection mechanism, the broker guarantees there will never be gaps or swapped message orders in the commit log, even if the producer sends multiple in-flight requests in parallel.

sequenceDiagram
    participant P as "Idempotent Producer"
    participant B as "Kafka Broker"
    
    Note over P,B: Initialization: Producer gets PID 9999
    
    P->>B: Send Message A (PID=9999, Seq=0)
    P->>B: Send Message B (PID=9999, Seq=1)
    
    Note over B: Message A written successfully
    B-->>P: ACK Message A (Success)
    
    Note over B: Message B written successfully, but ACK lost in network
    B--xP: ACK Message B (Delivery Failed)
    
    P->>B: Retry Send Message B (PID=9999, Seq=1)
    Note over B: Broker detects Seq 1 already exists (Duplicate)
    B-->>P: ACK Message B (Success, Duplicate Ignored)
    
    P->>B: Send Message C (PID=9999, Seq=2)
    B-->>P: ACK Message C (Success)

Consumer-Side Guarantees: Handling Concurrency Problems #

Even if our data is written on the broker in perfect order, all those guarantees can be destroyed instantly when the data reaches the consumer application side. This problem almost always stems from one thing: unmanaged parallel processing.

Anti-Pattern: Unpinned Thread Pool #

By design, Kafka consumers operate with a single-thread model (single-threaded poll loop). One KafkaConsumer instance calls .poll() to fetch one batch of messages from the broker, processes them sequentially, then commits the last offset.

However, in the real world, processing messages sequentially in one thread is often too slow, especially when the process involves heavy I/O operations like calling external APIs or writing to SQL databases. To speed up processing, developers are often tempted to distribute the messages from the .poll() batch into a thread pool (ExecutorService) asynchronously.

Let’s look at this wrong Java implementation example:

// ANTI-PATTERN: Destroying message ordering guarantees asynchronously
public class DangerousConsumer {
    private final ExecutorService threadPool = Executors.newFixedThreadPool(10);
    private final KafkaConsumer<String, String> consumer = new KafkaConsumer<>(configs);

    public void start() {
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                // Distributing messages directly to the thread pool without caring about the key
                threadPool.submit(() -> {
                    processBusinessLogic(record); // ✗ Order broken here!
                });
            }
            // Commit offsets automatically or manually here (critical: offset committed before processing finishes!)
        }
    }
}

Why Is the Code Above Very Dangerous? #

  1. Non-deterministic Thread Scheduling: Imagine two messages from the same partition arrive in one batch: Message 1 (Update User Name to ‘Budi’) and Message 2 (Update User Name to ‘Joko’). If Message 1 is delegated to Thread A and Message 2 to Thread B, there’s no guarantee Thread A executes first in the OS. If Thread B runs faster, the user name in the database changes to ‘Joko’ first, then gets overwritten by Thread A to ‘Budi’. The database data is now out of sync with the actual event order.
  2. Commit Before Finish: The offset is committed immediately after the loop finishes distributing tasks to the thread pool, not after the tasks are actually finished by worker threads. If one thread crashes while processing a message, that message is lost forever from a processing perspective because its offset was already committed as successful.

Solution: Key-Pinned Worker Pool (Thread-Affinity) #

To increase throughput through parallel processing without destroying ordering guarantees, we must apply the Key-Pinned Worker Pool pattern.

The basic concept is very simple: we divide the workload among several worker threads, but we must guarantee that messages with the same key (or from the same partition) must always be allocated to the same worker thread consistently.

flowchart TD
    subgraph ConsumerProcess["Kafka Consumer Process"]
        CThread["Consumer Thread (Poll Loop)"]
        
        subgraph KeyPinnedPool["Key-Pinned Worker Pool"]
            direction TB
            Q0["Worker Queue 0"]
            Q1["Worker Queue 1"]
            Q2["Worker Queue 2"]
            
            T0["Worker Thread 0"]
            T1["Worker Thread 1"]
            T2["Worker Thread 2"]
            
            Q0 --> T0
            Q1 --> T1
            Q2 --> T2
        end

        CThread -->|"Hash(Key_A) % 3 -> Queue 0"| Q0
        CThread -->|"Hash(Key_B) % 3 -> Queue 1"| Q1
        CThread -->|"Hash(Key_A) % 3 -> Queue 0"| Q0
    end

    style KeyPinnedPool stroke:#e5e7eb
    style Q0 stroke:#0288d1,stroke-width:2px
    style Q1 stroke:#2e7d32,stroke-width:2px
    style Q2 stroke:#f57c00,stroke-width:2px

Here’s a robust Java implementation example for the Key-Pinned Worker Pool pattern:

// CORRECT: Securing ordering guarantees using a Key-Pinned Worker Pool
public class SafeKeyPinnedConsumer {
    private final KafkaConsumer<String, String> consumer;
    private final List<BlockingQueue<Runnable>> workerQueues;
    private final List<Thread> workerThreads;
    private final int numWorkers;

    public SafeKeyPinnedConsumer(int numWorkers, Properties configs) {
        this.numWorkers = numWorkers;
        this.consumer = new KafkaConsumer<>(configs);
        this.workerQueues = new ArrayList<>(numWorkers);
        this.workerThreads = new ArrayList<>(numWorkers);

        // Initialize queues and worker threads manually
        for (int i = 0; i < numWorkers; i++) {
            BlockingQueue<Runnable> queue = new LinkedBlockingQueue<>(1000);
            workerQueues.add(queue);
            
            final int workerId = i;
            Thread thread = new Thread(() -> {
                try {
                    while (!Thread.currentThread().isInterrupted()) {
                        Runnable task = queue.take();
                        task.run();
                    }
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }, "Worker-Thread-" + workerId);
            
            workerThreads.add(thread);
            thread.start();
        }
    }

    public void start() {
        try {
            consumer.subscribe(Collections.singletonList("user-actions"));
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    // Find the worker thread index based on the message key hash
                    int workerIndex = getWorkerIndex(record.key());
                    BlockingQueue<Runnable> targetQueue = workerQueues.get(workerIndex);

                    // Send the task to the specific worker thread for this key (blocking write)
                    targetQueue.put(() -> {
                        processBusinessLogic(record);
                    });
                }
                // Note: Safe offset commit management requires tracking offsets
                // per thread that were successfully completed before committing to the broker.
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } finally {
            close();
        }
    }

    private int getWorkerIndex(String key) {
        if (key == null) {
            // If there's no key, send to a random worker or fall back to partition
            return 0;
        }
        // Make sure the index value is always positive
        return Math.abs(key.hashCode()) % numWorkers;
    }

    private void processBusinessLogic(ConsumerRecord<String, String> record) {
        // Our business logic runs safely here sequentially per key
        System.out.printf("[%s] Processing key: %s, Value: %s%n", 
            Thread.currentThread().getName(), record.key(), record.value());
    }

    public void close() {
        for (Thread thread : workerThreads) {
            thread.interrupt();
        }
        consumer.close();
    }
}

Handling Failure Cases Without Breaking Order #

The next biggest challenge in maintaining data ordering is when we face processing errors (poison pill / corrupted messages). How do we handle messages that fail to process without destroying the ordering guarantee of subsequent data?

The Dead Letter Queue (DLQ) Dilemma #

In standard messaging architecture, the common approach to handling failures is sending the failed message to a special topic called the Dead Letter Queue (DLQ), then the consumer continues to the next message.

However, in systems requiring strict ordering, this standard DLQ approach is a fatal mistake.

flowchart TD
    subgraph Orig["Original Message Flow on Broker"]
        P1["Message 1 (Update Address: St. A)"] --> P2["Message 2 (Update Address: St. B)"]
    end
    
    subgraph Fail["Failure Scenario"]
        direction TB
        Step1["1. Consumer processes Message 1 (database error occurs, e.g., timeout)"] --> Step2["2. Consumer sends Message 1 to DLQ"]
        Step2 --> Step3["3. Consumer reads and processes Message 2 (St. B)<br>Successfully updates the address in the DB to 'St. B'"]
        Step3 --> Step4["4. Ops team takes Message 1 from DLQ and reprocesses it<br>The DB address is updated back to 'St. A'"]
    end

The final result: The database data is wrong (back to St. A), even though the user’s last legitimate change was St. B.

Safe Error Handling Strategies #

If we can’t immediately discard messages to DLQ, what alternatives do we have?

1. Stop and Retry (Blocking Retry) #

The consumer stops processing data for that partition and retries continuously until the message is successfully processed or the infrastructure problem is resolved.

  • Advantage: Data ordering is guaranteed 100% because the next message won’t be processed before the problematic message finishes.
  • Disadvantage: Causes severe processing lag. One corrupted message (poison pill) holds the entire partition hostage and stops processing for thousands of other users who happen to share the same partition.

2. Pause & Resume Consumer (Recommended Solution) #

To mitigate the impact of blocking retry, we can use the pause() and resume() APIs from the Kafka consumer.

When a processing failure happens on a specific partition:

  1. We call consumer.pause(collections.singleton(partition)) to stop fetching new messages specifically for that partition.
  2. The consumer keeps processing data for other healthy partitions without interruption.
  3. We send the failed message to an internal retry mechanism (for example, storing it in a local database or temporary memory with a certain wait period).
  4. After the wait period or database fix completes, we reprocess that message.
  5. If successful, we call consumer.resume(collections.singleton(partition)) to start fetching data from that partition again.

With this method, processing lag only affects users whose data is on the problematic partition, while other partitions continue processing data at maximum throughput.

Summary #

  • Limited Guarantee: Apache Kafka only guarantees message ordering at the partition level (partition-level), not globally at the topic level (topic-level).
  • Message Key Role: Use message keys consistently to ensure chronologically dependent events land in the same partition through the Murmur2 hash algorithm.
  • Idempotent Producer: Always enable enable.idempotence=true on the producer side to prevent data order corruption from network retry scenarios.
  • Consumer Concurrency Danger: Avoid distributing .poll() batch results directly to a standard thread pool. Use the Key-Pinned Worker Pool pattern to maintain ordering on the application side.
  • DLQ Dilemma: Instantly sending failed messages to DLQ destroys logical data ordering. Use the blocking retry strategy or leverage the Pause & Resume feature for safe error isolation.
← Previous: Partition Strategy
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact