Key vs No-Key Message #

When we send data to Apache Kafka using a producer client, every message is wrapped in a ProducerRecord object. Inside this object, we can decide whether the message will be sent with a key or without a key (no-key). This seemingly simple architectural decision has an enormous impact on how data is distributed across cluster partitions, producer memory batch-writing efficiency, network performance, and even message ordering guarantees on the consumer side. Choosing the wrong delivery strategy can cause massive throughput drops or extreme load imbalance on brokers. Therefore, we must deeply understand the internal mechanism differences, advantages, and disadvantages of both approaches.

Sending Messages with a Key (Hashed Key Partitioning) #

When we create a ProducerRecord object with a key, for example:

ProducerRecord<String, String> record = new ProducerRecord<>("orders-topic", "User_A", "Order_1001");

We’re telling Kafka that the message "Order_1001" has a logical relationship with the identity "User_A".

How It Works: The Murmur2 Hashing Algorithm #

By default, if a message has a key, Kafka’s built-in Partitioner processes that key using the Murmur2 hash algorithm. The resulting binary hash is then converted to a positive number and modulo’d with the total number of active partitions on that topic to determine the destination partition number:

$$\text{Partition Number} = \text{Utils.toPositive}(\text{Utils.murmur2}(\text{keyBytes})) \pmod{\text{Total Partitions}}$$

The important characteristic of hashing algorithms is deterministic consistency. As long as the partition count on that topic doesn’t change, the same key always produces the same partition number. If key "User_A" is routed to Partition 2 today, then tomorrow or anytime a message with key "User_A" is sent, it will always go to Partition 2.

Data Ordering Guarantee #

Why do we need the guarantee that the same key always goes to the same partition? The answer is processing order. Kafka only guarantees message order at the specific partition level, not the overall topic level.

By routing all of "User_A"’s activities (for example: OrderCreated, OrderPaid, OrderShipped) to the same partition (e.g., Partition 2), consumers reading from Partition 2 are guaranteed to process those events in chronological order. If we spread those events across different partitions, there’s a chance the OrderPaid event is read by another consumer before the OrderCreated event, destroying our business data integrity.

Sending Messages Without a Key (No-Key Message) #

If a message is sent without a key, for example:

ProducerRecord<String, String> record = new ProducerRecord<>("logs-topic", "Log_Message_Content");

Our main goal is usually even data distribution without caring about processing order between messages. However, the internal mechanism of how Kafka handles keyless messages has undergone a major evolution for performance efficiency.

The Classic Mechanism: Round-Robin Partitioner (Before Kafka 2.4) #

In old Kafka versions, keyless messages were distributed using the Round-Robin method (alternating sequentially). The first message goes to Partition 0, the second to Partition 1, the third to Partition 2, and so on in a circular fashion.

• Critical Weakness: This approach triggers severe memory inefficiency inside the producer’s RecordAccumulator. Because messages are sent one by one alternately to each partition, the memory batch for each partition fills up very slowly. As a result, producers are often forced to send very small network batches (for example, only 1 KB filled out of the 16 KB batch.size capacity) because the linger.ms time limit has expired. This wastes network bandwidth and overloads broker I/O.

The Efficiency Revolution: Sticky Partitioner (Kafka 2.4+) #

To overcome Round-Robin’s inefficiency, Kafka introduced the Sticky Partitioner starting from version 2.4. This is a new, very clever standard for distributing keyless data.

How the Sticky Partitioner Works #

Instead of spreading messages thinly across all partitions alternately, the Sticky Partitioner picks one partition at random and sticks to that partition for all subsequent keyless messages.

1. The producer picks Partition 0 as the active target.
2. All new non-key messages are placed into Partition 0’s batch until that batch reaches the batch.size capacity (e.g., 16 KB).
3. After Partition 0’s batch is full and closed for the Sender Thread to send, the Sticky Partitioner picks a new random partition (e.g., Partition 2) as the next active partition.
4. This process repeats continuously.

flowchart TD
    subgraph Hashed["1. Hashed Key Partitioner (With Key)"]
        direction TB
        K1["Keyed Message 'User_A'"] --> H1["Murmur2 Hash"]
        K2["Keyed Message 'User_B'"] --> H2["Murmur2 Hash"]
        H1 -->|"Partition 0"| B0["Partition 0 Batch (User_A)"]
        H2 -->|"Partition 1"| B1["Partition 1 Batch (User_B)"]
    end

    subgraph RoundRobin["2. Round-Robin Partitioner (Without Key - Old)"]
        direction TB
        R1["Message 1"] --> P0["Partition 0 (1 KB Batch)"]
        R2["Message 2"] --> P1["Partition 1 (1 KB Batch)"]
        R3["Message 3"] --> P2["Partition 2 (1 KB Batch)"]
        NoteRR["Overhead: Batches never fill up, sent immediately due to timeout"]
    end

    subgraph Sticky["3. Sticky Partitioner (Without Key - Kafka 2.4+)"]
        direction TB
        S1["Message 1"] --> SP0["Partition 0 (Active Batch)"]
        S2["Message 2"] --> SP0
        S3["Message 3"] --> SP0
        SP0 -->|"Batch Full (16 KB)"| Send["Send Partition 0 Batch"]
        S4["Message 4 (Moves to New Partition)"] --> SP1["Partition 1 (New Active Batch)"]
    end

Sticky Partitioner Advantages #

• Higher Throughput: By maximizing batch filling to full capacity before sending, the payload per network request becomes far more efficient.
• Reduced Latency: Reduces the number of write requests (produce requests) the broker must handle, cutting broker CPU load.
• Even Distribution at Scale: Although temporarily sticking to one partition, data stays evenly distributed across all partitions within seconds because the partition target shifts periodically.

Data Distribution Problem: Key Skewness #

When we decide to use keyed messages, we must be aware of the risk of severe data distribution imbalance in the cluster, known as Key Skewness.

Why Does Key Skewness Happen? #

The Murmur2 hashing mechanism assumes key value variations are distributed randomly and evenly. However, business data reality is often different.

Imagine we have an orders topic with the key being customerId.

• Regular customers (e.g., individuals) only make 1-2 transactions per day.
• However, there’s one large corporate customer (e.g., a B2B/reseller account) making 1,000,000 transactions per day.
• Because all of that corporate customer’s transactions use the same key (e.g., key "CUST_B2B_MEGA"), based on the Murmur2 hash formula, all one million transactions go to the same partition (e.g., Partition 1).
• As a result, Partition 1 swells enormously (causing full disk consumption), Partition 1’s leader broker works extra hard (CPU spike), and the consumer assigned to read Partition 1 suffers severe processing bottlenecks (consumer lag), while consumers on other partitions sit idle.

Key Skewness Solution: The Key Salting Technique #

To solve the Key Skewness problem without losing the benefits of keyed message delivery, we can apply a technique called Key Salting.

What is Key Salting? #

Key Salting is the technique of adding a random value or sequential suffix (salt) to the end of our main key before sending it to Kafka. This forces the Murmur2 hashing algorithm to split that single key into several different key variations, spreading messages across several different partitions.

For example, we set a salt range from 1 to 5. The key "CUST_B2B_MEGA" is transformed into:

• "CUST_B2B_MEGA_1" -> goes to Partition 0
• "CUST_B2B_MEGA_2" -> goes to Partition 1
• "CUST_B2B_MEGA_3" -> goes to Partition 2
• …and so on.

Key Salting Implementation in Java #

Here’s an implementation comparison example without salting (anti-pattern) versus the correct salting implementation:

// ANTI-PATTERN: Sending a large key without salting, triggering partition imbalance (key skewness)
public class NaiveOrderProducer {
    public void sendOrders(KafkaProducer<String, String> producer, Order order) {
        // If customerId is a large B2B account, millions of records go to the same partition
        String key = order.getCustomerId(); 
        ProducerRecord<String, String> record = new ProducerRecord<>("orders-topic", key, order.toJson());
        producer.send(record);
    }
}

// CORRECT: Applying Key Salting to break delivery load evenly across several partitions
import java.util.concurrent.ThreadLocalRandom;

public class SaltingOrderProducer {
    private static final int SALT_RANGE = 5; // Splitting data across at most 5 different partitions

    public void sendOrders(KafkaProducer<String, String> producer, Order order) {
        String key = order.getCustomerId();
        
        // Check whether this customer is a giant B2B account needing salting
        if (order.isMegaCorporate()) {
            // ✓ Add a random salt at the end of the key to trigger a different binary hash
            int randomSalt = ThreadLocalRandom.current().nextInt(1, SALT_RANGE + 1);
            key = key + "_" + randomSalt; // Result: "CUST_B2B_MEGA_3"
        }
        
        // Send the record with the salted key
        ProducerRecord<String, String> record = new ProducerRecord<>("orders-topic", key, order.toJson());
        producer.send(record);
    }
}

Key Salting Consequences to Watch Out For: #

• Loss of Absolute Ordering: Because the key is split into several variations, message order between that corporate customer’s transactions is no longer globally guaranteed. B2B data processing is only guaranteed sequential within sub-partitions (for example, all data ending in _3 stays ordered on Partition 2).
• Deserialization Complexity: Our consumer applications must be aware of this salting technique and must strip the _salt suffix (e.g., _3) when processing data in downstream databases.

Murmur2 Algorithm Characteristics and Why It Was Chosen #

Why did Kafka choose the Murmur2 hash algorithm as the foundation of key-based partition division? There are several technical reasons behind this architectural decision:

1. Non-Cryptographic Hash: Cryptographic algorithms like MD5, SHA-1, or SHA-256 are designed for high-level security and resistance to collision attacks. However, these complex computations require enormous CPU power. In contrast, Murmur2 is a non-cryptographic algorithm fully optimized for fast hash table lookups. Its computation speed can reach several Gigabytes of data per second on a single CPU core. This prevents the serializer/partitioner from becoming a bottleneck on our main application thread.
2. Excellent Avalanche Effect: A crucial property of a good hash function is the avalanche effect. If we change just one bit of the key input (e.g., changing "User_A" to "User_B"), the output binary hash changes dramatically and randomly (nearly 50% of output bits fluctuate). This property ensures that keys with similar string naming patterns are still spread evenly across different partitions.
3. Low Collision Rate: Although very fast, Murmur2 has a very low key collision probability. The 32-bit binary output distribution it produces is highly uniform for various string and numeric data types.

The Impact of Adding Partitions on Key Compliance (Re-hashing Problem) #

Key-based data ordering guarantees in Apache Kafka have one crucial physical limitation: The partition count on the destination topic must remain constant.

If our data traffic volume surges sharply and we decide to increase the topic’s partition count (for example, scaling orders-topic from 3 partitions to 6 partitions), our modulo division formula changes instantly.

Re-hashing Problem Illustration #

Suppose the Murmur2 hash result of key "User_A" is the positive integer 10:

• Before Scaling (3 Partitions): $$\text{Target Partition} = 10 \pmod 3 = \text{Partition } 1$$ All old "User_A" messages are stored on Partition 1.
• After Scaling (6 Partitions): $$\text{Target Partition} = 10 \pmod 6 = \text{Partition } 4$$ New "User_A" messages sent after the partition increase are routed to Partition 4.

Bad Impact #

Because "User_A" messages are now split across two different physical partitions (Partition 1 for old data, and Partition 4 for new data), consumers assigned to read that topic deserialize and process data from both partitions concurrently and asynchronously. As a result, "User_A"’s chronological transaction order is instantly broken.

Here’s a comparison of the wrong scaling handling scenario (anti-pattern) versus the recommended solution:

// ANTI-PATTERN: Changing the partition count directly on an active transactional topic
// ✗ Ignoring the fact that adding partitions breaks existing keyed data write ordering
public class NaiveScaleUp {
    public void scaleTopicPartitions() {
        // Using AdminClient to suddenly increase orders-topic partitions in production
        // Triggers instant re-hashing problems on all active producers
    }
}

// CORRECT: Creating a new topic with the desired partitions and performing structured data migration
// ✓ Keeping order-topic data processing finished before switching to orders-v2
public class SafeScaleUp {
    // Standard Operating Procedure (SOP):
    // 1. Create a new 'orders-v2' topic with a larger partition count (e.g., 12 partitions).
    // 2. Deploy new consumer applications reading from 'orders-v2'.
    // 3. Point application producers to start writing to the new 'orders-v2' topic.
    // 4. Let old consumers finish draining the remaining queue in 'orders-v1'.
    // 5. Shut down old consumers and decommission 'orders-v1'.
}

Alternative Solution: Consistent Hashing Partitioner #

If we anticipate frequent partition count changes in the future, we can write a Custom Partitioner implementing the Consistent Hashing concept (like ring hashing using the Ketama algorithm). With consistent hashing, when partitions are added, only a small portion of keys are moved to new partitions, while the majority of other keys stay committed to their old partitions.

Selection Guide: When to Use Key vs No Key #

Choosing between using a key or not must align with our application’s functional and non-functional needs. Here’s a decision matrix as a reference:

Summary #

• Ordering Guarantee: Sending messages with a key ensures all data with the same key goes to the same partition, guaranteeing chronological order on the consumer side.
• Murmur2 Hashing: Key-based partition determination uses the Murmur2 hash algorithm deterministically as long as the topic partition count stays stable.
• Round-Robin vs Sticky: The Sticky Partitioner (Kafka 2.4+) revolutionized non-key message efficiency by concentrating batching on one partition until full, multiplying throughput compared to classic round-robin.
• Key Skewness: The broker workload imbalance risk from one dominant key draining memory and CPU on a single partition.
• Key Salting: The solution for breaking partition congestion by attaching random suffixes to dominant keys, spreading load proportionally across several partitions.

← Previous: Serialization Next: Acks, Retries, & Linger.ms →