Partition Strategy #

In Apache Kafka’s distributed storage architecture, Partition is the smallest unit determining physical scalability, load separation, and system parallelism. Kafka splits topics into multiple partitions so data can be spread across several broker servers simultaneously. However, determining the ideal number of partitions and how to distribute messages across those partitions is a critical architectural design decision.

Mistakes in partition allocation strategy can cause server load imbalance (Key Skewness), data queue blockage (Head-of-Line blocking), and drastic throughput degradation. Through this article, we’ll formulate the partition count calculation formula based on producer and consumer target throughput capacity, dissect how the built-in partitioners work (Sticky Partitioner vs Hashed Key Partitioner), build a Custom Partitioner implementation for critical traffic isolation, and learn hot partition mitigation techniques through the Key Salting method.

Formula for Calculating the Ideal Partition Count #

A common myth among developers is “the more partitions, the better Kafka’s performance”. This isn’t entirely true. Setting an excessive partition count (for example, immediately creating 500 partitions for a small topic) triggers high overhead load for the cluster:

  • File Handle Limits: Every partition is represented by a physical directory containing log and index files. Thousands of partitions mean thousands of open file descriptors burdening the broker OS.
  • Controller Overhead: When a broker fails, the Controller must lead the leader election process for all partitions that lost their leader. The more partitions, the longer the cluster recovery time.
  • Client Buffer Memory: Kafka producers allocate buffer memory (Record Accumulator) per partition. The more partitions, the more exponentially our producer client application’s RAM consumption swells.

To calculate the partition count scientifically, we must use the following target throughput formula:

$$P = \max \left( \frac{T_P}{t_p}, \frac{T_C}{t_c} \right)$$

Where:

  • $P$ — The ideal partition count needed.
  • $T_P$ — The desired total producer write throughput target (in MB/second).
  • $t_p$ — The maximum write speed one producer can achieve on a single partition (usually around 10 MB/second, depending on network and hardware).
  • $T_C$ — The desired total consumer read throughput target (in MB/second).
  • $t_c$ — The maximum business processing speed one consumer can achieve on a single partition (usually around 2 MB/second to 5 MB/second, heavily dependent on the application’s external database performance).

Example Case Scenario: #

We want to design a payment transaction processing system with these specs:

  • Target incoming data volume ($T_P$) = 50 MB/second.
  • Target outgoing data processing volume ($T_C$) = 50 MB/second.
  • One partition write speed ($t_p$) = 10 MB/second.
  • One consumer read and processing speed ($t_c$) = 2.5 MB/second (because consumers must query an internal MySQL database).

Then the ideal partition calculation is:

$$P = \max \left( \frac{50}{10}, \frac{50}{2.5} \right) = \max (5, 20) = 20 \text{ partitions}$$

This tuning guarantees we have enough partitions (20 partitions) to run up to 20 active consumers in parallel within one Consumer Group to chase the outgoing data processing throughput target. Always add a reserve tolerance (headroom) of 20% to 30% to the final calculated value to anticipate future load spikes.

Dissecting the Built-in Partitioner Mechanism #

When a producer sends data to Kafka, the Partitioner module routes messages to the destination partition number. This route division mechanism splits into two scenarios depending on whether the message has a Key:

1. Hashed Key Partitioner (Messages Using Keys) #

When a producer sends a message with a key (key is not null), Kafka uses the built-in Murmur2 hashing algorithm to produce an integer representation of the key, then performs a modulo operation against the topic’s total active partition count:

$$\text{Partition ID} = \text{abs}(\text{Murmur2}(\text{Key})) \pmod{\text{numPartitions}}$$

  • Route Consistency Guarantee: The main advantage of this method is guaranteeing that all messages with the same key always end up on the same physical partition. This is absolutely required for chronological message ordering guarantees.
  • Danger of Adding Partitions: If we increase the topic’s partition count mid-stream (for example, from 10 partitions to 15 partitions), the numPartitions value in the modulo formula changes. As a result, new messages with the same key get routed to a different partition number, destroying our historical message ordering integrity. Never add partitions to an active topic whose data relies on key ordering without a data migration plan.

2. Sticky Partitioner (Messages Without Keys) #

Before version 2.4, Kafka used the classic Round-Robin algorithm for keyless data. In the Round-Robin model, the producer divides messages one by one sequentially to each partition. This creates a big problem: the Accumulator memory batch for each partition never fills efficiently, triggering thousands of tiny TCP packet sends. This wastes CPU cycles and network bandwidth.

As a solution, Kafka introduced the Sticky Partitioner in version 2.4:

  • How It Works: The Sticky Partitioner locks one random destination partition and routes all keyless messages to that partition until the send batch memory (Record Accumulator) is full (reaching batch.size or hitting the linger.ms timeout). After the batch is sent to the broker, the partitioner picks the next partition at random to collect a new batch.
  • Advantage: This method multiplies producer network performance by cutting TCP overhead. By grouping messages into one partition at a time, batch data compression capacity works at maximum, yielding much higher throughput and minimal CPU latency compared to Round-Robin.

Partitioner Flow Comparison Diagram #

Let’s compare how messages are distributed from a producer to broker partitions using the asynchronous Sticky Partitioner vs Hashed Key Partitioner methods:

flowchart TD
    subgraph InputEvents["Client Event Stream"]
        E1["Msg 1 (Key: 'UserA')"]
        E2["Msg 2 (Key: null)"]
        E3["Msg 3 (Key: 'UserA')"]
        E4["Msg 4 (Key: null)"]
    end

    subgraph PartitionerEngine["Partitioner Engine"]
        direction TB
        KeyCheck{"Does the Event Have a Key?"}
        KeyCheck -- "Yes" --> HashCalc["Murmur2(Key) % Partitions"]
        KeyCheck -- "No" --> StickyCalc["Sticky Batching (Fill One Partition First)"]
    end

    subgraph BrokerPartitions["Target Broker Partitions"]
        direction LR
        P0[("Partition 0")]
        P1[("Partition 1")]
    end

    E1 --> KeyCheck
    E2 --> KeyCheck
    E3 --> KeyCheck
    E4 --> KeyCheck

    HashCalc -->|"UserA always goes to P0"| P0
    StickyCalc -->|"Batch 1 (Msg 2 & 4) goes to P1"| P1

    style P0 fill:#ddffdd,stroke:#88ff88
    style P1 fill:#ddffdd,stroke:#88ff88

Key Skewness Danger (Hot Partitions) and Its Mitigation #

Although using the Hashed Key Partitioner sounds ideal for preserving data ordering, this method is prone to the Key Skewness (load imbalance) phenomenon, often called Hot Partitions.

Why Do Hot Partitions Happen? #

Load imbalance happens when key distribution in our business traffic isn’t even.

  • For example, we partition transactions by the country_code key (ID = Indonesia, SG = Singapore, US = United States). Because 95% of our business transactions happen in Indonesia, the partition holding the ID hash is extremely burdened (CPU and broker disk space full), while the other partitions for SG and US sit completely empty.
  • The broker storing the hot partition suffers performance degradation, causing slowdowns for the entire cluster.

Imbalance Impact on the Consumer Side #

The problem doesn’t stop at the broker side. In a Consumer Group, each consumer is allocated exclusive ownership of specific partitions.

  • If Key Skewness happens, the consumer assigned the hot partition (the Indonesia Partition) is extremely burdened. That consumer must process 95% of total business data traffic, triggering severe consumer lag on that partition.
  • Meanwhile, other consumers holding the Singapore and US partitions sit idle, wasting our server container compute resources. This negates the load balancing benefits of the Consumer Group feature.

Mitigation Technique: Key Salting #

To break this load imbalance, we can apply the Key Salting technique. We add a random number periodically at the end of the key to distribute data across several different partitions.

  • Salting Formula: SaltedKey = Key + "_" + random(1, S) where $S$ is the salt range.
  • Application Example: If our original key is ID and $S = 3$, the randomly generated keys become ID_1, ID_2, or ID_3. Indonesian transaction data now spreads evenly across 3 different partitions, reducing broker load and balancing consumer work allocation.
  • Trade-off: We lose the global data ordering guarantee for all Indonesian transactions. However, we still get local ordering guarantees within each salted partition (for example, ID_1 data stays ordered in its own partition).

Anti-pattern vs Solution: Traffic Isolation with a Custom Partitioner #

One of the biggest architectural mistakes is mixing high-priority message traffic (like VIP premium customer payment transactions) with ordinary high-volume messages (like regular application analytics metrics) in the same partitions without isolation.

Consequences: If a partition experiences a long queue from slow-processing analytics data (consumer lag), VIP customer transactions get stuck in the same queue too (Head-of-Line blocking). This damages our business customer satisfaction.

Solution: Isolated Custom Partitioner #

We can write a programmatic Custom Partitioner on the producer side to route messages to dedicated partitions based on business criteria (for example, if message metadata indicates a VIP account, route to a dedicated priority partition).

Let’s look at the implementation difference between the vulnerable default routing code and the isolated custom routing using Java:

// =========================================================================
// ANTI-PATTERN: Relying on the Default Partitioner for All Scenarios
// Important VIP data gets mixed into the same partitions as non-VIP data.
// =========================================================================
public void sendUntunedData(KafkaProducer<String, String> producer, String customerId, String payload) {
    // ✗ DON'T: Send VIP data without a dedicated route. The message gets hashed
    // by default and is vulnerable to getting stuck in lag on the same partitions as ordinary data.
    ProducerRecord<String, String> record = new ProducerRecord<>("transactions", customerId, payload);
    producer.send(record);
}

// =========================================================================
// THE CORRECT SOLUTION: Isolated Custom Partitioner Implementation
// The routing logic separates the VIP partition (Partition 0) from ordinary partitions (Partition 1 and up).
// =========================================================================
import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.Cluster;
import java.util.Map;
import java.util.Properties;

public class VipPartitionSelector implements Partitioner {
    private static final int VIP_PARTITION_ID = 0; // Partition 0 dedicated to VIP

    @Override
    public void configure(Map<String, ?> configs) {
        // Additional configuration if needed
    }

    @Override
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        int numPartitions = cluster.partitionsForTopic(topic).size();
        String customerKey = (String) key;

        // ✓ CORRECT: Isolate VIP customer data to a dedicated partition
        if (customerKey != null && customerKey.startsWith("VIP_")) {
            // Always route VIP customers to Partition 0
            return VIP_PARTITION_ID;
        }

        // For regular customers, distribute hashed to partitions 1 and up
        int regularPartitionsCount = numPartitions - 1;
        if (regularPartitionsCount <= 0) {
            return 0; // Fallback if the total partition count is only 1
        }
        
        // Modulo hashing for the remaining regular partitions (1, 2, ..., numPartitions-1)
        int hashedPartition = (Math.abs(org.apache.kafka.common.utils.Utils.murmur2(keyBytes)) % regularPartitionsCount) + 1;
        return hashedPartition;
    }

    @Override
    public void close() {
        // Resource cleanup
    }
    
    // Demonstrating producer property integration
    public static Properties getProducerProperties(String bootstrapServers) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        
        // ✓ CORRECT: Register the custom partitioner into the producer configuration
        props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, VipPartitionSelector.class.getName());
        return props;
    }
}

By attaching this custom VipPartitionSelector class to our producer properties, we guarantee smooth VIP customer transaction data writes unaffected by regular consumer data processing slowdowns.

Summary #

  • Partition Scalability — Partitions are the smallest unit determining parallelism in Kafka. Determine partition capacity based on write and processing target throughput.
  • Partition Formula — The partition count is calculated using the mathematical formula: $P = \max(T_P / t_p, T_C / t_c)$ plus a 20% reserve tolerance headroom.
  • Hashed Key Partitioner — The Murmur2 modulo algorithm guaranteeing route consistency for keyed messages to the same partition for chronological data ordering.
  • Sticky Partitioner — A high-performance algorithm for keyless (null key) messages that groups messages in one partition to maximize network batching.
  • Key Skewness — The broker load imbalance (hot partition) phenomenon caused by unevenly distributed message keys in business traffic.
  • Consumer Impact — Key skewness triggers severe lag in one consumer group member while other consumers sit idle, negating workload division efficiency.
  • Key Salting — The technique of adding random strings/numbers at the end of message keys to break hot data across several partitions evenly.
  • Custom Partitioner — Writing a custom routing class programmatically to isolate high-priority (VIP) data streams from regular data Head-of-Line blocking.

← Previous: Topic Design Next: Ordering Guarantee →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact