Partition Assignment Strategy #

When we launch a Consumer Group to process data from a topic in Apache Kafka, one of the most important decisions to make is how the topic’s partitions are divided among the active consumers. Kafka doesn’t do this division rigidly from the broker side. Instead, coordination is delegated to the client library using a component called the Partition Assignor. Through choosing the right partition assignment strategy, we can control load balancing fairness, minimize rebalance overhead, and determine how resilient our application is when consumer instances are dynamically added or removed. Letting default configuration run without deep analysis is often a time bomb triggering resource imbalance in production.

Why Is the Partition Assignment Strategy So Important? #

In enterprise-scale Kafka clusters, a Consumer Group often subscribes to several topics at once. Each topic has a different partition count characteristic. On the consumer side, we may have several application pods (for example, in Kubernetes) running in parallel.

If partition division is unfair, we’ll face situations where:

• One consumer is overwhelmed because it gets allocated partitions with very dense traffic volume.
• Another consumer in the same group sits idle because it only gets a few partitions or none at all.
• The rebalance process takes very long because the broker must stop the entire data flow just to move partition ownership that actually didn’t need moving.

The configuration property responsible for determining this behavior is partition.assignment.strategy. The Kafka consumer client lets us set one or several strategies at once as a comma-separated priority list.

Dissecting the 4 Main Partition Assignor Strategies #

Let’s deeply dissect the four built-in strategies most commonly used in Apache Kafka, along with their mathematical characteristics and visualizations.

1. Range Assignor (org.apache.kafka.clients.consumer.RangeAssignor) #

This is the default strategy used by the Kafka consumer client if we don’t specify any configuration.

How Range Assignor Works #

Range Assignor works independently for each topic. For every subscribed topic, Range Assignor:

1. Sorts all available partitions numerically (e.g., 0, 1, 2, …).
2. Sorts all active consumers in the group lexicographically (e.g., C1, C2).
3. Divides the topic partition count by the consumer count to determine how many partitions each consumer gets. If there’s a division remainder, the first consumers get the extra partitions.

Mathematically, for a single topic with $n$ partitions and $m$ consumers:

• Each consumer gets $n / m$ partitions.
• The first $n % m$ consumers get one additional partition.

Range Assignor Weakness: Partition Imbalance #

The big problem happens if consumers subscribe to many topics simultaneously. Because the division calculation is done in isolation per topic, the division remainder is always charged to the first consumer (C1).

Suppose we have 2 topics (Topic-A and Topic-B), each with 3 partitions (P0, P1, P2). We have 2 consumers (C1 and C2) in one group.

• Topic-A Calculation: 3 partitions / 2 consumers = 1.5. C1 gets 2 partitions (P0, P1), C2 gets 1 partition (P2).
• Topic-B Calculation: 3 partitions / 2 consumers = 1.5. C1 gets 2 partitions (P0, P1), C2 gets 1 partition (P2).
• Final Allocation Result:
  • C1 holds 4 partitions: Topic-A (P0, P1) and Topic-B (P0, P1).
  • C2 only holds 2 partitions: Topic-A (P2) and Topic-B (P2).

If we increase the number of topics to dozens, C1 carries twice as heavy a workload compared to C2. This can trigger high memory consumption on the C1 pod and end in an OutOfMemory (OOM) crash.

2. Round Robin Assignor (org.apache.kafka.clients.consumer.RoundRobinAssignor) #

To overcome the load imbalance produced by Range Assignor, we can switch to the Round Robin strategy.

How Round Robin Assignor Works #

Unlike Range Assignor, Round Robin puts all partitions from all subscribed topics into one single list, then distributes them rotating one by one to each consumer.

Let’s use the same example: 2 topics (Topic-A and Topic-B), each with 3 partitions. Our group members are C1 and C2.

1. All partitions are sorted collectively: Topic-A-P0, Topic-A-P1, Topic-A-P2, Topic-B-P0, Topic-B-P1, Topic-B-P2.
2. Circular distribution runs:
   • Topic-A-P0 -> C1
   • Topic-A-P1 -> C2
   • Topic-A-P2 -> C1
   • Topic-B-P0 -> C2
   • Topic-B-P1 -> C1
   • Topic-B-P2 -> C2
3. Final Allocation Result:
   • C1 holds 3 partitions: Topic-A-P0, Topic-A-P2, Topic-B-P1.
   • C2 holds 3 partitions: Topic-A-P1, Topic-B-P0, Topic-B-P2.

Round Robin Weakness: Rebalance Overhead #

Although the distribution is very even, Round Robin has a big weakness during rebalance processes. Because it doesn’t track previous allocations, small changes in group membership can trigger massive, random partition ownership moves. This forces consumers to close old socket connections, clear local caches, and open new connections to the newly allocated partitions’ leader brokers.

3. Sticky Assignor (org.apache.kafka.clients.consumer.StickyAssignor) #

The Sticky Assignor strategy is designed to solve the dilemma between Range and Round Robin: providing even load distribution while maintaining partition allocation stability during rebalances.

The Sticky Assignor’s main goals are:

1. Primary Balance: Distribute partitions as fairly as possible among active consumers (like Round Robin).
2. Maximum Stickiness: When a rebalance happens, ensure partitions previously held by consumers aren’t moved to other consumers, unless absolutely necessary to maintain balance.

Stickiness Case Example #

Imagine we have 3 consumers (C1, C2, C3) reading 3 partitions (P0, P1, P2). Initial state: C1->P0, C2->P1, C3->P2. If C3 crashes:

• Round Robin: It might shuffle the allocation to C1->P1, P2 and C2->P0. Here, C1 loses P0 and C2 loses P1.
• Sticky Assignor: Guarantees previous allocations stay sticky: C1 keeps holding P0 and C2 keeps holding P1. The only partition reallocated is P2, belonging to the dead C3. P2 is given to C1 or C2.

By maintaining this stickiness, we save significant compute and network bandwidth resources because most consumers can keep processing data from their old partitions without connection interruptions.

4. Cooperative Sticky Assignor (org.apache.kafka.clients.consumer.CooperativeStickyAssignor) #

Introduced since Apache Kafka 2.4, this is the peak of partition assignment strategy evolution in Kafka.

The Eager Rebalance Problem in Old Libraries #

Before version 2.4, all rebalances in Kafka used the Eager Rebalance protocol. This protocol applies a Stop-The-World pattern. When a rebalance happens, all consumers in the group must release all their partitions simultaneously, stop processing data, rejoin the group, and wait for the new partition allocation calculation to finish. This triggers data processing pauses (lag spikes) that are very disruptive for real-time systems.

Solution: Cooperative Rebalance (Incremental) #

CooperativeStickyAssignor uses the Cooperative Rebalance protocol. Instead of resetting the whole world, this strategy processes changes incrementally:

1. When a new consumer joins, the group leader calculates which partitions need to move to achieve balance.
2. Consumers not affected by those partition moves are allowed to keep reading and processing data without stopping.
3. Only the partitions that will move are revoked from their old owners in an orderly fashion, then handed to the new owner in the next small rebalance cycle.

This mechanism eliminates the stop-the-world processing pause entirely for most group members, making our data pipeline much smoother.

flowchart TD
    subgraph Eager["Eager Protocol (Range / RoundRobin / Sticky)"]
        direction TB
        E1["Rebalance Starts"] --> E2["ALL Consumers Release ALL Partitions"]
        E2 --> E3["Processing Stops Completely (Stop-The-World)"]
        E3 --> E4["New Allocation Calculation Done"]
        E4 --> E5["Consumers Start Reading Again"]
    end

    subgraph Cooperative["Cooperative Protocol (Cooperative Sticky)"]
        direction TB
        C1["Rebalance Starts"] --> C2["Calculate Which Partitions Need to Move"]
        C2 --> C3["Only Revoke Partitions Changing Owners"]
        C3 --> C4["Other Consumers Keep Processing Without Pauses"]
        C4 --> C5["Hand Selected Partitions to New Owners"]
    end

    style Eager stroke:#c62828,stroke-width:2px
    style Cooperative stroke:#2e7d32,stroke-width:2px

Partition Assignor Strategy Comparison Table #

The table below summarizes the comparative differences of the four partition assignment strategies:

Mermaid Flowchart: Choosing the Right Partition Assignor #

The decision flow chart below can help us determine which assignor strategy is most suitable for our system needs:

flowchart TD
    Start["Start Assignor Strategy Evaluation"] --> Q1{"Does the consumer only subscribe to 1 Topic?"}
    
    Q1 -- "Yes" --> Q2{"Are you using Kafka version 2.4 or newer?"}
    Q1 -- "No" --> Q3{"Is the data traffic very sensitive to rebalance pauses?"}
    
    Q2 -- "Yes" --> ChooseCooperative["Recommendation: CooperativeStickyAssignor"]
    Q2 -- "No" --> ChooseRange["Use Default: RangeAssignor"]
    
    Q3 -- "Yes" --> ChooseCooperative
    Q3 -- "No" --> Q4{"Is even load distribution the top priority?"}
    
    Q4 -- "Yes" --> ChooseRoundRobin["Use: RoundRobinAssignor"]
    Q4 -- "No" --> ChooseSticky["Use: StickyAssignor"]

    style ChooseCooperative stroke:#2e7d32,stroke-width:2px
    style ChooseRange stroke:#455a64,stroke-width:2px
    style ChooseRoundRobin stroke:#f57c00,stroke-width:2px
    style ChooseSticky stroke:#0288d1,stroke-width:2px

Implementation Code and Assignor Configuration in Clients #

Let’s look at the configuration code implementation comparison between the wrong default handling and the optimal handling using the Java SDK library.

Java SDK Anti-Pattern: Using Default RangeAssignor on Multi-Topic #

The code below shows a common mistake where developer teams let the default allocation run when the application must consume many topics simultaneously.

// ANTI-PATTERN: Letting RangeAssignor manage multi-topic subscribers
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "multi-topic-consumer-group");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

// ✗ DON'T: Let the default strategy (RangeAssignor) stay active if our application
// subscribes to many topics with uneven partition counts.
// This piles excess partition load on the first consumer in the group.
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

// Subscribing to 5 topics each with 3 partitions
consumer.subscribe(Arrays.asList(
    "order-events", 
    "payment-events", 
    "shipping-events", 
    "inventory-events", 
    "notification-events"
));

Java SDK Solution: Applying CooperativeStickyAssignor #

To switch to an advanced processing strategy that’s pause-tolerant and fair in load division, we must declare the CooperativeStickyAssignor class in the consumer configuration parameter.

// CORRECT: Enabling CooperativeStickyAssignor for maximum efficiency
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "multi-topic-consumer-group");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

// ✓ Setting the partition assignment strategy to CooperativeStickyAssignor
// This distributes partitions evenly across topics and prevents the stop-the-world effect
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, 
    Collections.singletonList(CooperativeStickyAssignor.class.getName())
);

props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

// Subscribing to many topics safely
consumer.subscribe(Arrays.asList(
    "order-events", 
    "payment-events", 
    "shipping-events", 
    "inventory-events", 
    "notification-events"
));

Multi-Strategy Fallback Configuration #

Interestingly, Kafka allows us to define several assignor classes at once. This step is useful when gradually migrating a consumer group from one assignor to another without shutting down the application cluster.

// CORRECT: Providing a strategy fallback for gradual migration scenarios (Rolling Upgrade)
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, Arrays.asList(
    CooperativeStickyAssignor.class.getName(),
    StickyAssignor.class.getName()
));

• How It Works: During a rolling upgrade migration, new consumers starting with the configuration above negotiate with old consumers. Because both have a match on StickyAssignor (as the second choice for new consumers), the consumer group temporarily uses StickyAssignor until all pods finish upgrading to the new configuration fully supporting CooperativeSticky.

Summary #

• Partition Assignor — The client library responsible for calculating and distributing topic partition allocations among active consumers in a group.
• Range Assignor — The default strategy dividing partitions per topic sequentially. Risks triggering partition imbalance if subscribing to many topics at once.
• Round Robin Assignor — Distributes all partitions from all topics evenly in a circular fashion. However, triggers high rebalance overhead because it’s prone to randomly moving partition allocations.
• Sticky Assignor — Balances partition division fairly while striving to keep old partition ownership stuck to the same consumers when rebalances happen.
• Cooperative Sticky Assignor — The modern strategy eliminating global processing pauses (stop-the-world) using the incremental Cooperative Rebalance protocol.
• partition.assignment.strategy — The key configuration property used on the consumer side to determine or change the partition division tactic in use.
• Rolling Upgrade Safe — Defining a comma-separated assignor strategy list allows the cluster to automatically fall back during production application update transitions.

← Previous: Consumer Group Next: Rebalance Process →