Partition & Broker Sizing: Designing Partition Counts and Apache Kafka Cluster Scale #

When designing topic and cluster architectures in Apache Kafka, the two most fundamental questions we must answer are: “How many partitions do we need for this topic?” and “How many broker servers must we provide in the cluster?” Unfortunately, these decisions are often made without mature calculations. Some teams choose to create thousands of partitions for all topics assuming “the more the better,” while others leave the default partition count at one, making their applications unable to scale.

Errors in determining partition sizes and broker counts can be fatal later. Partition Bloat overburdens metadata coordination, slows recovery latency during broker crashes, increases JVM memory consumption, and exhausts operating system file descriptor limits. Conversely, partition shortages limit consumer parallelism capacities, trigger data queues (consumer lag), and limit utilization of servers we’ve rented.

In this guide, we’ll dissect correlations between partitions and parallelism, learn mathematical formulas for calculating ideal partition counts based on target throughput, understand partition bloat risks in the ZooKeeper vs KRaft eras, and calculate physical broker node scale needs for our production clusters.

The Relationship Between Partitions, Parallelism, and Scalability #

To understand the importance of partition size determination, we must recall basic Kafka architecture principles: Partitions are the smallest unit of scalability and parallelism in Kafka.

flowchart TD
    subgraph Partitions["payment.orders Topic with 3 Partitions"]
        direction TB
        P0["Partition 0"]
        P1["Partition 1"]
        P2["Partition 2"]
    end
    subgraph Consumers["Consumer Group"]
        direction TB
        CA["Consumer A (Actively processing data)"]
        CB["Consumer B (Actively processing data)"]
        CC["Consumer C (Actively processing data)"]
        CD["Consumer D (Idle - Doesn't get any partition)"]
    end
    P0 --> CA
    P1 --> CB
    P2 --> CC

On consumer sides, one partition can only be consumed by a maximum of one consumer member inside the same Consumer Group at one time. If we have a topic with 3 partitions, then run 4 consumer application instances, the 4th instance stays idle without receiving data. Thus, partition counts determine the upper limit of our consumer applications’ parallelism capabilities.

Formula for Calculating Ideal Partition Counts #

To avoid guesswork, we can scientifically determine per-topic partition counts using target throughput formulas.

The mathematical formula for calculating minimum partition counts is:

$$\text{Partition Count} = \max\left(\frac{T}{p}, \frac{T}{c}\right)$$

Where:

• $T$ (Target Throughput): The total read/write throughput the business wants to achieve on that topic (in MB/s units or messages per second).
• $p$ (Single Producer Throughput): The maximum throughput one single producer thread can send to one partition without obstacles (usually valued at $\approx 10 \text{ to } 20 \text{ MB/s}$ depending on encryption and compression schemes).
• $c$ (Single Consumer Throughput): The maximum throughput one single consumer instance can process from one partition. Consumers usually process data slower because they must do I/O operations (like writing to relational databases, calling external APIs, or doing complex memory processing). Consumer throughput ranges around $\approx 1 \text{ to } 5 \text{ MB/s}$.

Partition Calculation Case Example #

Suppose we want to build a topic for a payment log processing system:

• Target Throughput ($T$): The business needs the system to process peak data flows of $80 \text{ MB/s}$.
• Producer Performance ($p$): Based on tests, one Java producer can send data at an average of $20 \text{ MB/s}$.
• Consumer Performance ($c$): Our consumer application must write data to PostgreSQL databases, limiting consumer processing speeds to only $4 \text{ MB/s}$ per thread.

Let’s plug those numbers into the formula: $$\text{Partitions from the Producer Side} = \frac{80 \text{ MB/s}}{20 \text{ MB/s}} = 4 \text{ partitions}$$ $$\text{Partitions from the Consumer Side} = \frac{80 \text{ MB/s}}{4 \text{ MB/s}} = 20 \text{ partitions}$$ $$\text{Minimum Partition Count} = \max(4, 20) = 20 \text{ partitions}$$

In this scenario, we must create a minimum of 20 partitions for that topic. This count guarantees we can run up to 20 parallel consumer instances to keep up with producer data delivery rates of $80 \text{ MB/s}$ without triggering lag pileups.

Partition Determination Decision Flow Diagram (Decision Tree) #

Here’s a decision tree flow diagram guiding us to choose the right partition counts based on application functional constraints:

flowchart TD
    Start["Start Topic Design"] --> Q1{"Is message ordering absolutely required?"}
    
    Q1 -- "Yes, Whole Topic" --> SinglePart["Use 1 Partition<br/>(Parallelism limited to 1 consumer)"]
    Q1 -- "Yes, Based on Key" --> Q2["Identify the Message Key (e.g., user_id)"]
    Q1 -- "No (Free Ordering)" --> CalcMath["Calculate Partitions with the Formula:<br/>Max(T/p, T/c)"]
    
    Q2 --> CalcMath
    
    CalcMath --> DoubleCheck{"Is the calculation result > 100?"}
    
    DoubleCheck -- "Yes" --> Q3{"Are we using KRaft mode?"}
    DoubleCheck -- "No" --> RoundUp["Round up for future growth<br/>(e.g., multiples of the broker count)"]
    
    Q3 -- "Yes" --> ApplyKraft["Use that number safely"]
    Q3 -- "No (ZooKeeper)" --> LimitZK["Limit partitions, optimize consumer code<br/>so per-thread throughput rises"]
    
    RoundUp --> End["Apply the Partition Count"]
    ApplyKraft --> End
    LimitZK --> End

Capacity Testing: Measuring Real Producer & Consumer Speeds #

To get accurate $p$ (producer throughput) and $c$ (consumer throughput) values in our own network environments, we’re advised to use Apache Kafka’s built-in load testing tools.

1. Measuring Producer Performance with kafka-producer-perf-test.sh #

This command sends synthetic messages flows to single partitions to measure maximum write capacities:

kafka-producer-perf-test.sh --topic test-perf-topic \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092 \
    acks=1 \
    compression.type=zstd

Example Output:

1000000 records sent, 98212.1 records/sec (95.91 MB/sec), 15.2 ms avg latency, 450.0 ms max latency.

From the example above, our producer can send data up to $95 \text{ MB/s}$ per partition using Zstd compression.

2. Measuring Consumer Performance with kafka-consumer-perf-test.sh #

This command downloads data from target topics as fast as possible to measure optimal read speeds:

kafka-consumer-perf-test.sh --bootstrap-server localhost:9092 \
  --topic test-perf-topic \
  --messages 1000000 \
  --threads 1

Example Output:

start.time, end.time, data.consumed.in.MB, throughput.in.MB_sec
2026-06-08 16:10:00, 2026-06-08 16:10:15, 1024.0000, 68.2667

From this example, one Java consumer thread can process data at $68 \text{ MB/s}$ (without database processing loads). If our application code adds database I/O, this number drops drastically, e.g., to $5 \text{ MB/s}$, and this is the number we must enter as the $c$ variable in sizing formulas.

Dynamically Adding Partitions: Implications and Dangers #

One interesting Kafka feature is that we can dynamically add partition counts to already-running topics using the kafka-topics.sh --alter command. However, we must be very careful because this action has serious consequences for data logic integrity:

1. Key-Routing Breaks: By default, Kafka routes messages with keys using hashing algorithms: $$\text{Partition} = \text{hash}(\text{key}) \pmod{\text{Partition Count}}$$ If partition counts change mid-way (e.g., from 10 to 15), the modulo division formula above produces different destination partitions for the same keys. As a result, new messages with the user-123 key get sent to different partitions than old messages, so per-key message ordering guarantees break.
2. No Partition Scale-Down: Kafka doesn’t support partition count reductions (scaling down). Once a topic is configured with 50 partitions, we can’t lower it to 20. The only way is deleting that topic (losing data) or creating new topics with different names then migrating data.

Designing Internal Topic Sizes: __consumer_offsets #

Every time consumers commit offsets, Kafka writes those commit messages to an internal topic named __consumer_offsets. Because this topic serves all consumer groups in the cluster, its planning must be done carefully at the broker level (server.properties):

• Default Partition Count: Determined by the offsets.topic.num.partitions parameter (default $50$). This number is very ideal for most middle-class clusters. Don’t set it to minimal default values (like 1) in production so commit loads can spread evenly across several brokers.
• Replication Factor: Determined by offsets.topic.replication.factor (default $3$). In production, make sure this parameter is set to 3 to guarantee consumer offset commits keep running even if one coordination leader broker experiences failures.

Negative Impacts of Partition Bloat #

If partitions are the key to scalability, why don’t we create 1,000 partitions for every topic from the start? This partition excess phenomenon is known as Partition Bloat and brings real negative impacts to broker operational stability.

1. Metadata Recovery Delays (Failover Latency) #

Every partition on brokers is led by one leader broker and replicated to several follower brokers. If a broker holding 10,000 partitions crashes, cluster controllers must process leader election changes 10,000 times.

• This election process requires writing metadata status changes to KRaft logs or coordinating with ZooKeeper.
• The more partitions involved, the longer the recovery time clusters need. During this recovery period, some partitions stay offline and inaccessible to clients.

2. Operating System File Descriptor Leaks (Open Files Exhaustion) #

At the operating system (Linux) level, every data partition is represented by a physical directory on disks. Inside those directories, every log segment has at least three active files: .log data files, .index index files, and .timeindex time files.

• If a broker manages 5,000 partitions, that server minimally opens $5,000 \times 3 = 15,000$ active files.
• If the OS file descriptor limit (ulimit -n) is set too low (e.g., Linux defaults are only 1024), brokers immediately crash with Too many open files error messages.

3. Excessive JVM Heap Memory Consumption #

Every partition needs memory buffer allocations inside broker JVM heaps for managing replication threads, tracking offset positions, and managing metadata caches. Piling too many partitions on one broker JVM triggers very frequent Garbage Collection (GC) activities, enlarges GC pause durations (stop-the-world), and triggers Out of Memory (OOM) risks.

Partition Limits on ZooKeeper vs KRaft #

Safe partition count limits heavily depend on the cluster metadata coordination modes we use:

1. ZooKeeper-Based Clusters (Legacy) #

In ZooKeeper mode, all partition leadership status changes must be synchronously written to ZooKeeper nodes. Because ZooKeeper limits metadata write throughput, the practical safe limits for ZooKeeper-based clusters are:

• A maximum of 4,000 partitions per broker JVM.
• A maximum of 200,000 partitions for the entire cluster.

2. KRaft-Based Clusters (Apache Kafka 3.x / 4.x) #

KRaft mode removes ZooKeeper and moves metadata management directly into internal Kafka Raft consensus. Because KRaft consolidates metadata into one efficient distributed log file, scalability limits surge drastically:

• A maximum of 10,000 to 20,000 partitions per broker JVM (depending on RAM and CPU specifications).
• A maximum of 1,000,000+ partitions for the entire cluster.

Even though KRaft can accommodate massive partition counts, we’re still advised not to create partitions wastefully to maintain page cache memory usage efficiency.

Calculating Broker Count Scales in Clusters #

After determining partition counts, we must calculate how many physical broker nodes are needed to support those loads.

Broker scale calculation steps can be formulated through the following two boundary conditions:

Constraint 1: Based on Hardware Throughput Capacity #

We must divide total cluster data throughput by the safe throughput capacity one broker server unit can handle:

$$\text{Broker Count (Throughput)} = \lceil \frac{\text{Total Cluster Throughput (Inbound + Outbound)}}{\text{Maximum Throughput per Node (Hardware Limit)}} \rceil$$

Where:

• Total Cluster Throughput: The combination of all inbound data (producers + replication) and outbound data (consumers).
• Maximum Throughput per Node: The weakest performance limit between network card (NIC) bandwidth and disk read-write speeds on one server.

Constraint 2: Based on Failure Tolerance (High Availability) #

Kafka clusters must have enough minimum nodes to safely distribute data replicas. Broker counts must not be smaller than the largest replication factor we use on cluster topics:

$$\text{Broker Count} \ge \text{Replication Factor (RF)}$$

Broker Calculation Example: #

• Total Cluster Throughput: Estimated to reach $600 \text{ MB/s}$ during peak traffic (inbound + outbound).
• Server Specifications: Each broker server is equipped with $10 \text{ Gbps}$ network cards ($\approx 1,000 \text{ MB/s}$ practical) and SSD disk systems able to serve stable throughput up to $150 \text{ MB/s}$ for mixed read-write operations.
• Replication Factor (RF): Set to 3 for all important topics.

Let’s calculate the needed broker counts:

1. Based on Throughput (the biggest bottleneck is $150 \text{ MB/s}$ disk performance): $$\text{Broker Count} = \lceil \frac{600 \text{ MB/s}}{150 \text{ MB/s}} \rceil = 4 \text{ brokers}$$
2. Based on Replication Needs: $$\text{Broker Count} \ge 3$$

By comparing both conditions above, we must provide a minimum of 4 brokers in the cluster. To give a safe headroom reserve of $25%$ anticipating if one broker dies (N-1 redundancy), we’re advised to add 1 backup broker, so our total production cluster has 5 brokers.

Operational Compliance and Partition & Broker Sizing Audit Checklist #

Do the following audit steps on our cluster design plans to ensure partition and broker allocations meet operational stability criteria:

Summary #

• Partitions for Parallelism — Partition counts determine the maximum number of active consumer instances that can process data simultaneously inside one consumer group.
• Use Throughput Formulas — Calculate topic partition counts using the $\max\left(\frac{T}{p}, \frac{T}{c}\right)$ formula by comparing target throughput against producer and consumer thread speeds.
• Prevent Partition Bloat — Don’t randomly create excessive partitions to avoid slow metadata recovery, OS file descriptor exhaustion, and JVM memory pressure.
• Adjust Coordination Mode Limits — Limit partitions to a maximum of 4,000 per broker in ZooKeeper mode, and leverage KRaft mode scalability up to 10,000+ partitions per broker for giant-scale needs.

← Previous: Throughput Planning Next: Performance Tuning →