Throughput Planning: A Guide to Apache Kafka Cluster Capacity Planning #

When building distributed data infrastructure using Apache Kafka, one of the most fatal mistakes we often make is ignoring capacity planning from the start. Many operations teams directly install Kafka using instinctive or random server specifications, then are shocked when their clusters collapse during traffic spikes, run out of network sockets, or experience performance degradation from too-small page cache memory.

Kafka is designed to handle massive throughput loads by maximizing hardware efficiency. However, this efficiency can only be achieved if we carefully synchronize business data traffic needs with network bandwidth allocations, disk I/O capacity, operating system page cache memory sizes, and CPU computing power. Mature throughput planning helps us avoid cost waste from over-provisioning while preventing production failure disasters from under-provisioning.

In this guide, we’ll discuss structured Kafka cluster capacity calculation methodologies. We’ll learn mathematical formulas for calculating network bandwidth needs, design disk storage capacity and IOPS estimates, calculate optimal RAM allocations for OS page cache systems, and identify factors triggering broker CPU loads.

Kafka Capacity Planning Introduction #

Capacity planning in Kafka is based on the principle that brokers act as high-speed pipeline channels moving data from producers to consumers as efficiently as possible. Kafka heavily relies on linear storage architectures and Zero-Copy data transfers. Therefore, the main bottleneck in Kafka is usually not CPU processing power, but Network Card Interface bandwidth and Disk read-write speeds (Disk I/O).

To accurately calculate cluster capacities, we must first collect key metrics from our business needs:

1. Write Throughput: How many megabytes of data are produced per second during busy times?
2. Replication Factor: How many data copies do we want to store for failure tolerance?
3. Read Throughput: How many independent consumer groups will read that data simultaneously?
4. Retention Period: How long must data be stored before brokers delete it?

Measuring Network Throughput Needs (Bandwidth) #

Networks are the first gate of our Kafka cluster. All incoming and outgoing messages must pass through Network Interface Cards (NICs) on every broker. Failing to calculate network bandwidth causes queue congestion on network processor threads and triggers client-side timeouts.

1. Inbound Network Bandwidth Calculations #

Data entering our cluster consists of data sent by producers plus inter-broker data replication traffic (replication traffic).

The formula for measuring total inbound traffic on clusters is:

$$\text{Total Inbound Bandwidth} = \text{Producer Data Rate (MB/s)} \times \text{Replication Factor (RF)} \times (1 + \text{Metadata Overhead})$$

Where:

• Producer Data Rate: The net data volume produced by our applications per second.
• Replication Factor: The partition copy count. If RF is set to 3, every incoming message must be duplicated 2 more times to follower brokers.
• Metadata Overhead: Additional allocations for message headers, offsets, and protocol handshakes (we’re advised to set a safe buffer of $10%$ or $0.1$).

2. Outbound Network Bandwidth Calculations #

Data leaving our cluster is dominated by consumer data pulls plus internal replication activities (where leaders send data to followers). However, because replication traffic is already counted in the inbound portion of follower brokers, on the outbound side we focus on active consumer group counts.

The formula for measuring total outbound traffic on clusters is:

$$\text{Total Outbound Bandwidth} = (\text{Producer Data Rate} \times \text{Consumer Group Count}) + \text{Replication Traffic}$$

Where:

• Consumer Group Count: How many unique applications read those topics simultaneously.
• Replication Traffic: Valued at $\text{Producer Data Rate} \times (\text{RF} - 1)$ because leaders send data copies to follower brokers.

Bandwidth Calculation Case Study #

Suppose we have the following production scenario:

• Producer Data Rate: $20 \text{ MB/s}$
• Replication Factor (RF): $3$
• Consumer Group Count: $4$ independent consumer groups.
• Metadata Overhead: Set to $10%$.

Let’s calculate total network bandwidth needs:

Inbound Bandwidth: $$\text{Total Inbound} = 20 \text{ MB/s} \times 3 \times 1.1 = 66 \text{ MB/s}$$

Outbound Bandwidth: $$\text{Replication Traffic} = 20 \text{ MB/s} \times (3 - 1) = 40 \text{ MB/s}$$ $$\text{Consumer Traffic} = 20 \text{ MB/s} \times 4 = 80 \text{ MB/s}$$ $$\text{Total Outbound} = 80 \text{ MB/s} + 40 \text{ MB/s} = 120 \text{ MB/s}$$

Total Cluster Network Load: $$\text{Total Bandwidth} = 66 \text{ MB/s} \text{ (In)} + 120 \text{ MB/s} \text{ (Out)} = 186 \text{ MB/s}$$

If we divide this load evenly into a cluster with 3 brokers, the average load per broker is:

• Average Inbound per Broker: $22 \text{ MB/s}$ (Equivalent to $\approx 176 \text{ Mbps}$)
• Average Outbound per Broker: $40 \text{ MB/s}$ (Equivalent to $\approx 320 \text{ Mbps}$)

From these numbers, a $1 \text{ Gbps}$ network card per server is very sufficient, with the note that there are no spikes exceeding 3x the average. However, if producer rates rise to $100 \text{ MB/s}$, we must switch to $10 \text{ Gbps}$ network cards to avoid saturation.

Cluster Capacity Component Mapping Diagram #

Here’s a relationship diagram showing how business variables are mapped to physical resource allocations (Network, CPU, Memory, Disk Space, Disk I/O) on our Kafka cluster:

flowchart TD
    subgraph BusinessMetrics["Business Needs"]
        RateProd["Producer Data Rate (MB/s)"]
        RF["Replication Factor (RF)"]
        NumGroups["Consumer Group Count"]
        RetHours["Retention Time (Hours/Days)"]
    end

    subgraph HardwareResource["Broker Physical Resource Allocations"]
        NetBandwidth["Network: Interface Bandwidth (NIC)"]
        DiskSpace["Storage: Total Disk Capacity"]
        DiskIO["Disk I/O: IOPS & Sequential Throughput"]
        OSCache["Memory: OS RAM Page Cache"]
        CPUPower["Compute: CPU Cores (SSL/Compression)"]
    end

    RateProd --> NetBandwidth
    RF --> NetBandwidth
    NumGroups --> NetBandwidth

    RateProd --> DiskSpace
    RF --> DiskSpace
    RetHours --> DiskSpace

    RateProd --> DiskIO
    RF --> DiskIO
    NumGroups --> DiskIO

    RateProd --> OSCache
    RF --> OSCache
    
    RateProd --> CPUPower

Storage Capacity Planning (Disk Storage & IOPS) #

After designing networks, we must plan data storage media. Disk planning covers two dimensions: Disk Space for holding data retention, and Disk I/O performance (IOPS) for serving write and read speeds.

1. Calculating Total Disk Capacity (Disk Space) #

We must make sure servers have enough storage space to hold historical data according to retention times before automatic deletion.

The net disk capacity calculation formula needed by clusters is:

$$\text{Total Disk Capacity} = \text{Producer Data Rate (MB/s)} \times 86400 \times \text{Retention (Days)} \times \text{RF} \times (1 + \text{Buffer Space})$$

Where:

• 86400: The number of seconds in one day.
• Buffer Space: Additional allocations for anticipating dirty segment cleanup delays, internal indexes, and safe operating system free space (recommended at a minimum of $20%$ or $0.2$).

Disk Space Calculation Example: #

Using the previous scenario (Producer $20 \text{ MB/s}$, $\text{RF} = 3$) with a 7-Day Retention target and $20%$ buffer: $$\text{Disk Capacity} = 20 \text{ MB/s} \times 86400 \times 7 \times 3 \times 1.2 = 43,545,600 \text{ MB} \approx 43.5 \text{ TB}$$

If our cluster consists of 3 brokers, then each broker must prepare storage space of: $$\frac{43.5 \text{ TB}}{3} = 14.5 \text{ TB per Broker}$$

We can fulfill this capacity using JBOD (Just a Bunch of Disks) configurations containing several local disks (e.g., 4 x 4 TB disks per server).

2. Choosing Storage Types: HDD vs SSD #

Disk type choices directly impact read-write latency and throughput:

• HDD (Hard Disk Drive): Very efficient for sequential writes which are the basic pattern of Kafka log segment writing. HDDs offer large capacities at cheap prices. However, if consumers read old (cold data) data from disks, HDD read heads move randomly (random reads), triggering drastic I/O performance degradation (disk thrashing).
• SSD (Solid State Drive / NVMe): Offers very high random IOPS. Highly recommended if our cluster serves latency-sensitive real-time applications, or if we anticipate consumers frequently reading historical data randomly from disks.

Memory Capacity Planning (OS Page Cache) #

One of the unique aspects of Apache Kafka architecture is that it doesn’t store message data in JVM heap memory. Broker JVMs are only advised to use small heaps (around 4 GB to 6 GB) for managing metadata structures and socket connections. The rest, Kafka fully hands over log segment data caching management to the Operating System Page Cache (Free RAM).

When producers write messages, data is actually written to RAM page caches first. The OS then flushes these page caches to disks asynchronously. If consumers request just-produced data, Kafka reads that data directly from RAM page caches without touching physical disks at all (Zero-Copy reads). This scenario is called a Page Cache Hit.

flowchart TD
    Producer["PRODUCER"] --> Cache["OS PAGE CACHE (Main Memory / RAM)"]
    Cache -- "Zero-Copy" --> Consumer["CONSUMER"]
    Cache -- "Asynchronous Flush" --> Disk["PHYSICAL DISK"]
    Disk -- "Only read if data is cold / Cache Miss" --> Consumer

RAM Page Cache Need Estimation Formula #

To make our clusters operate with minimal latency, we must ensure all currently active written topic partitions can be fully accommodated in RAM page caches.

The minimal rule of thumb for sizing page cache memory is storing data produced during the last 1 to 2 hours in RAM:

$$\text{RAM Page Cache Needs} = \text{Producer Data Rate (MB/s)} \times 3600 \text{ seconds} \times \text{Replication Factor (RF)}$$

RAM Calculation Example: #

For a producer rate of $20 \text{ MB/s}$ and $\text{RF} = 3$: $$\text{RAM Page Cache} = 20 \text{ MB/s} \times 3600 \times 3 = 216,000 \text{ MB} \approx 216 \text{ GB}$$

If we divide into 3 brokers: $$\text{RAM Allocation per Broker} = \frac{216 \text{ GB}}{3} = 72 \text{ GB per Broker}$$

So, our broker server specifications are advised to have a minimum total RAM capacity of 80 GB (72 GB for OS Page Cache, 6 GB for JVM Heap, and 2 GB for the operating system). If servers only have 16 GB RAM, page caches quickly fill and force consumers to read directly from physical disks, causing sharply spiking latencies (high latency spikes).

CPU Capacity Planning (Compression & Encryption) #

Although Kafka brokers are generally CPU-efficient because they don’t parse message contents, there are two security and optimization features that can trigger significant CPU load spikes on brokers:

1. Transit Encryption (SSL/TLS) and SASL Authentication #

When we enable SSL/TLS encryption to secure data transmitted between clients and brokers, broker CPUs must do real-time data packet encryption/decryption processes on every network socket. Repeated SSL Handshake processes from thousands of clients also heavily burden CPUs.

2. Message Decompression for Validation #

In general, producers compress messages (e.g., with Zstd) and consumers decompress them, so brokers only store raw binaries. However, brokers are forced to do server-side decompression if:

• We configure record validation features (e.g., validating message timestamps).
• The compression type set on brokers (compression.type on brokers) differs from the compression type sent by producers. This condition forces brokers to decompress payloads and re-compress them using target algorithms before writing to disks.

CPU Sizing Recommendations #

• Non-SSL / No Broker Compression Clusters: Just use 4 to 8 CPU cores per broker.
• Active SSL + SASL + Active Compression Clusters: Use a minimum of 16 to 32 CPU cores per broker, and make sure we enable hardware encryption features (Intel QuickAssist or AES-NI instructions) at the processor level to lighten cryptographic loads.

Operational Compliance and Capacity Planning Audit Checklists #

Evaluate our cluster architecture designs using the following capacity audit table to guarantee readiness before entering production deployment phases:

Summary #

• Networks Are the Main Bottleneck — Always calculate total inbound and outbound bandwidth by including replication factors and consumer group counts to avoid network card saturation.
• Provide RAM for Page Caches — Allocate sufficient free RAM so the hottest log segment data (at least the last 2 hours) stays in OS page cache memory to guarantee low read latencies.
• Use SSDs for Low Latency — Choose SSD/NVMe storage media if our clusters serve random reads by slow consumers or if IOPS loads are very fluctuating.
• Optimize CPUs from Cryptography — Prepare additional CPU core capacities if clusters enable SSL/TLS transit encryption and SASL authentication to dampen encryption overhead.

← Previous: Debugging Message Flow Next: Partition & Broker Sizing →