Performance Tuning: Optimizing Apache Kafka Performance at Production Scale #

When we operate Apache Kafka to support large-scale systems, we’re often faced with complaints from application teams about slow data delivery or high CPU resource usage on brokers. Apache Kafka is indeed designed to have outstanding out-of-the-box performance. However, factory default configurations are usually optimized for general purpose workloads. For industries with extreme data traffic needs, we must do specific performance tuning at various layers.

Performance tuning in Kafka isn’t a one-size-fits-all activity. Instead, this process is the art of managing trade-offs. We can’t get near-zero millisecond latency while expecting very bandwidth-efficient gigabytes-per-second delivery throughput. We must understand our application’s business needs first: whether our application is a financial platform needing instant latency, or a giant log analytics platform needing massive throughput with efficient storage costs.

In this guide, we’ll explore the latency vs throughput trade-off dilemma, optimize Linux operating system kernel settings for brokers, tune internal Kafka thread pool allocations, and adjust producer and consumer configuration parameters to reach stable peak performance points in production.

The Trade-Off Dilemma: Latency vs Throughput #

Before modifying parameters, it’s very important to understand the natural laws of distributed networking and storage: Latency and Throughput have a mutually opposing relationship.

1. LOW LATENCY OPTIMIZATION:
   * Goal: Send messages as fast as possible to consumers once data is produced.
   * Approach: Disable batching (send messages one by one), remove delays.
   * Impact: Network card (NIC) overhead is very high because it processes millions of small packets.
   * Total throughput becomes LOW because networks are busy processing TCP protocol overhead.

2. HIGH THROUGHPUT OPTIMIZATION:
   * Goal: Move data in the largest possible volume per second.
   * Approach: Enable aggressive batching (collect thousands of messages in memory before sending).
   * Impact: Networks are very efficient because they send large packets at once.
   * Per-message latency becomes HIGH because the first message must wait for batches to fill completely.

Successful performance tuning requires us to determine the tolerance limits acceptable to our businesses, then shift system configurations toward the appropriate profile.

Batch Size Relationship Diagram for Latency and Throughput #

Here’s a conceptual relationship visualization between increasing delivery batch sizes and total cluster latency and throughput values:

flowchart TD
    subgraph BatchTuning["Batch Size Adjustment (Batch Size & Linger.ms)"]
        BatchSmall["Batch Size: Very Small"]
        BatchLarge["Batch Size: Very Large"]
    end

    subgraph LatencyThroughput["Network Performance Characteristics"]
        LatencyLow["Per-Message Latency: Very Low"]
        LatencyHigh["Per-Message Latency: Higher"]
        ThroughputLow["Network Throughput: Low"]
        ThroughputHigh["Network Throughput: Very High"]
    end

    BatchSmall --> LatencyLow
    BatchSmall --> ThroughputLow

    BatchLarge --> LatencyHigh
    BatchLarge --> ThroughputHigh

Operating System Parameter Optimization (OS Kernel Tuning) #

As an application running on the JVM but heavily depending on Linux kernels for disk I/O and network operations, Kafka needs loose operating system configurations.

We’re advised to modify the /etc/sysctl.conf file to apply the following Linux kernel optimizations on broker servers:

1. Virtual Memory Management (Swappiness & Dirty Pages) #

• vm.swappiness = 1 (or 0): By default, Linux moves inactive memory data to disk swap partitions. This swap process is very slow and can trigger very long GC pauses on Kafka JVMs. By setting this value to 1, we order the OS to avoid swapping except in truly critical conditions.
• vm.dirty_background_ratio = 5: This parameter determines the system memory percentage that can hold dirty pages (data written to RAM page caches but not yet flushed to disks) before OS background threads (pdflush/flush) start writing them to disks. We set it lower than the default (usually 10) so the OS steadily installs data writes to disks to avoid I/O queue spikes.
• vm.dirty_ratio = 10: The maximum memory percentage holding dirty pages before active write processes are blocked and forced to directly write to disks. We set it to 10 (default 20) to limit the accumulation of unwritten data amounts in RAM.
• vm.dirty_expire_centisecs = 2000 (20 seconds): The data expiry time in page caches. This value (default 30 seconds) determines how long dirty data can settle in RAM before being marked for physical disk writes.
• vm.dirty_writeback_centisecs = 100 (1 second): Determines how often kernel daemons wake up to check whether dirty data needs flushing to disks. By setting it to 1 second (default 5 seconds), we smooth disk I/O curves so sudden I/O congestion doesn’t happen.

2. Memory Mapping Limits #

• vm.max_map_count = 1048576: Because Kafka uses the Java MappedByteBuffer class to map log index files into virtual memory address spaces, brokers with large partition counts create many memory mapping areas. The Linux default limit ($65530$) won’t be enough and must be significantly raised.

3. TCP Network Buffering #

Add the following socket buffer settings so network cards can handle massive data traffic without dropping data packets (packet drops):

net.core.somaxconn = 32768
net.core.netdev_max_backlog = 100000
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Modifying the rmem and wmem buffer parameters above is very crucial if broker servers connect with consumers across geographic regions having high network latencies (high bandwidth delay products).

Tuning on the Kafka Broker Side #

Inside broker server.properties configuration files, there are several internal thread allocation parameters we must adjust according to server physical CPU core capacities.

flowchart TD
    Clients["Clients"] -- "TCP Connections" --> NPT["Network Processor Threads (num.network.threads)"]
    NPT -- "Put Requests into the RequestChannel Queue" --> RHT["Request Handler Threads (num.io.threads)"]
    RHT -- "Logic Processing: Reading/Writing Data Logs to Disks" --> Disk["Disk Platters / OS Page Cache"]

1. Tuning Broker Thread Pools #

• num.network.threads: The thread count brokers allocate for reading and writing data to network sockets. These threads accept client requests and put them into request queues. We’re advised to set this value equal to the physical CPU core count on broker servers.
• num.io.threads: The thread count used for processing requests from request queues, including doing data log read-write operations to physical disks. We’re advised to set this value at a minimum of 2x the physical CPU core count or the physical disk count used on servers.

2. Broker Socket Buffer Configuration #

• socket.send.buffer.bytes = 1048576 (1 MB): TCP socket send buffer sizes. Raising this value helps smooth outbound traffic to consumers outside VPCs.
• socket.receive.buffer.bytes = 1048576 (1 MB): TCP socket receive buffer sizes for accommodating massive deliveries from producers.

3. Avoid Synchronous Disk Writes (Flush Settings) #

We’re highly advised to not modify the log.flush.interval.messages and log.flush.interval.ms parameters on brokers. Leave these parameters at defaults (unset). Why? Because forcing Kafka to synchronously flush data to disks on every message destroys broker write performance. Kafka is designed to trust inter-node broker data replication as durability guarantees, and hand over asynchronous physical disk writes to OS page caches.

Tuning on the Producer Side (Producer Tuning) #

Producers are data entry gates. Producer message packaging speeds directly impact network bandwidth utilization and broker performance.

1. linger.ms and batch.size Synergy #

For high throughput, we must maximize batching utilization:

• batch.size: The maximum memory size limit (in bytes) for grouping messages into one partition (default $16,384$ or 16 KB). In production, we’re advised to raise it to $65,536$ (64 KB) or $131,072$ (128 KB).
• linger.ms: The maximum wait time (in milliseconds) for producers to hold messages in memory before sending, to give other messages chances to enter the same batch (default 0 ms). By raising this value to $5 \text{ to } 20 \text{ ms}$, we give producers time to collect data into one large batch, increasing compression efficiency and dramatically raising throughput with slight latency sacrifices.

2. Choosing the Right Compression Algorithm #

Enabling producer compression is mandatory for large-scale clusters. Algorithm choices have different performance characteristics:

• LZ4: Offers the fastest compression and decompression speeds with very efficient CPU usage. Very ideal for low latency profiles.
• ZSTD: A modern algorithm developed by Facebook, offering the highest compression ratios (saving up to $50%$ disk space) with reasonable CPU overhead during decompression. Highly recommended for high throughput in production.
• Snappy: Offers balanced compression, but is generally still less efficient than LZ4.
• GZIP: Produces high compression but is very CPU-hungry, not recommended for real-time applications.

3. Choosing Durability Levels (acks) #

The acks setting determines how quickly producers get success confirmations:

• acks=0: Producers don’t wait for broker confirmations. Highest throughput, lowest latency, but very high data loss risks.
• acks=1: Producers wait for confirmations after leader brokers successfully write to their local logs. A safe moderate choice.
• acks=all (or -1): Producers wait for confirmations from all active ISR members. Offers absolute durability, but slightly higher latency because it must wait for replication handshakes to finish.

Tuning on the Consumer Side (Consumer Tuning) #

On consumer sides, optimization focuses on request (fetch requests) frequency efficiency to brokers to reduce network negotiation overhead.

1. Tuning Data Fetch Parameters (Fetch Buffering) #

• fetch.min.bytes: The minimum data amount (in bytes) brokers must collect before responding to consumer poll requests (default 1 byte). By raising this value to $1024$ or $8192$ (8 KB), we force brokers to wait until enough data is collected before sending to consumers, reducing network request counts and lowering broker CPU loads.
• fetch.max.wait.ms: The maximum time limit (in milliseconds) brokers hold responses if collected data amounts haven’t reached fetch.min.bytes limits (default 500 ms). Set it to $100 \text{ ms}$ if we want to limit consumer response delays.

Profile Comparison: High Throughput vs Low Latency #

Here’s a configuration parameter comparison table we can directly copy based on our application’s profile needs in production:

Operational Compliance and Performance Tuning Audit Checklist #

Make sure all our performance tuning steps meet the following production compliance criteria before releasing clusters to end users:

Summary #

• Determine Business Focus — Choose one main profile from the start: extreme low latency optimization (minimal delays) or massive high throughput (efficient bandwidth), because both can’t be achieved simultaneously.
• Configure OS Kernels — Do Linux virtual memory setting modifications (vm.swappiness, vm.dirty_background_ratio, vm.max_map_count) so brokers aren’t held back by swap processes and virtual memory mappings.
• Tune Broker Thread Allocations — Align num.network.threads settings with CPU core counts and num.io.threads at a minimum of 2x them to prevent broker request queues.
• Optimize Producer Batching — Arrange linger.ms ($5 \text{ - } 20 \text{ ms}$) and batch.size ($64 \text{ - } 128 \text{ KB}$) synergies on producers to multiply network throughput.