Broker #

In the Apache Kafka ecosystem, Broker is the term for a single physical or virtual server instance responsible for receiving events from producers, writing them persistently to local disk storage, and serving data read requests from consumers. A Kafka cluster consists of several brokers coordinating with each other to distribute workload evenly and tolerate failures. Understanding the broker’s internal functions — from physical storage structure at the file system level, metadata coordination, to how to allocate system memory optimally — is key for keeping cluster performance stable under very heavy production workloads.

Main Roles of a Kafka Broker #

Functionally, Kafka brokers are designed as highly efficient, relatively simple servers (dumb broker, smart client). Unlike other message brokers that must track which messages each consumer has read in server memory, Kafka brokers don’t store consumer read state information at all.

A Kafka broker’s main tasks focus on three things:

  1. Data Reception: Receiving binary message packets from producers and writing them as fast as possible into partition log segments on local disk using sequential I/O operations.
  2. Data Storage: Managing the file storage lifecycle based on configured retention limits (time and size).
  3. Data Delivery: Serving consumer data pull requests by leveraging OS Page Cache memory and Zero-Copy techniques to send binary bytes directly to the Network Interface Card without CPU processing in JVM user space.

Physical Data Storage Structure (Log Directory) #

When we set the Kafka data storage directory through the log.dirs parameter in the server.properties configuration file, Kafka creates a special directory structure for every partition of the topics it manages. The physical storage folder name follows the format:

$$\text{topic-name}-\text{partition-index}$$

As a concrete example, if we have the topic prod.finance.payment.fact.transaction-completed with 3 partitions, the broker storing those partitions will have folders named prod.finance.payment.fact.transaction-completed-0, prod.finance.payment.fact.transaction-completed-1, and so on.

Inside each partition folder, data is stored in small binary file segments divided into three main file types:

flowchart TD
    subgraph FolderPartisi["Physical Partition Folder (e.g. topik-transaksi-0)"]
        direction TB
        LogFile["Log Segment File (.log) <br/> (Contains the actual binary event payloads)"]
        IndexFile["Offset Index File (.index) <br/> (Maps Offsets to physical byte positions in the .log file)"]
        TimeIndexFile["Time Index File (.timeindex) <br/> (Maps Timestamps to message offsets)"]
    end

1. Log Segment File (.log) #

This file contains the actual collection of binary events written sequentially (append-only). By default, when the log file size reaches 1 GB (set by log.segment.bytes), Kafka closes that file, makes it read-only, and opens a new empty active segment file to accept the next writes.

2. Offset Index File (.index) #

Reading a gigabyte-sized log file from the beginning to find a particular offset is a very slow operation. To speed up lookups, Kafka creates a special index file.

However, instead of indexing every message (which would waste disk space), Kafka uses a Sparse Index. Kafka only records an index entry every certain number of data bytes (set by log.index.interval.bytes, default 4 KB). This index file maps offset numbers directly to physical byte positions inside the .log file. Consumers can jump to the nearest byte position of a target offset instantly ($O(1)$) using in-memory binary search.

3. Time Index File (.timeindex) #

This file is used to find data by timestamp. The .timeindex file maps timestamp values to the corresponding offset numbers. This is very useful when consumer applications want to replay data starting from a particular time, for example “re-read all transactions from yesterday at 08:00”.

Partition Replication Management (Leader vs Follower) #

In a distributed cluster, we must not rely on a single server because physical servers are prone to sudden crashes, disk failures, or power outages. Kafka handles this by replicating topic partitions across several different brokers.

When managing this partition replication, Kafka brokers split their roles into two:

flowchart LR
    subgraph Client["Client Applications"]
        Producer["Producer"]
        Consumer["Consumer"]
    end

    subgraph KafkaBrokers["Kafka Broker Cluster"]
        direction TB
        subgraph Broker1["Broker 1 (Server A)"]
            LeaderPart[("Partition 0 (Leader)")]
        end
        subgraph Broker2["Broker 2 (Server B)"]
            FollowerPart1[("Partition 0 (Replica / Follower)")]
        end
        subgraph Broker3["Broker 3 (Server C)"]
            FollowerPart2[("Partition 0 (Replica / Follower)")]
        end
    end

    Producer -->|Write Data to Leader| LeaderPart
    Consumer -->|Read Data from Leader| LeaderPart
    FollowerPart1 -. "Pull Replication Data" .-> LeaderPart
    FollowerPart2 -. "Pull Replication Data" .-> LeaderPart

1. Leader Broker #

For each partition, one of the brokers storing its replicas is designated as the Leader. All data writes from producers and data reads from consumers are by default directed to this Leader broker. The Leader broker holds full control over the correctness of that partition’s log data ordering.

2. Follower Broker #

Other replica brokers act as Followers. They behave passively like internal consumers; their only job is to periodically send pull requests to the Leader broker to copy the latest data segments to their own local disks.

In-Sync Replicas (ISR) #

The Leader broker continuously monitors the replication progress of Followers. If a Follower successfully copies the latest data within a certain time limit (set by replica.lag.time.max.ms, default 30 seconds), that Follower joins the In-Sync Replicas (ISR) group. If a Follower experiences network congestion or slow disk causing it to lag more than 30 seconds, the Leader removes it from the ISR group. ISR membership is very important because only brokers registered in the ISR are eligible to be elected as the new Leader if the current Leader crashes.

Controller Broker: The Cluster Coordination Brain #

Although all brokers in a Kafka cluster can serve clients, there must be one designated single broker acting as the cluster coordination leader. This broker is called the Controller.

In modern KRaft-based clusters (Kafka Raft Metadata Mode), this coordination is elegantly managed by a group of elected brokers acting as the Controller Quorum.

The Controller’s main tasks include:

  • Broker Failure Detection: Monitoring the health of all brokers in the cluster. If a broker is detected dead or leaving the cluster, the Controller is responsible for notifying other brokers.
  • Partition Leader Failover (Leader Election): When the broker holding the Leader role for a partition dies, the Controller immediately looks at that partition’s ISR list, elects a healthy Follower from the ISR as the new Leader, and publishes this new leadership metadata change to all brokers in the cluster so clients (producers/consumers) can update their connection routes.
  • Topic Management: Coordinating new topic creation, topic deletion, or adding new partitions across the cluster.

Optimal Memory Allocation: JVM Heap vs OS Page Cache #

One of the most fatal and common mistakes made by system administrators when tuning Kafka broker servers is allocating the JVM Heap as large as possible (for example, allocating 64 GB of server RAM specifically for the JVM Heap).

Although Kafka is written in Java, it’s designed with an architecture philosophy that deeply respects the operating system (OS-friendly architecture). Kafka is deliberately designed to use as little JVM Heap as possible (ideal recommendation is only 4 GB to 5 GB for a general broker process) and leave the rest of the server’s abundant physical RAM to be managed by the OS Page Cache.

Let’s study why this memory allocation is so decisive for broker performance:

  • Garbage Collection (GC) Overhead: If we set the JVM Heap too large (e.g., 32 GB or more), when the JVM Garbage Collector runs to clean up unused memory objects, the system experiences very long execution pauses (stop-the-world GC pause) that can reach seconds to minutes. During this pause, the broker stops responding to heartbeats and client connections, triggering wild rebalances and cluster failures. By limiting the JVM heap to around 5 GB, GC pause duration can be kept under a few milliseconds.
  • Page Cache Utilization for Zero-Copy: Data written by producers to disk is actually first written by the OS into RAM acting as Page Cache. When consumers request to read that recently written data, Kafka doesn’t need to read the physical file from slow disk. Kafka directly reads the data from RAM’s Page Cache and sends it straight to the network card using the system’s Zero-Copy instruction (sendfile on Linux). This yields binary data transfer speeds equivalent to RAM memory performance (RAM-speed transfer).

The Secret of Data Transfer Speed: Zero-Copy Technology #

To understand why this technique is so revolutionary, let’s compare the regular data transfer path with Zero-Copy below:

  • Conventional Data Read Path (4 Copies):

    1. The OS reads data from physical Disk into Kernel Space Page Cache.
    2. The application (JVM) copies that data from Kernel Space into User Space (JVM Heap) via the read() syscall.
    3. The application copies the data again from JVM Heap into Socket Buffer (Kernel Space) via the write() syscall.
    4. The OS copies the data from Socket Buffer into NIC Buffer (Network Interface Card) before sending it over the network cable.
    • Drawback: This process requires 4 data copies in RAM and triggers 4 CPU context switches between User Mode and Kernel Mode. This wastes CPU cycles enormously.

  • Zero-Copy Path with sendfile() (2 Copies):

    1. The OS reads data from Disk into Kernel Space Page Cache.
    2. Using the sendfile() instruction, the OS directly copies the data descriptor from Page Cache into NIC Buffer directly using DMA (Direct Memory Access) hardware.
    • Advantage: Data never enters User Space (JVM Heap) memory at all. No CPU cycles are wasted copying data bytes. Processing time is drastically cut and network transfer throughput increases spectacularly.

The Graceful Shutdown Lifecycle (Controlled Shutdown) #

As cluster administrators, we must always shut down brokers cleanly (graceful controlled shutdown) using the SIGTERM signal or management commands, never by force-cutting server power (SIGKILL).

During a controlled shutdown:

  1. The broker stops accepting new client connections.
  2. The broker performs leader migration for all Leader partitions it holds to other Follower brokers in an orderly fashion.
  3. All remaining unsaved Page Cache memory is flushed to disk storage to prevent data corruption.
  4. The broker leaves cluster membership cleanly.
  • Benefit: Minimizes system recovery downtime, prevents inconsistent data states, and avoids emergency leader election processes that trigger unexpected rebalances for clients.

Common Mistakes (Anti-patterns) in Broker Management #

Here are Kafka broker operational configuration mistakes we must avoid:

1. Running Brokers on Shared Disk (NAS / SAN / NFS) #

Storing the broker’s log.dirs data directory in shared network storage (Network Attached Storage) like NFS or SAN for capacity management convenience.

Consequences: Kafka is designed assuming direct access to direct attached storage with low, consistent I/O latency. Storing log data on NAS triggers network bandwidth contention between Kafka’s internal replication activity and disk storage I/O paths. This causes disk write latency to spike dramatically, triggers replication sync failures, kicks brokers out of ISR, and eventually makes cluster performance plummet or die entirely. Always use local SSD/NVMe disks directly attached to the physical broker server.

// ANTI-PATTERN in server.properties:
log.dirs=/mnt/shared-nfs/kafka-data   // Using shared network storage (Very Dangerous!)

// The CORRECT solution:
log.dirs=/var/lib/kafka/data          // Always point to the server's local SSD/NVMe disk

2. Ignoring the Linux File Descriptor Limit (Open Files Limit) #

Running the Kafka broker process on Linux with a very low default open files limit configuration (usually only 1024 files by default).

Consequences: As discussed, every topic partition needs at least 3 continuously open files (.log, .index, .timeindex). If we have hundreds of topic partitions and several active client TCP socket connections, the broker quickly hits the OS limit and throws the fatal error java.io.IOException: Too many open files. The broker process crashes instantly. Make sure the Linux open file limit is raised to at least 65536 or 100000 before running Kafka.

Summary #

  • Broker Definition — A broker is one physical or virtual server instance in a Kafka cluster responsible for receiving, storing, replicating, and delivering binary event data streams sequentially and efficiently.
  • Physical Structure — Data is stored in special file-system-level partition folders divided into .log segment files (binary payloads), .index files (sparse offset index for fast lookup), and .timeindex files (timestamp index).
  • Replication Role — Partition replication is managed using the Leader (serves client reads/writes) and Follower (copies data asynchronously from the Leader) scheme. Healthy replica brokers are registered in the In-Sync Replicas (ISR) group.
  • Controller Role — The Controller is the cluster coordination brain responsible for detecting broker failures, managing partition leadership transitions, and coordinating cluster metadata.
  • Memory Tuning — Avoid overly large JVM Heap allocations to prevent long Garbage Collection pauses. Allocate a modest JVM Heap (4-5 GB) and leave abundant server RAM for the OS Page Cache to support the Zero-Copy transfer feature.
  • Use Local Storage — Always use local SSD/NVMe storage directly attached to the broker server; never use shared network storage (NAS/SAN) because it destroys disk I/O performance.

← Previous: Consumer Next: Cluster →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact