Cluster #

In modern computing architecture, a system must not depend on the performance of a single server because physical servers are prone to hardware failures, power outages, or network disruptions. Apache Kafka solves this challenge by uniting several broker servers into a single distributed system called a Cluster. Kafka clusters are designed from the start to provide high availability, unlimited scalability, and robust fault tolerance. Understanding how cluster components coordinate, how metadata is managed with KRaft, and how to design cluster architectures across physical racks is fundamental to keeping our business data pipelines running without interruption.

Basic Concept: The Kafka Cluster as a Distributed System #

Simply put, a Kafka cluster is a group of brokers (servers) working together to manage topics and partitions collectively. To producer and consumer applications, a Kafka cluster looks like one single huge server entity. Client applications don’t need to care about where a message is physically stored; they just interact with the cluster, and the cluster handles workload distribution internally.

There are three main pillars underlying Kafka cluster design:

  1. Horizontal Scalability: We can instantly increase the cluster’s storage capacity and data processing capability by adding new brokers without shutting down running systems (zero-downtime scaling).
  2. Data Redundancy (Replication): Every incoming piece of data is copied to several different brokers in the cluster. If one server suffers total disk failure, our data stays safe and accessible from backup servers.
  3. No Single Point of Failure (NSPoF): The cluster is designed so it has no single point of failure. If one broker dies (even if that broker acts as the cluster leader), its role is immediately taken over by another broker automatically within milliseconds.

Cluster Topology: Brokers, ZooKeeper, and KRaft #

To ensure all brokers in the cluster work harmoniously, the cluster needs a state and metadata management system (such as the list of active topics, partition counts, broker status, and partition leadership locations).

Throughout its development history, Apache Kafka has adopted two different cluster coordination methods:

1. The Classic Era: Apache ZooKeeper #

In Kafka 2.x and below, Kafka clusters heavily depended on Apache ZooKeeper to manage cluster metadata. ZooKeeper ran as a separate cluster outside the Kafka brokers.

  • Problem: This architecture is very complex because we must manage and monitor two different distributed systems simultaneously. Additionally, there’s a performance bottleneck when partition counts reach tens of thousands; metadata update coordination from ZooKeeper to Kafka brokers becomes very slow and prone to data desynchronization during large-scale cluster failures.

2. The Modern Era: KRaft (Kafka Raft Metadata Mode) #

Starting from version 3.x and becoming the official standard in version 4.x, Kafka completely removed ZooKeeper and replaced it with KRaft. KRaft adopts the Raft consensus protocol integrated directly into Kafka brokers. Metadata is now stored and managed as a secret internal topic named @metadata replicated among elected brokers acting as the Controller Quorum.

Let’s study the distributed coordination topology of a modern Kafka cluster using KRaft mode:

flowchart TD
    subgraph ControllerQuorum["Controller Quorum (Metadata Leader & Followers)"]
        direction LR
        Ctrl1["Controller 1 <br/> (Follower)"]
        Ctrl2["Controller 2 <br/> (Leader / Active Metadata Store)"]
        Ctrl3["Controller 3 <br/> (Follower)"]
    end

    subgraph BrokerData["Data Brokers (Worker Brokers)"]
        direction LR
        B1["Broker 4"]
        B2["Broker 5"]
        B3["Broker 6"]
    end

    Ctrl1 -. Raft Replication .-> Ctrl2
    Ctrl3 -. Raft Replication .-> Ctrl2

    B1 -->|Request Metadata & Update Status| Ctrl2
    B2 -->|Request Metadata & Update Status| Ctrl2
    B3 -->|Request Metadata & Update Status| Ctrl2

With KRaft, propagating metadata changes to all worker brokers becomes nearly instant. Modern Kafka clusters can support up to millions of active partitions without performance degradation, and cluster recovery time (failover time) during server failures is cut from minutes to just a few milliseconds.

How KRaft Quorum Consensus Works #

The KRaft protocol divides metadata handling responsibilities within the Controller Quorum in a structured way:

  • Controller Leader (Active Controller): One elected controller acting as the active leader. It receives all metadata changes from worker brokers (for example, when a new topic is created) and writes those changes to the internal metadata log.
  • Controller Followers (Standby Controllers): Other controllers act as followers. They actively replicate metadata log changes from the Leader to keep their data in sync.
  • Epoch Numbers (Preventing Split-Brain): Every Controller Leader leadership term is marked with a unique number called the Epoch Number. If a network partition isolates the old Leader, other quorum members detect it and elect a new Leader with a higher Epoch Number. When the isolated broker reconnects, it realizes its Epoch Number is stale and safely surrenders leadership, avoiding a dual-leader (split-brain) condition.

Client Connection Mechanism: Bootstrap Servers #

When configuring a Kafka producer or consumer, we supply the bootstrap.servers configuration parameter as a list of several brokers (for example bootstrap.servers=broker1:9092,broker2:9092).

Many developers mistakenly think clients send all data through all those bootstrap servers.

Actually, the bootstrap connection is only used for the initial connection. The client randomly picks one healthy broker from that list to request Cluster Metadata. That broker responds with the list of all active brokers in the cluster and the partitions each leads. After metadata is received, the client immediately opens a new TCP socket connection to the physical broker acting as the Leader of the destination partition, bypassing the initial bootstrap server intermediary. This guarantees direct path data delivery with minimal latency.

Dynamic Configuration Management #

In production cluster operations, we often need to change configuration parameters without shutting down and restarting brokers. This is crucial for keeping system availability at 100% while administrator teams (SRE) do optimization or emergency troubleshooting. Kafka divides its configuration into two:

  • Static Configurations: Basic configurations that must be written in the server.properties file and require a broker restart to apply (like the listeners port address or the log.dirs storage directory).
  • Dynamic Configurations: Topic-level or broker-level configurations that can be changed in real-time while the system is active using the incrementalAlterConfigs API.

In the KRaft era, when we run a dynamic configuration change command (for example, changing retention policy or raising the max.message.bytes size limit on a particular topic), the command is received by the Controller Leader, written as a new metadata log entry in the @metadata log, and automatically synchronized to all worker brokers within milliseconds. Worker brokers then immediately apply that configuration change in memory dynamically without interrupting running producer or consumer connections.

Fault Tolerance & Failover Mechanism #

One of the main strengths of a Kafka cluster is its ability to detect failures automatically and recover (failover) on its own without human intervention (self-healing cluster).

Let’s study the failure handling workflow when one broker in the cluster suddenly dies:

sequenceDiagram
    participant B4 as Broker 4 (Leader Partition 0)
    participant Ctrl as KRaft Controller (Leader)
    participant B5 as Broker 5 (Follower Partition 0 / ISR)
    participant Client as Producer / Consumer

    Note over B4: Broker 4 server suddenly dies (Crash/power outage)
    Note over Ctrl: Controller detects Broker 4's Heartbeat is missing
    Ctrl->>Ctrl: 1. Declare Broker 4 Offline
    Ctrl->>Ctrl: 2. Look for healthy replica in Partition 0 ISR (Found Broker 5)
    Ctrl->>Ctrl: 3. Elect Broker 5 as the NEW LEADER of Partition 0
    Ctrl->>B5: 4. Update Broker 5 status to Leader
    Ctrl->>Client: 5. Publish New Metadata to all Clients
    Client->>B5: 6. Switch read/write connections directly to Broker 5
    Note over Client: Data delivery continues smoothly!

Failover Step Explanation: #

  1. Heartbeat Loss Detection: Every worker broker must periodically send heartbeat signals to the Controller broker. If the Controller doesn’t receive a heartbeat from Broker 4 past the zookeeper.connection.timeout.ms limit (in the old era) or the KRaft heartbeat parameter, the Controller declares Broker 4 as having left the cluster (offline).
  2. Emergency Leader Election: The Controller checks the list of partitions led by Broker 4. For each partition, the Controller looks for a Follower replica whose data state is in sync and registered in the In-Sync Replicas (ISR) on another broker (in the example above, Broker 5).
  3. Promoting the New Leader: The Controller promotes Broker 5 as the new Leader for that partition and writes this change to the metadata log.
  4. Client Metadata Update: The Controller propagates this latest metadata information to all active brokers and client applications. Connected producers and consumers automatically update their TCP socket connection routes to Broker 5 transparently without interrupting our business application flow.

Designing Highly Available Clusters #

Running several Kafka brokers in the same physical server rack or in the same Data Center still carries high failure risk. If that rack’s network switch breaks or a total power outage happens in that data center, our entire Kafka cluster dies instantly.

To design a truly reliable production-grade cluster, we must apply the Rack Awareness strategy and spread across Availability Zones (AZ).

1. Rack Awareness #

We must configure the broker.rack parameter on every broker configuration file to tell Kafka which physical rack or Availability Zone the broker is in:

  • broker.rack=us-east-1a (for Broker 1)
  • broker.rack=us-east-1b (for Broker 2)
  • broker.rack=us-east-1c (for Broker 3)

With this configuration enabled, when we create a new topic with Replication Factor = 3, Kafka geographically guarantees that partition replicas are distributed across three different racks or zones. Data will never pile up on the same rack.

2. Cross-Zone Data Reliability Parameters #

Cross-zone distribution must be paired with safe data write parameters on producer applications and topics:

  • replication.factor (Minimum 3): Guarantees data is copied to 3 different brokers in different zones.
  • min.insync.replicas (Minimum 2): Guarantees every data write must be successfully recorded on at least 2 in-sync brokers before being considered successful.
  • acks = all on the Producer Side: Forces producers to wait for confirmation from at least the number of brokers specified by min.insync.replicas.

With the combination of the three parameters above, if one Availability Zone suffers a natural disaster or total outage, our Kafka cluster in other zones stays alive, partition leadership switches over automatically, and not a single byte of data is lost.

Common Mistakes (Anti-patterns) in Cluster Management #

Here are some fatal mistakes in designing and operating Kafka clusters in production:

1. Running All Cluster Brokers in the Same Rack / Availability Zone #

Installing 3 Kafka brokers on 3 different VMs (Virtual Machines), but all three VMs run on the same physical hypervisor server or sit in the same hardware rack.

Consequences: Having 3 brokers gives an illusion of high availability. However, if that physical hypervisor server crashes from memory failure or the rack loses power, all three brokers die simultaneously. The Kafka cluster suffers total quorum failure and data becomes inaccessible. Always distribute cluster physical brokers evenly across hardware and geographic zones.

flowchart TD
    subgraph AntiPattern["ANTI-PATTERN"]
        direction TB
        B1["Broker 1 (VM A)"] --> X1["Physical Server X (Zone A)"]
        B2["Broker 2 (VM B)"] --> X1
        B3["Broker 3 (VM C)"] --> X1
    end
    
    subgraph Correct["The CORRECT solution (Rack Awareness)"]
        direction TB
        RB1["Broker 1 (VM A)"] --> RX["Physical Server X (Zone A)"] --> R1["broker.rack=zone-a"]
        RB2["Broker 2 (VM B)"] --> RY["Physical Server Y (Zone B)"] --> R2["broker.rack=zone-b"]
        RB3["Broker 3 (VM C)"] --> RZ["Physical Server Z (Zone C)"] --> R3["broker.rack=zone-c"]
    end

2. Setting min.insync.replicas Equal to the replication.factor #

Configuring a topic with replication.factor = 3 and setting min.insync.replicas = 3 for maximum data safety.

Consequences: This configuration is a fatal trap for cluster availability. If one of the 3 brokers undergoes routine maintenance or temporarily dies, the number of in-sync active replicas (ISR) drops to 2. Because min.insync.replicas demands confirmation from 3 brokers, producers sending data with acks=all immediately receive the NotEnoughReplicasException error. Our cluster rejects all new data write activity, crashing the application system entirely. Always leave at least 1 server of failure tolerance using the formula:

$$\text{min.insync.replicas} = \text{replication.factor} - 1$$

Summary #

  • Cluster Definition — A Kafka cluster is a collection of distributed broker servers working together to share storage load, data processing, and provide high fault tolerance for client applications.
  • KRaft Mode — Modern Kafka clusters use KRaft mode to manage cluster metadata internally through the Raft consensus protocol, replacing the external dependency on Apache ZooKeeper.
  • Failover Process — When a partition Leader broker dies, the Controller detects the lost heartbeat and immediately promotes a healthy ISR Follower as the new Leader automatically.
  • Rack Awareness — The broker.rack parameter configuration is mandatory to ensure partition replicas are spread across different physical racks or Availability Zones for geographic reliability.
  • Failure Tolerance — Set production cluster configurations safely with replication.factor at minimum 3, min.insync.replicas at minimum 2, paired with the producer delivery parameter acks=all.
  • Avoid Strict Limits — Don’t set min.insync.replicas equal to the replication.factor value because it eliminates cluster failure tolerance and triggers write congestion when one server is being maintained.

← Previous: Broker Next: Distributed Commit Log →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact