Zookeeper vs KRaft #

In the evolution of distributed data infrastructure technology, how a cluster coordinates to manage its metadata state is one of the most important architectural decisions determining the system’s scalability limits. For over a decade, Apache Kafka heavily depended on Apache ZooKeeper as an external coordination system to manage broker status, topics, partitions, and leader designation. However, this dual-system architecture gave birth to various operational complexity problems, synchronization vulnerabilities, and physical partition count limits.

As a solution, the modern Kafka era introduced KRaft (Kafka Raft Metadata Mode), which unifies metadata management directly into Kafka brokers using a deeply optimized Raft consensus protocol. In this article, we’ll deeply dissect the architectural differences between ZooKeeper vs KRaft, why KRaft can multiply partition capacity limits to millions of partitions, the metadata controller failover flow, and practical configuration guidance for quorum parameters and KRaft cluster initialization on our servers.

The Classic Era: Coordination with Apache ZooKeeper #

In the classic Kafka architecture (version 2.x and below), a Kafka cluster cannot run on its own. We must run an Apache ZooKeeper cluster (usually called a ZooKeeper Ensemble with at least 3 nodes) alongside our Kafka brokers.

In this topology, data and metadata responsibility is rigidly separated:

  • Data Storage: Kafka brokers are only responsible for physical binary log storage and serving client requests.
  • Metadata Storage: ZooKeeper acts as the single source of truth for all cluster state. ZooKeeper stores a hierarchical binary tree structure (znodes) containing data about: active brokers, topic configurations, client quota limits, and the In-Sync Replicas (ISR) and Leader lists for every partition.

The Controller Broker Role and the Watcher Mechanism #

To interact with ZooKeeper, the Kafka cluster designates one broker to take on the Controller role through a key contention mechanism (znode locking) in ZooKeeper. The Controller acts as a single bridge:

  1. The Controller installs Watchers on ZooKeeper znodes.
  2. When a status change event occurs (for example, a data broker dies), ZooKeeper triggers a Watch Notification to the Controller.
  3. The Controller reads the latest state from ZooKeeper, updates its local memory, then sends binary metadata update commands (LeaderAndIsrRequest and UpdateMetadataRequest) to all remaining data brokers in the cluster over TCP connections.

ZooKeeper’s Fatal Weaknesses #

Although ZooKeeper works very well for small-scale clusters, this architecture has fatal weaknesses when data volumes and partition counts in our cluster swell:

  1. Serial Synchronization Bottleneck (Metadata Bottleneck) When a broker leading tens of thousands of partitions suddenly dies, the Controller must elect new leaders for those tens of thousands of partitions. The Controller must write these changes to ZooKeeper one by one in sequence (serial write). ZooKeeper then processes and returns a success response, and only then does the Controller propagate this new metadata information to all brokers in the cluster. This chained synchronization process is very slow. On large clusters with more than 50,000 partitions, the recovery time after a broker failure can take 20 to 40 minutes, during which the cluster is unstable.

  2. Network Watch Count Limits Every time a partition status changes, ZooKeeper must trigger a notification to the Controller. When partition counts reach hundreds of thousands, the number of active watchers on ZooKeeper swells exponentially. This consumes enormous network bandwidth and RAM on the ZooKeeper side. If ZooKeeper experiences network congestion, it can unilaterally drop session connections with the Controller, triggering wild cascade controller elections that worsen cluster chaos.

  3. GC Pause and Split-Brain Vulnerability If the JVM process on the Controller broker experiences a long memory freeze (Garbage Collection pause), the Controller stops responding to ZooKeeper. ZooKeeper considers that Controller dead because the session timeout is exceeded, and immediately elects another broker as the new Controller. When the old Controller wakes up from the GC pause, it may not realize it has been dethroned. This can trigger the Split-Brain phenomenon (two brokers both believing they’re the active Controller), often leading to cluster metadata corruption.

  4. Double Operational Overhead For our infrastructure operations teams, maintaining two different distributed systems is a heavy burden. ZooKeeper has a configuration format, network ports, security mechanisms (SASL/JAAS), and monitoring systems completely different from Kafka. SSL security must be configured twice: once for Kafka-to-Kafka, and once more for Kafka-to-ZooKeeper.

The Modern Era: Integrated Consensus with KRaft #

Starting from version 3.0 and becoming the mandatory standard in version 4.0, Apache Kafka completely removed ZooKeeper and switched to KRaft (Kafka Raft). In KRaft mode, Kafka adopts the Raft consensus protocol, specially modified for event streaming metadata needs.

Under the KRaft architecture, a Kafka cluster no longer depends on an external system. Metadata management runs internally on Kafka brokers designated as the Controller Quorum:

  • Controller Quorum: A special group of brokers (usually 3 or 5 nodes for production clusters) running the Controller role.
  • Active Controller: One Quorum member is elected through Raft consensus as the active leader (Leader), while other Quorum members act as warm backups (standby/Followers).
  • Metadata Log: All cluster metadata is stored in a secret internal topic named @metadata, replicated among all Controller Quorum members using the Raft consensus engine. Every ordinary data broker (Data Broker) also continuously asynchronously syncs this metadata log.

The Metadata Image and Metadata Delta Concepts #

One of KRaft’s speed secrets is the use of Metadata Image and Metadata Delta data structures in every broker’s RAM:

  • Metadata Image: A fully materialized view of the cluster state in broker memory. Brokers can instantly read cluster state with O(1) complexity from RAM without network queries.
  • Metadata Delta: The partial change (diff) generated by the Active Controller when a new event occurs (for example, topic creation). This delta is written to the @metadata topic and broadcast to all brokers. Brokers then apply this delta to their local Metadata Image to update state asynchronously and very quickly.

Main Advantages of the KRaft Architecture #

KRaft elegantly solves all of ZooKeeper’s architectural weaknesses:

  1. Sub-Second Failover Because all backup Controller Quorum members (standby) continuously replicate the metadata log in real-time using Raft, they maintain an identical copy of metadata state in their memory (Warm Standby). When the Active Controller suddenly dies, the quorum elects a new Active Controller within milliseconds. The new Controller doesn’t need to reinitialize or read data from an external database; it can immediately process cluster metadata requests because the data is already in its local memory.

  2. Extraordinary Scalability (Up to Millions of Partitions) Because metadata is managed in Kafka’s own highly optimized internal commit log form, KRaft can handle up to millions of partitions per cluster without performance degradation. The partition limit per broker is no longer constrained by external coordination system performance.

  3. Infrastructure Simplification We only manage one type of software technology (Kafka). There’s only one configuration file type, one network security protocol, one SSL certificate method, and one metric monitoring system. This significantly reduces deployment complexity and operational costs for our infrastructure teams.

Architecture Comparison: ZooKeeper vs KRaft #

Let’s compare the differences between the ZooKeeper-era cluster coordination topology and the modern unified KRaft-based architecture through the following comparative diagram:

flowchart TD
    subgraph EraZK["Classic Era: ZooKeeper Coordination"]
        direction TB
        ZK["ZooKeeper Ensemble <br/> (External Cluster)"] <===>|"Watch & Metadata Sync"| CtrlBroker["Kafka Broker (Controller)"]
        CtrlBroker -->|"LeaderAndIsr / UpdateMetadata"| DB1["Data Broker 1"]
        CtrlBroker -->|"LeaderAndIsr / UpdateMetadata"| DB2["Data Broker 2"]
    end

    subgraph EraKRaft["Modern Era: KRaft Consensus"]
        direction TB
        subgraph Quorum["Controller Quorum (Raft)"]
            ActiveCtrl["Active Controller <br/> (Raft Leader)"] <== Raft Replication ==> StandbyCtrl["Standby Controller <br/> (Raft Follower)"]
        end
        ActiveCtrl -. Metadata Publish .-> KDB1["Data Broker 1"]
        ActiveCtrl -. Metadata Publish .-> KDB2["Data Broker 2"]
    end

    style ZK fill:#ffdddd,stroke:#ff8888
    style Quorum fill:#ddffdd,stroke:#88ff88
    style ActiveCtrl fill:#ddffdd,stroke:#88ff88

KRaft Initialization and Parameter Tuning Guide #

To enable KRaft mode, we must define special configuration properties in our broker configuration file (for example config/kraft/server.properties). Here are the key parameters we must set:

Key Configuration Parameters #

  • process.roles: Determines the node’s role. Valid values are:
    • broker — Acts as an ordinary data broker managing business data topics.
    • controller — Acts as a metadata manager (Controller Quorum member).
    • broker,controller — The node plays a dual role (shared/combi mode). Only suitable for development environments or small non-critical clusters.
  • node.id: The unique integer identity number for the node (replacing the broker.id role from the ZooKeeper era).
  • controller.quorum.voters: The list of all Controller Quorum members along with their addresses and communication ports. The format is node_id1@host1:port1,node_id2@host2:port2,....
  • listeners: The network protocol ports the server listens on. For controller connections, we must include a special internal metadata port name (usually port 9093 with the CONTROLLER listener name).
    • Example: listeners=PLAINTEXT://:9092,CONTROLLER://:9093
  • metadata.log.dir: The KRaft metadata log storage directory (the @metadata topic). It’s highly recommended to separate this directory onto a different physical disk from our business data topics to avoid I/O interference.

KRaft Cluster Initialization Steps via Command Line #

After the configuration file is prepared, we must format the metadata storage before we can run the cluster. Here are the command execution steps:

Step 1: Generate a Unique Cluster ID #

We must create a unique UUID that will act as the official identifier of our cluster. Run the command:

# Create a unique cluster ID using Kafka's built-in script
bin/kafka-storage.sh random-bootstrap-id

The output of this command is a unique UUID string, for example: J8n-o5w2R3S7pE8fX9gQ1w

Step 2: Format Metadata Storage #

Run the format command on every broker including the UUID from step 1:

# Format the log storage directory with the cluster ID
bin/kafka-storage.sh format \
  -t J8n-o5w2R3S7pE8fX9gQ1w \
  -c config/kraft/server.properties

Step 3: Run the Kafka Broker #

After formatting succeeds, we can immediately start the broker server using the KRaft properties:

# Start the Kafka server in KRaft mode
bin/kafka-server-start.sh config/kraft/server.properties

Debugging Tool: kafka-metadata-shell #

One challenge of the KRaft era is the loss of the ZooKeeper CLI (zookeeper-shell.sh) that developers used to directly inspect metadata. As a replacement, Kafka provides a new tool called kafka-metadata-shell.sh.

This tool lets us read the @metadata log file interactively, like browsing a local directory using standard Linux commands (ls, cd, cat).

# Run the metadata shell to read the KRaft binary log
bin/kafka-metadata-shell.sh --snapshot /tmp/kafka-metadata-logs/__cluster_metadata-0/00000000000000000000.log

# Inside the shell, we can navigate:
>> ls /
brokers  features  local  metadataQuorum  topicIds  topics
>> cd /brokers
>> ls
1  2  3
>> cat 1/registration
{"brokerId":1,"incarnationId":"...","listeners":{"PLAINTEXT":{"host":"localhost","port":9092}},...}

This tool is very useful for diagnosing broker registration status, topic partition structures, and cluster configurations directly from binary metadata log files.

Anti-pattern vs Solution: Production Quorum Configuration #

Many developers make fatal mistakes when designing Controller Quorum topologies in production because they misunderstand the nature of Raft consensus.

1. Using 2 Controller Nodes #

Some teams choose to deploy 2 Controller Nodes to save server resources, thinking 1 backup is enough.

Consequences: Raft uses an absolute majority quorum formula:

$$\text{Minimum Quorum} = \lfloor \frac{N}{2} \rfloor + 1$$

Where $N$ is the total number of voting members (voters).

  • If $N = 2$, the majority quorum needed is $\lfloor 2/2 \rfloor + 1 = 2$.
  • If one of the 2 Controllers fails, only 1 Controller stays alive. Since 1 is smaller than the minimum quorum limit (2), the Controller Quorum crashes entirely instantly. We have zero failure tolerance.
  • Always use an odd minimum of 3 controllers (tolerating 1 node failure) or 5 controllers (tolerating 2 node failures) in our production environments.

2. Mixing Broker and Controller Roles in Heavy Production #

Setting process.roles = broker,controller on all nodes in a large-scale production cluster to simplify server management.

Consequences: When a data broker experiences very heavy data traffic, JVM heap threads undergo intensive Garbage Collection and disk I/O CPU load runs very high. If that node also manages KRaft metadata consensus, long GC pauses can trigger the node’s removal from the consensus quorum. The cluster experiences metadata coordination instability due to disrupted internal data processes.

Always strictly separate roles in production: run 3 small dedicated servers as dedicated controllers (process.roles = controller) and other servers as dedicated brokers (process.roles = broker).

Summary #

  • ZooKeeper Mode — The classic Kafka topology where cluster metadata state is managed in an external ZooKeeper cluster and propagated by one elected Controller broker.
  • KRaft Mode — The modern Kafka architecture integrating metadata coordination directly into brokers using an optimized Raft consensus protocol.
  • Raft Metadata Log — The internal @metadata topic in KRaft replicating all cluster configuration transactions to all controllers safely and consistently.
  • Controller Quorum — The designated special broker group managing metadata, consisting of one Active Controller (Leader) and several Standby Controllers (Followers).
  • Fast Recovery — KRaft cuts cluster recovery (failover) time from tens of minutes in the ZooKeeper era to just a few milliseconds because metadata is maintained directly in standby memory.
  • Storage Formatting — The kafka-storage.sh utility must be used to bind the cluster under one shared Cluster ID UUID before brokers are started.
  • Odd Quorum — Per the Raft majority consensus rule, the controller count must be odd (minimum 3) to avoid leader election deadlocks (quorum deadlock).
  • Role Separation — Run dedicated controllers separate from data brokers in production clusters to avoid consensus disruption from data I/O loads or GC pauses.
← Previous: In-Sync Replica
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact