Rolling Upgrade & Maintenance: Cluster Maintenance Without Downtime #

When managing Apache Kafka infrastructure at production scale serving non-stop 24/7 business operations, completely stopping entire clusters (offline upgrades) for hardware maintenance or software version updates is unacceptable. Every second of Kafka cluster downtime can stop transaction data flows, lose business events, and fail dependent downstream systems.

For that, we must implement Rolling Upgrade strategies. Through this approach, we update broker nodes one by one in rotation. Clusters stay active serving client application writes and reads throughout maintenance processes. However, doing rolling upgrades on stateful distributed systems like Kafka requires strict protocol discipline. We must ensure new binary versions, inter-broker protocol communication versions, and on-disk message storage formats (log message formats) are updated gradually and regularly to avoid metadata incompatibilities.

In this guide, we’ll discuss rolling upgrade philosophies, unpack the 3 main stages of Kafka version upgrades, demonstrate CLI commands for emptying broker partitions (drain nodes) during hardware maintenance, and arrange post-upgrade verification checklists to guarantee cluster health.

Zero-Downtime Rolling Upgrade Philosophy #

The main goal of rolling upgrades is maintaining high cluster availability (high availability) on client sides while minimizing system failure risks. During rolling upgrade processes, our clusters are in hybrid conditions or Mixed-Version States, where some brokers run with new versions and others still run with old versions.

flowchart TD
    B1["Broker 1 (v3.7)"] <== "v3.6 Protocol" ==> B2["Broker 2 (v3.6)"]
    B1 -- "v3.6 Protocol" --> B3["Broker 3 (v3.6)"]
    B2 -- "v3.6 Protocol" --> B3

• Broker 1 has been upgraded to v3.7, but it’s forced to speak using v3.6 protocol language so Brokers 2 & 3 that haven’t been upgraded still understand each other.

So new-version brokers can communicate with old-version brokers without damaging metadata coordination, Kafka provides two special configuration parameters for locking protocol compatibility:

1. inter.broker.protocol.version: Limits the communication protocol version used between brokers. During transitions, this parameter is locked to the oldest version.
2. log.message.format.version: Limits the on-disk message record storage format version. This ensures message byte structures written on disks stay uniform so they can be read by old-version brokers.

3-Stage Rolling Upgrade Workflow #

Safe Apache Kafka version upgrade processes must go through three main sequential stages. Skipping these stages or combining them all at the start triggers unclean shutdowns or fatal quorum failures.

Stage 1: Broker Software Binary Upgrades (Broker-by-Broker) #

In this stage, we replace old Kafka binary code with new version binaries on every server, one by one.

For every broker in the cluster (do one by one in rotation):

1. Lock Protocols: Before shutting down the first broker, make sure server.properties files on all brokers have configured inter.broker.protocol.version and log.message.format.version at the current old version.
2. Stop Brokers: Cleanly turn off Kafka processes (systemctl stop kafka).
3. Binary Updates: Install new-version Kafka binary packages (e.g., replacing JAR file versions).
4. Start Brokers: Revive brokers. These new brokers run using new binaries, but are forced to communicate using old protocol languages because of the locking settings in properties files.
5. Verify: Wait until those brokers finish log synchronization (catch up) and re-enter all ISR groups before continuing to the next broker.

Stage 2: Inter-Broker Protocol Version Upgrades #

After all brokers in the cluster successfully run using new-version binaries:

1. Change Protocol Configurations: Update the inter.broker.protocol.version property values in all broker server.properties files to the new version.
2. Second Rolling Restart: Do rolling restarts once more for every broker so those new protocol settings become active. After this stage finishes, all brokers communicate using more efficient new-version protocols.

Stage 3: Message Format Version Upgrades #

After new protocols are stably active across the entire cluster:

1. Change Message Formats: Update log.message.format.version property values in all broker server.properties files to new message format versions.
2. Third Rolling Restart: Do the last rolling restart on every broker to apply new disk write formats.

Important: Starting Kafka 3.0+, default message formats are already internally managed based on inter-broker protocols for most configurations. However, for migrated old clusters, these explicit tuning steps remain crucial.

Metadata Feature Upgrades on KRaft Mode Clusters #

For modern clusters running KRaft mode (without ZooKeeper), metadata compatibility adjustment processes are no longer done by writing static parameters to properties files. KRaft simplifies these processes using Metadata Version features centrally managed inside internal metadata logs.

After all KRaft brokers are updated to new-version binaries (e.g., 3.7-version binaries):

1. Check Current Metadata Versions: Run the following command to see currently active metadata versions:

   kafka-features.sh --bootstrap-server localhost:9092 \
     --command-config /etc/kafka/client.properties \
     describe
   

2. Upgrade Metadata Features: Send metadata upgrade instructions to controller quorums without needing to restart brokers once more:

   kafka-features.sh --bootstrap-server localhost:9092 \
     --command-config /etc/kafka/client.properties \
     upgrade --metadata 3.7
   

KRaft controller quorums write these metadata change messages to internal replica logs. All KRaft data brokers read these changes in real-time and instantly activate new 3.7-version features.

Parameter Modifications: Dynamic vs Static Configurations #

When doing cluster maintenance, we must distinguish which configuration parameters require rolling restarts (Read-Only Configs) and which can be instantly changed without touching restart buttons (Dynamic Configs).

We’re advised to use kafka-configs.sh commands for directly modifying dynamic parameters:

+-------------------------------------------------------------------------------+
||                        CONFIGURATION PARAMETER CLASSIFICATION                ||
||                                                                               ||
||  1. READ-ONLY CONFIGS (Need Rolling Restarts):                                ||
||     * Properties: zookeeper.connect, listeners, log.dirs, num.io.threads      ||
||     * Verification: Must be changed in server.properties & restart brokers.   ||
||                                                                               ||
||  2. DYNAMIC CONFIGS (Without Restarts):                                       ||
||     * Properties: min.insync.replicas, max.message.bytes, log.cleaner.threads ||
||     * Verification: Change via kafka-configs.sh --alter --entity-type brokers.||
+-------------------------------------------------------------------------------+

An example of dynamically changing topic message size limits without cluster restarts:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --alter \
  --entity-type topics \
  --entity-name payment.orders \
  --add-config max.message.bytes=10485760

Partition Leadership Rebalancing (Leader Rebalancing) #

After a broker is revived post-maintenance, it rejoins clusters as followers for all partitions. This condition makes leadership loads on clusters unbalanced because non-restarted brokers bear heavier client read-write I/O loads.

By default, Kafka manages this process automatically through the following properties:

• auto.leader.rebalance.enable=true: Enables background controller threads for monitoring leadership imbalances.
• leader.imbalance.per.broker.percentage=10: Imbalance tolerance thresholds (default $10%$). If the percentage of partitions not led by preferred leaders on a broker exceeds these limits, automatic rebalances are triggered.
• leader.imbalance.check.interval.seconds=300: How often controllers check imbalances (default 5 minutes).

Large-Scale Recommendations: On very high data traffic clusters, daytime automatic rebalances can trigger network latency spikes from sudden client route shifts. We’re advised to set auto.leader.rebalance.enable=false and trigger preferred leader elections manually using cron job schedules at night when data traffic is quiet (off-peak hours).

Maintenance Automation Using Cruise Control #

For those of us managing large-scale clusters with dozens of brokers and thousands of partitions, doing partition reassignments and rolling restarts manually using CLIs is very error-prone and slow. Large industries use Cruise Control (a LinkedIn automation tool) to manage cluster lifecycles.

Cruise Control continuously monitors CPU resource usage, memory, disk I/O, and network bandwidth of every broker. When we trigger maintenance commands, Cruise Control will:

• Calculate optimal partition reassignment plans for emptying servers without burdening network cards.
• Dynamically limit data replication throughput (bandwidth throttling) so data transfers don’t disturb client application latencies.
• Do automatic broker rolling restarts by safely checking ISR statuses.

Safe Hardware Maintenance Procedures (Drain Nodes) #

Often we must turn off physical broker servers not for updating Kafka versions, but for Bare Metal hardware maintenance (like adding RAM, replacing corrupted disks, or servicing motherboards). To do this without triggering replication degradation, we must do controlled Node Drains (Partition Emptying).

Here’s the drain node process flow diagram before shutting down target servers:

flowchart TD
    Start["Start Target Broker Maintenance (e.g., Broker 1)"] --> Reassign["1. Prepare the Partition Reassignment JSON:<br/>Move replicas from Broker 1 to other brokers"]
    Reassign --> ExecReassign["2. Run the kafka-reassign-partitions.sh --execute Command"]
    
    ExecReassign --> CheckProgress{"Is the reassignment progress 100% complete?"}
    
    CheckProgress -- "No" --> Wait["Wait & Verify with --verify"]
    Wait --> CheckProgress
    
    CheckProgress -- "Yes" --> ForceLeader{"3. Move remaining leadership (Leader) statuses<br/>using preferred leader elections"}
    
    ForceLeader --> StopKafka["4. Turn off Kafka processes: systemctl stop kafka"]
    StopKafka --> Maintenance["5. Do Physical Server Maintenance"]
    
    Maintenance --> StartKafka["6. Revive the server and Kafka processes"]
    StartKafka --> RevertPart["7. Return original data partitions if needed"]
    RevertPart --> End["Maintenance Finished"]

Drain Node Working Steps via CLI: #

1. Evacuate Data Partitions #

We must move all partition replicas on target brokers (e.g., Broker 1) to other healthy brokers in the cluster. Create a reassignment configuration file named reassign.json:

{
  "version": 1,
  "partitions": [
    { "topic": "payment.orders", "partition": 0, "replicas": [2, 3, 4], "log_dirs": ["any", "any", "any"] }
  ]
}

Run the asynchronous data transfer process using the command:

kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --reassignment-json-file reassign.json \
  --execute

Monitor data transfer progress until $100%$ complete:

kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --reassignment-json-file reassign.json \
  --verify

2. Transfer Remaining Partition Leadership (Leader Elections) #

After replica transfers finish, make sure there are no active partition leadership leftovers pointing to broker 1 by triggering preferred leader elections to other brokers:

kafka-leader-election.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --election-type preferred \
  --all-topic-partitions

After all partitions and leadership are clean from Broker 1, we can safely turn off the server without triggering replication failure alarms in SRE teams.

Operational Compliance and Rolling Upgrade Audit Checklist #

Use the following operational compliance checklist when planning production cluster maintenance to minimize downtime risks:

Summary #

• Apply 3-Stage Upgrades — Always do version upgrades through three separate rotating phases: binary version upgrades, communication protocol version upgrades, and finally disk message format upgrades.
• Lock Compatibility Parameters — Use inter.broker.protocol.version settings to block new-version brokers from sending instructions not understood by old-version brokers during transition periods.
• Do Controlled Node Drains — Move all partition replicas out of target brokers using kafka-reassign-partitions.sh commands before shutting down servers for physical hardware maintenance.
• Verify ISRs Before Continuing — Never restart the next broker before newly upgraded brokers finish log recovery and re-enter cluster ISR groups.

← Previous: Broker Recovery Process Next: Centralized vs Decentralized Clusters →