Production Failure Scenarios: Overcoming Critical Failures in Production #

Operating Apache Kafka at production scale means we must be ready to face the reality that infrastructure failures aren’t a matter of “if”, but “when”. As a stateful distributed system processing sensitive data in real-time, Kafka clusters are vulnerable to various physical and network disasters—from sudden server deaths, storage disk sector damage, network partitions, to flapping networks triggering metadata coordination inconsistencies.

When critical failures happen in the middle of the night, operations teams must not make decisions based on instinct. Forcefully restarting desynchronized leader brokers or forcing unclean leader elections can be fatal to permanent business transaction data loss. We must precisely understand how internal Kafka protocols respond to every failure scenario and the safe mitigation steps for recovering services.

In this guide, we’ll dissect the four most frightening production failure scenarios in Apache Kafka: sudden broker deaths (sudden crashes), split-brain disasters on KRaft/ZooKeeper quorums, local disk failures on JBOD, and unstable network latency (network flapping), complete with safe recovery instructions.

Production Failure Classification in Apache Kafka #

Before entering per-scenario technical discussions, let’s map Kafka failures into two main categories:

  1. Hard Failures: Totally dead nodes, power outages, burned disks, or physically disconnected network cables. These failures are easy to detect by watchdog systems because nodes immediately stop responding.
  2. Soft/Partial Failures: Disks experiencing bad sectors (I/O becomes very slow but doesn’t die), network ports experiencing $30%$ packet loss, or extreme CPU overloads. These failures are far more dangerous because nodes look alive (liveness probes succeed) but can’t correctly serve traffic, triggering domino effects for cluster stability.

Scenario 1: Sudden Broker Death (Sudden Broker Crash) #

Sudden broker deaths (from power failures, OS kernel panics, or JVM OOM crashes) force clusters to do instant partition leadership reorganizations.

+-------------------------------------------------------------------------------+
||                       IMPACT OF SUDDEN CRASH ON PARTITION LEADERS            ||
||                                                                               ||
||  * INITIAL GROUP (Broker 1=Leader, Broker 2=Follower, Broker 3=Follower)      ||
||    - Clients write data to Broker 1.                                          ||
||                                                                               ||
||  * BROKER 1 SUDDENLY CRASHES:                                                 ||
||    1. Controllers detect the loss of Broker 1 heartbeats.                     ||
||    2. Controllers elect a new Leader from the ISR list (e.g., Broker 2).      ||
||    3. Clients are directed to write to Broker 2 transparently.                ||
||                                                                               ||
||  * UNCLEAN LEADER ELECTION DANGERS (If Brokers 2 & 3 are outside the ISR):    ||
||    - If unclean.leader.election.enable=true  -> Broker 2 becomes the new leader ||
||      but old unreplicated data will be LOST forever.                          ||
||    - If unclean.leader.election.enable=false -> Partition goes offline (Safe). ||
+-------------------------------------------------------------------------------+

1. How Kafka Responds #

When leader brokers die:

  1. Controller quorums detect the loss of heartbeats from those brokers.
  2. Controllers immediately remove dead brokers from In-Sync Replicas (ISR) lists for all partitions they led.
  3. Controllers elect one remaining healthy follower broker in ISR lists to become new leaders.
  4. Producer and consumer clients receive new cluster metadata updates and automatically reroute their connections to new leaders without manual application-level interventions.

2. Unclean Leader Election Dangers #

The real disaster happens if, when leaders die, all remaining follower brokers are lagging outside ISR lists (e.g., from previous replication delays). If this happens, we face the unclean.leader.election.enable configuration dilemma:

  • Set to true: Kafka is allowed to elect out-of-ISR followers as new leaders. The advantage: partitions stay active serving data writes (high availability). The disadvantage: all data not yet replicated by old leaders to those followers is permanently deleted when old leaders come back, triggering offset desynchronizations (data loss).
  • Set to false (Highly Recommended for Financial Data): Kafka forbids out-of-ISR leader elections. Those partitions immediately go Offline and can’t be accessed for read-write until original leaders with complete data are successfully revived. This guarantees absolute data durability and consistency at the cost of temporary availability.

3. Identifying Partition Status with the kafka-topics.sh CLI #

We can verify partition statuses impacted by broker deaths using the following CLI command:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --describe \
  --topic payment.orders

Example Output when Broker 1 is Dead:

Topic: payment.orders   TopicId: xF23s1   PartitionCount: 3   ReplicationFactor: 3   Configs: min.insync.replicas=2
    Topic: payment.orders   Partition: 0    Leader: 2   Replicas: 1,2,3   Isr: 2,3    OfflineReplicas: 1
    Topic: payment.orders   Partition: 1    Leader: 3   Replicas: 2,3,1   Isr: 2,3    OfflineReplicas: 1
    Topic: payment.orders   Partition: 2    Leader: 2   Replicas: 3,1,2   Isr: 2,3    OfflineReplicas: 1

In the output above, broker 1 is recorded in the OfflineReplicas column and automatically removed from the Isr column, but partitions still have active leaders (brokers 2 and 3) so no downtime happens.

Scenario 2: Split-Brain on Metadata Quorums (KRaft / ZooKeeper) #

Split-Brain is a condition where network partitions divide Kafka clusters into two isolated regions, and each region feels entitled to act as the active controller leader.

1. How Kafka Protocols Prevent Double Writes #

To prevent mutually colliding data writes (split-brain writing), Kafka uses Leadership Epoch mechanisms (on data brokers) and Quorum Majority concepts (on metadata controllers).

Suppose we have a metadata cluster with 5 controller nodes. Based on distributed system theory, the minimum node count for forming a majority region (quorum) is: $$\text{Minimum Quorum} = \lfloor \frac{N}{2} \rfloor + 1 = \lfloor \frac{5}{2} \rfloor + 1 = 3 \text{ nodes}$$

If network partitions divide the cluster into two regions: Region A (2 controllers) and Region B (3 controllers):

  • Region A (Minority): Can’t form quorums (only 2 of the 3 minimum). This region automatically deactivates itself and rejects all metadata change requests from admins.
  • Region B (Majority): Successfully forms quorums (3 nodes). This region stays active and can elect new controller leaders if old leaders are in Region A.

When old leaders in Region A try sending instructions to brokers, brokers check the Controller Epoch values on those instructions. Because Region B raised epochs when electing new leaders, brokers reject all instructions from old Region A leaders because their epochs are already expired (fencing out the old leader).

Scenario 3: Local Disk Failures (Bad Disks) on JBOD #

In JBOD scheme deployments, we register several independent physical disk platters to broker log.dirs properties. Local disk failures can range from totally dead disks to congested disk I/O conditions (disk hangs).

flowchart TD
    Start["Detect Data Disk I/O Activity"] --> IO_Check{"Did I/O operations succeed?"}
    
    IO_Check -- "Yes" --> Normal["Broker runs normally"]
    IO_Check -- "No (Input/Output Error)" --> DiskFail["LogManager: Detect disk directory failures"]
    
    DiskFail --> MarkOffline["Mark those directories as OFFLINE"]
    MarkOffline --> KeepBrokerAlive["Keep the Broker JVM process ALIVE"]
    
    KeepBrokerAlive --> CheckPartitions{"Are partitions on the corrupted disk LEADERS?"}
    
    CheckPartitions -- "Yes" --> LeaderMigrate["Send metadata to the Controller:<br/>Move partition leadership to other brokers"]
    CheckPartitions -- "No" --> KeepFollower["Deactivate follower replicas on those disks"]
    
    LeaderMigrate --> Alert["Send SRE Alerts: 'Offline Log Directory' for disk replacements"]
    KeepFollower --> Alert

1. How Kafka Handles Offline Log Directories #

When one physical disk (e.g., /mnt/data2) experiences read-write sector failures, Kafka will:

  1. Record IOException errors on server logs.
  2. Change those directory statuses to Offline Log Directories.
  3. Brokers are kept alive. Brokers don’t do total shutdowns if there are still other healthy data disks inside log.dirs properties.
  4. For all partitions with leaders on those corrupted disks, brokers send signals to controllers to move leader election status to healthy follower brokers on other servers.
  5. Brokers only reject read-write requests for partitions on those corrupted disks, while partitions on other healthy disks keep running at $100%$ normal.

2. Disk Hang Dangers (I/O Blockage) #

Disk hang cases are far more dangerous than total disk deaths. In disk hangs, operating systems don’t immediately throw IOException errors. Instead, Kafka disk write threads are blocked forever waiting for confirmations from OS kernels currently trying to retry writes on damaged hardware.

  • Impact: Broker write threads stall, causing request queues to pile up. Brokers fail to respond to heartbeats and are considered dead by controllers, even though their JVM processes still look active.
  • Solution: We must configure OS disk timeout parameters (/sys/block/sd[x]/device/timeout) to low values (e.g., 10-20 seconds) so OS kernels immediately throw I/O errors to Kafka rather than holding them too long at kernel levels.

Scenario 4: Network Flapping #

Network flapping is a network failure where inter-broker connectivity frequently disconnects and reconnects within seconds constantly (ping-pong connections).

1. Why Flapping Is Very Damaging to Kafka Clusters? #

When network connections between follower brokers (e.g., Broker 3) and leader brokers (Broker 1) disconnect for several seconds:

  • Broker 3 is removed from ISR lists by leaders for lagging.
  • One second later, connections reconnect. Broker 3 does lightning-fast synchronization and re-enters ISR lists.
  • One second later, connections disconnect again.

This recurring ISR exit-entry cycle (ISR Churn) forces massive metadata change writes to controller quorums every second.

  • This process burdens controller heap memories and network processor threads.
  • If left alone, clusters experience metadata processing congestion (metadata lag), causing cluster coordination to temporarily stop and triggering client authentication failures.

2. Flapping Dampening Parameters (ISR Churn Dampening) #

To dampen flapping effects, Kafka provides the replica.lag.time.max.ms parameter (default 30000 ms or 30 seconds). This parameter determines how long followers may lag behind leaders before officially being declared out of the ISR.

  • Tuning: Don’t set this parameter too small (e.g., below 5 seconds) for fluctuating multi-region network clusters, to give temporary network pause tolerances without triggering unnecessary ISR exit-entry processes.

Leadership Recovery Procedures (Leader Election Recovery) #

After recovering crashed broker nodes or resolving network problems, we must restore partition leadership status so it spreads evenly as before (balanced clusters).

1. Triggering Preferred Leader Elections #

By default, Kafka has the auto.leader.rebalance.enable=true parameter trying to automatically balance leadership. However, this automatic rebalance process can trigger I/O jolts. We’re advised to trigger it controlledly using the following CLI:

kafka-leader-election.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --election-type preferred \
  --all-topic-partitions

2. Emergency Scenarios: Forcing Unclean Leader Elections #

If incidents happen where all followers are outside ISR lists and we choose to sacrifice consistency for reviving systems (recovery speeds), we can force unclean leader elections for specific partitions:

kafka-leader-election.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --election-type unclean \
  --topic payment.orders \
  --partition 0

Warning: The command above causes some old data on previous leader disks to be permanently deleted.

Production Incident Handling Checklist (Incident Response Checklist) #

When cluster alarms sound in the middle of the night, follow the incident response emergency guide below in order to safely diagnose and recover clusters:

KAFKA FAILURE MITIGATION EMERGENCY STEPS:

[ STEP 1: IDENTIFICATION ]
  □ Check UnderReplicatedPartitions and OfflinePartitionsCount metrics in Grafana.
  □ If OfflinePartitionsCount > 0, there are partitions that can't be accessed at all!

[ STEP 2: BROKER LOCALIZATION ]
  □ Run server log search queries to find keywords:
    - "Shrinking ISR" -> To track brokers disconnected by network/GC pauses.
    - "IOException" or "offline log directory" -> To track physical disk damage.
    - "Epoch" or "Denied Operation" -> To track quorum/security problems.

[ STEP 3: VERIFY KRAFT / ZK QUORUMS ]
  □ Make sure the majority of controller nodes are active. If clusters lose quorums 
    (e.g., 3 of 5 controllers die), DON'T restart data brokers before recovering 
    controller quorums first!

[ STEP 4: KAFKA DISK RECOVERY ]
  □ If log directories are offline:
    1. Evacuate data partitions to other brokers using kafka-reassign-partitions.sh.
    2. Replace corrupted physical disks.
    3. Format new disks with XFS file systems and mount them with appropriate flags.
    4. Revive brokers to trigger dynamic re-replication.

Summary #

  • Understand Unclean Election Policies — Set unclean.leader.election.enable=false in production to prevent permanent data loss during partition leadership failures.
  • Maintain Controller Quorums — Always maintain odd controller node counts ($2N+1$) to anticipate split-brain network failures without triggering double writes.
  • Tune OS Kernel Timeouts — Set operating system disk I/O timeout limits to low values to speed up disk hang detection and trigger Kafka offline log directory transitions.
  • Dampen Network Flapping — Set replica.lag.time.max.ms values logically to give temporary connectivity pause tolerances without triggering metadata-burdening ISR exit-entry processes.

← Previous: Kafka on Docker & Kubernetes Next: Broker Recovery Process →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact