Data Loss vs Availability Trade-off #

In the world of distributed systems, one of the most fundamental natural laws we must face is the CAP Theorem (Consistency, Availability, Partition Tolerance). This theorem states that when a network partition occurs, a system can only choose between maintaining data consistency or service availability. In Apache Kafka, the most tangible manifestation of this trade-off lies in how we configure failure handling when a partition leader dies suddenly, while the remaining follower replicas are not in sync status (out-of-sync). This architectural decision determines whether our system chooses data loss to keep operations running, or chooses temporary downtime to secure every bit of data.

Understanding Follower Categories: In-Sync vs Out-of-Sync #

Before discussing the architectural trade-off in depth, we must first clarify how Kafka distinguishes follower replica status.

1. In-Sync Replicas (ISR) #

The ISR is the group of follower brokers actively communicating with the leader and successfully copying data with minimal time lag. This liveness status is governed by the parameter:

$$\text{replica.lag.time.max.ms}$$

If a follower successfully sends fetch requests and doesn’t lag more than that time duration (default: 30 seconds), that follower is recognized as a valid ISR member. ISR members are considered to hold data identical to the leader.

2. Out-of-Sync Replicas (OSR) #

If a follower experiences process failure, physical crash, or severe network latency so that it fails to send fetch requests within the replica.lag.time.max.ms window, that broker is kicked out of ISR membership and enters out-of-sync status. OSR brokers are considered to hold stale data lagging behind the leader.

flowchart TD
    subgraph Kluster["Kafka Cluster (Partition 0)"]
        direction TB
        L0["Broker 1 (Leader) <br> LEO: 100"]
        F1["Broker 2 (Follower - ISR) <br> LEO: 100 <br> Fetch: 1s ago"]
        F2["Broker 3 (Follower - Out-of-Sync) <br> LEO: 80 <br> Fetch: 45s ago"]
    end

    L0 -. Replication .-> F1
    L0 -. Replication (Failed/Slow) .-> F2

    style L0 stroke:#0288d1,stroke-width:2px
    style F1 stroke:#2e7d32,stroke-width:2px
    style F2 stroke:#c62828,stroke-width:2px

The Crisis Failover Scenario #

The architectural disaster happens when Broker 1 (Leader) dies suddenly, and at the same time, the only remaining live follower is Broker 3 (which is Out-of-Sync).

This is where Kafka asks the system architect through configuration: “Do we allow Broker 3 to become the new Leader?”

Dissecting the unclean.leader.election.enable Parameter #

The unclean.leader.election.enable configuration parameter is the main switch controlling leader election behavior in the crisis scenario above. This configuration can be set at the broker level (server.properties) or overridden at the topic level dynamically.

1. Option unclean.leader.election.enable = true (Prioritizing Availability) #

When we set this option to true, we consciously allow out-of-sync replicas to be elected as the new partition leader if no original ISR member is left alive.

Advantage: High Availability #

Client write and read operations keep running. The partition doesn’t experience long downtime because as soon as one live follower broker exists (even if its data lags far behind), that broker is immediately promoted as the new leader to serve producer and consumer requests.

Fatal Risk: Absolute Data Loss & Offset Corruption #

If Broker 3 (whose LEO data lags at offset 80) is promoted as the new leader even though the old leader wrote up to offset 100:

1. Lost Messages: Messages from offsets 81 to 100 that were written on the old leader are gone forever from the cluster.
2. Log Truncation: When Broker 1 (the old leader) comes back online, it rejoins as a follower. Broker 1 sees the new leader only has data up to offset 80. To realign, Broker 1 is forced to truncate its own physical log from offsets 81 to 100.
3. New Messages Overwrite Old Offsets: If the producer sends new messages to the new leader (Broker 3), those new messages get offsets 81, 82, etc. The new messages overwrite the old offset range with completely different content. This is extremely confusing for consumer applications tracking data sequentially.

flowchart TD
    subgraph Kondisi_Awal["1. Before Leader Crash"]
        direction TB
        L_A["Old Leader (Broker 1) <br> [Offsets: 0 - 100]"]
        F_A["Out-of-Sync Follower (Broker 2) <br> [Offsets: 0 - 80]"]
    end

    subgraph Crash_Event["2. Leader Crash & Unclean Election = true"]
        direction TB
        L_Dead["Broker 1 (Dead)"]
        F_Promoted["Broker 2 (Promoted as New Leader) <br> [Offsets: 0 - 80]"]
        Note_A["Offsets 81-100 Lost from the Cluster!"]
    end

    subgraph Recovery_Event["3. Old Leader Comes Back (As Follower)"]
        direction TB
        F_New["Broker 1 (Back Online) <br> [Log Truncated to Offset 80]"]
        L_New["Broker 2 (New Leader) <br> [Receiving New Messages at Offset 81]"]
    end

2. Option unclean.leader.election.enable = false (Prioritizing Durability/Consistency) #

This is the default value in modern Apache Kafka. When the crisis scenario above happens, Kafka strictly prohibits electing a leader from out-of-sync replicas.

Advantage: Absolute Data Consistency & Durability #

Kafka guarantees no messages are silently lost due to synchronization failures, and data offset linearity is perfectly preserved. There’s no forced log truncation on the old leader broker when it recovers.

Risk: Service Downtime (Unavailability) #

While Broker 1 (or other original ISR members) hasn’t come back online, the partition is declared OFFLINE. Producer clients trying to write to this partition receive the LeaderNotAvailableException error. Consumers trying to read from this partition also get no new data. Our business service for that partition is completely down during the broker recovery period.

The Role of Leader Epoch in Log Reconciliation #

Before version 0.11, Kafka relied on the High Watermark (HW) to perform log truncation during failover. This method had an architectural flaw that could cause data loss or irregular duplication even with unclean.leader.election.enable=false.

Since version 0.11, Kafka introduced the Leader Epoch concept. Leader Epoch is an integer (sequence counter) tracking how many times the partition leader has changed since the partition was first created.

How Does Leader Epoch Protect Log Integrity? #

Every time a new leader broker is elected, the Controller increments the Leader Epoch value. The new leader records its first offset along with its new epoch number in a local leader-epoch-checkpoint file.

When the old broker (for example, the previously dead Broker 1) comes back as a follower:

1. Broker 1 doesn’t immediately truncate its log to its local High Watermark.
2. Broker 1 sends a special request called OffsetsForLeaderEpochRequest to the current active leader (Broker 2).
3. Broker 2 replies with the end offset limit for the queried epoch.
4. Based on that response, Broker 1 can determine exactly at which offset its data diverges from the current active leader, then truncates its log only at that boundary.

This mechanism prevents divergent High Watermark scenarios and guarantees follower replicas never truncate data that the cluster quorum has actually confirmed safe.

Leader Election Mechanism in the Controller: ZooKeeper vs KRaft #

The dynamic leader election process is managed by a special broker acting as the cluster Controller. How the Controller executes this election evolved significantly with Kafka’s architecture transition.

The ZooKeeper Era #

1. When a leader broker dies, ZooKeeper detects the loss of that broker’s ephemeral session.
2. ZooKeeper sends a notification (watch trigger) to the Controller.
3. The Controller reads the partition membership and ISR list from the ZooKeeper node /brokers/topics/[topic]/partitions/[partition]/state.
4. If unclean.leader.election.enable=false, the Controller filters the list of live brokers and elects the first broker in the ISR. If no ISR broker is alive, the partition is declared offline.
5. The Controller updates the new state to ZooKeeper and distributes this new data to all cluster brokers using the LeaderAndIsrRequest command.

The KRaft Era (Modern) #

In the KRaft era, this process is far more efficient because the active Controller keeps a copy of metadata directly in memory (Active Controller Metadata Image).

1. Broker failure is detected directly through the broker’s lost heartbeat to the KRaft Controller quorum.
2. The KRaft Controller leader immediately computes the new state for that partition.
3. The Controller writes the partition leadership change record (PartitionRecord) to the internal @metadata metadata log.
4. All cluster brokers update their local state instantly by asynchronously reading the metadata delta stream from that log. This eliminates the slow serial network communication latency of the ZooKeeper era.

Qualitative Comparison Analysis #

The table below summarizes the essential differences between the two configuration options above to help us formulate cluster operational policies:

Real Case Study: Data Center Power Outage #

Let’s dissect an operational failure scenario common in the industry to see how both configuration choices impact downstream database integrity.

Problem Background #

A Kafka cluster with 3 brokers experiences a power failure incident on the main server rack housing Broker 1 (Leader) and Broker 2 (Follower/ISR). Broker 3 (Follower) is on a secondary rack getting backup power from a generator, so it stays alive. However, before the power outage, the network connection between the main rack and secondary rack was disrupted for 10 minutes, causing Broker 3 to be out-of-sync (lagging 50,000 messages).

Path A: Using unclean.leader.election.enable=true #

1. Broker 1 and Broker 2 die from the power outage.
2. The cluster detects Broker 3 is the only live broker. Because the option is set true, Broker 3 is promoted as the new Leader.
3. Producer clients send new transaction messages (for example, Transaction ID 900,000). These messages are written on Broker 3 at offset 150,000.
4. Two hours later, power in the main rack returns. Broker 1 and Broker 2 come back and register as followers to Broker 3.
5. Broker 1 detects that its own transaction data at offsets 150,000 to 200,000 (containing transaction IDs 850,000 to 899,999) doesn’t exist on Broker 3.
6. Broker 1 truncates its physical log from offset 150,000 upward, deleting 50,000 valuable transactions.
7. Consumer applications read from Broker 3 and copy new data to the main SQL database.
8. Final Result: The SQL database permanently loses 50,000 transactions without any error log on the producer side. There’s a financial bookkeeping discrepancy worth billions of rupiah.

Path B: Using unclean.leader.election.enable=false #

1. Broker 1 and Broker 2 die from the power outage.
2. Broker 3 detects it’s the only live broker. However, because the option is set false, the system forbids Broker 3 from promoting itself as leader because it’s out-of-sync.
3. The topic is declared offline. Producers trying to send new transactions are rejected with the LeaderNotAvailableException error. Web applications show the page “System Experiencing Disruption, Please Try Again in a Moment”.
4. Two hours later, power returns. Broker 1 (the original Leader) comes back online.
5. The cluster detects the original ISR broker has recovered. Broker 1 is re-designated as the active Leader.
6. Writes reopen. Queued messages in producers start being sent safely back to Broker 1.
7. Final Result: Not a single transaction is lost. The system experienced 2 hours of downtime, but financial data integrity is maintained 100%.

Impact on Producer Clients: Memory Buffering #

When we choose the safe path by setting unclean.leader.election.enable=false, our producers must be configured with proper failure tolerance to handle the temporary downtime of offline partitions.

When a partition is offline, producers can’t send data. However, asynchronous Kafka producers don’t crash immediately. They have a memory buffer area to hold messages temporarily.

Important Producer Properties During Partition Downtime #

• buffer.memory: The maximum memory size (default: 33,554,432 bytes / 32MB) producers can allocate to buffer unsent message records.
• max.block.ms: The maximum duration (default: 60,000ms / 1 minute) where .send() calls are blocked when the producer’s memory buffer is full or partition metadata is unavailable. If this time limit passes and the broker hasn’t recovered, the producer throws TimeoutException.
• retry.backoff.ms: The wait interval (default: 100ms) before the producer tries sending a failed request back to the leader broker.

By understanding these parameters, we can align client architecture to safely buffer messages in local memory during transient failures without overloading the application’s JVM memory.

Configuration Guide in Production Layers #

To apply the policy choices above, here are concrete configuration instructions we can run.

1. Global Cluster-Level Configuration #

Global tuning is done on the config/server.properties file (or config/kraft/broker.properties for KRaft mode) on every cluster broker.

# DON'T: Enable this globally on a multi-purpose production cluster
# unclean.leader.election.enable=true

# CORRECT: Disable by default to maintain global data durability
unclean.leader.election.enable=false

2. Topic-Specific Configuration (Dynamic) #

Often, in one same Kafka cluster, we have topics needing high durability (e.g., payments) and topics needing high availability (e.g., debug logs). We can override the global configuration at the topic level using Kafka’s built-in CLI tool:

# Enabling Unclean Election custom-only for telemetry logging topics
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name telemetry-logs \
  --alter \
  --add-config unclean.leader.election.enable=true

# Ensuring financial transaction topics stay safe (unclean = false)
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name bank-transfers \
  --alter \
  --add-config unclean.leader.election.enable=false

# Verifying the active configuration on a topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name bank-transfers \
  --describe

Summary #

• CAP Theorem in Kafka: Manifested in the choice between absolute data resilience (Consistency) or continuous write operability (Availability).
• ISR Status: Only followers in the ISR group are considered to hold valid replica data safe to promote as the new leader.
• unclean.leader.election.enable Configuration:
  • true: Chooses availability. Allows out-of-sync followers to become the new leader. Very high data loss and log truncation risk.
  • false (Default): Chooses consistency. Strictly prohibits out-of-sync follower promotion. The partition goes temporarily offline until the original ISR broker recovers.
• Leader Epoch: The modern mechanism using leadership generation numbers to prevent log corruption (split-brain) and defective log reconciliation post-failover.
• Main Recommendation: Always set the parameter to false for all transactional topics processing high-value stateful data (financial, inventory, medical records).

← Previous: Acknowledgement Next: Replication Factor →