In-Sync Replica #

In distributed data storage systems that rely on redundancy, one of the biggest engineering challenges is defining when a backup replica is considered “healthy” and truly in sync with the primary server (Leader). In Apache Kafka, the group of replicas meeting these strict data alignment criteria is called In-Sync Replicas (ISR). The ISR acts as a dynamic quorum guaranteeing our data’s availability and durability. Through this article, we’ll thoroughly explore the criteria for how a follower replica is categorized as entering or leaving the ISR group, how the leader dynamically manipulates the ISR list, the close relationship between the producer’s acks=all parameter and the topic’s min.insync.replicas parameter, and how to monitor the crucial Under-Replicated Partitions (URP) metric to maintain our Kafka cluster’s operational stability.

What Are In-Sync Replicas (ISR)? #

Simply put, In-Sync Replicas (ISR) is the subset of all partition replicas (including the Leader) that are currently actively keeping up with the Leader’s latest data writes. If we have a topic with replication factor 3, then in ideal conditions, all three replicas (for example on Broker 1, Broker 2, and Broker 3) are ISR members.

However, ISR membership status is dynamic. A Follower is only recognized as part of the ISR if it meets the following two main criteria:

1. Network Liveness Criterion #

The Follower must stay continuously connected to the cluster and periodically send data pull requests (FetchRequest) to the Leader. If the Follower totally crashes, the OS loses power, or the JVM process freezes from an overly long Garbage Collection (GC pause), it stops sending heartbeats and fetch requests to the Leader.

2. Data Accuracy Criterion (Lag Limit) #

The Follower must not lag too far behind the Leader in data synchronization. The single most important parameter controlling this criterion is replica.lag.time.max.ms (default 30000 milliseconds or 30 seconds).

This criterion is measured by time: if a Follower replica fails to synchronize until its local LEO offset matches the Leader’s LEO value within the specified window (e.g., 30 seconds), that replica is marked slow and removed from the ISR group.

In the past (Kafka 0.9 and below), there was a replica.lag.max.messages parameter that measured lag by message count (e.g., if lagging 4000 messages, remove from ISR). However, this parameter was dropped because it often triggered cluster instability during sudden traffic spikes; producers send messages so fast they exceed instantaneous replication capacity, so even healthy followers were considered out of sync and mass-kicked from ISR, causing coordination panic in the cluster. The time-based parameter (replica.lag.time.max.ms) is far fairer and more stable in assessing follower health.

Replication Lifecycle: ISR Shrinking and Expanding #

The ISR membership change process is dynamically managed by the broker acting as the partition Leader and recorded in cluster metadata through asynchronous interaction:

1. ISR Shrinking #

When a Follower experiences heavy disk I/O bottlenecks or partial network isolation, its data transfer speed slows down.

• The Leader continuously monitors each Follower’s LEO position through incoming fetch requests.
• If the Leader detects that a Follower’s LEO never catches up to the Leader’s LEO for longer than the replica.lag.time.max.ms limit, the Leader marks that Follower as leaving the ISR.
• The Leader sends an ISR status update request (Alter Partition Replicas) to the Controller.
• The Controller writes the new ISR list to cluster metadata (KRaft @metadata topic or ZooKeeper), then propagates this latest metadata to all brokers in the cluster so producers know about it.

2. ISR Expanding #

When the Follower’s disruption is resolved (for example, the network stabilizes or the GC pause finishes), the ReplicaFetcherThread on that Follower resumes marathon data pulling.

• The Follower copies all lagging data from the Leader’s log.
• Once the Follower’s LEO catches up to the Leader’s LEO, the Leader realizes the Follower’s data is back in sync.
• The Leader adds that Follower back into the ISR group.
• The Leader sends an ISR status update request to the Controller again for official recording in cluster metadata.

Let’s look at a visualization of this follower replica status transition in the diagram below:

flowchart TD
    subgraph ISRGroup["In-Sync Replicas (ISR) Group"]
        direction TB
        ActiveFollower["Active Follower <br/> (In Sync / LEO Catching Up)"]
    end

    subgraph OutGroup["Outside ISR (Out-of-Sync)"]
        direction TB
        LaggingFollower["Lagging Follower <br/> (Lag > replica.lag.time.max.ms)"]
    end

    ActiveFollower -->|"1. Network Disruption / I/O Wait Occurs"| LaggingFollower
    LaggingFollower -->|"2. Sync Process Successfully Catches Up to Leader LEO"| ActiveFollower

    style ActiveFollower fill:#ddffdd,stroke:#88ff88
    style LaggingFollower fill:#ffdddd,stroke:#ff8888

Reliability Collaboration: acks=all and min.insync.replicas #

Understanding ISR becomes crucial when discussing zero data loss delivery guarantees. The producer’s acks=all (or acks=-1) parameter is designed to ensure data is successfully written to several servers before being considered successful. However, acks=all doesn’t work alone; it heavily depends on the topic configuration min.insync.replicas.

The Trap Without min.insync.replicas #

Imagine we create a topic with replication factor 3 (Broker 1, Broker 2, Broker 3). The producer sends data with acks=all. We don’t set min.insync.replicas (defaults to 1).

• Problem Scenario: Broker 2 and Broker 3 die physically. The ISR list automatically shrinks until only Broker 1 (Leader) remains.
• Producer Behavior: When the producer sends a message with acks=all, the Leader broker (Broker 1) checks the ISR list. Since the current ISR member is only 1 node (itself), the Leader immediately writes the data to its local log and sends a success response to the producer. The producer considers the data safe because it was replicated to all active ISR members.
• Failure Consequences: If Broker 1 then totally crashes before Broker 2 and Broker 3 come back, the data the producer just sent is lost forever. We’ve lost data even using the safest parameter acks=all.

Solution: Setting min.insync.replicas Safely #

To prevent the bad scenario above, we must set the topic-level or global broker-level configuration:

$$\text{min.insync.replicas} = 2$$

When this configuration is active alongside acks=all:

• If the number of active ISR members is at least 2 (for example, Broker 1 and Broker 2 are alive), producer data write activity is accepted and processed normally.
• If the ISR member count drops below 2 (for example, only Broker 1 remains because the others died), the Leader broker rejects producer writes and throws the NotEnoughReplicasException error.
• Our producer application becomes aware of this failure and can perform emergency handling (like holding data in memory or temporarily redirecting storage to a local database) instead of letting data disappear unnoticed.

Durability vs Availability Comparison #

Setting this parameter combination is a trade-off between data durability and system availability:

Based on the table above, the RF = 3, min.insync.replicas = 2, acks = all configuration is the most recommended industry standard for production systems because it offers the best failure tolerance balance (able to tolerate 1 server death without stopping the system).

Monitoring the Under-Replicated Partitions (URP) Metric #

In daily operations, the most critical JMX monitoring metric for detecting Kafka cluster health problems is UnderReplicatedPartitions (often abbreviated URP).

What Does URP > 0 Mean? #

The URP metric counts how many partitions on a broker have fewer active replicas than the configured replication.factor value.

For example, a partition has replication factor 3, but the current ISR list only contains 2 brokers. That partition is reported as Under-Replicated.

Why Should URP Be Watched? #

Having partitions with URP status means our cluster is running in a vulnerable condition. If one more broker failure happens on the affected partitions, we lose full failure tolerance, triggering write downtime if min.insync.replicas is set to 2, or even permanent data loss.

We must install an alerting system on our monitoring tools (like Prometheus/Grafana or Datadog) if the URP metric stays above 0 continuously for more than 5 minutes. Temporary URP often occurs during routine rolling upgrades when brokers are intentionally shut down one by one, but long-persisting URP signals serious disk hardware problems or network congestion requiring manual intervention from our infrastructure team.

Anti-pattern vs Solution in Reliability Configuration #

Here are common cluster reliability configuration mistakes along with example Java error handling code:

Case: Rejecting Writes Due to ISR Shrinkage #

When our cluster experiences hardware failure until the ISR member count drops below the minimum, producers receive the NotEnoughReplicasException error. Developers who don’t anticipate this let their applications crash fatally.

Let’s compare the wrong and right error handling implementations:

// =========================================================================
// ANTI-PATTERN: Ignoring NotEnoughReplicasException handling
// The application isn't ready for quorum failure and lets data disappear.
// =========================================================================
public void sendTransactionDataAntiPattern(KafkaProducer<String, String> producer, ProducerRecord<String, String> record) {
    // ✗ DON'T: Send without an asynchronous callback or recovery plan when the cluster rejects writes.
    // If min.insync.replicas isn't met, this send triggers an unhandled exception.
    producer.send(record); 
}

// =========================================================================
// THE CORRECT SOLUTION: Resilient Design with Fallback Storage
// We monitor quorum failures asynchronously and save data to local storage
// while the Kafka cluster is in the recovery process (auto-recovery).
// =========================================================================
import org.apache.kafka.clients.producer.Callback;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class ReliableDataIngestor {
    private static final Logger log = LoggerFactory.getLogger(ReliableDataIngestor.class);

    public void sendDataWithFallback(KafkaProducer<String, String> producer, ProducerRecord<String, String> record) {
        producer.send(record, new Callback() {
            @Override
            public void onCompletion(RecordMetadata metadata, Exception exception) {
                if (exception != null) {
                    if (exception instanceof org.apache.kafka.common.errors.NotEnoughReplicasException || 
                        exception instanceof org.apache.kafka.common.errors.NotEnoughReplicasAfterAppendException) {
                        
                        // ✓ CORRECT: ISR quorum not met. Don't throw the data away!
                        // Redirect data to local emergency storage (e.g., local disk, SQLite, or local queue)
                        log.error("Failed to send data because the ISR quorum shrank below min.insync.replicas: " 
                            + exception.getMessage());
                        saveToLocalQueue(record);
                        
                    } else {
                        log.error("Delivery failed due to a non-quorum error: " + exception.getMessage());
                        saveToLocalQueue(record);
                    }
                } else {
                    log.info("Data successfully committed in ISR. Partition: {}, Offset: {}", 
                        metadata.partition(), metadata.offset());
                }
            }
        });
    }

    private void saveToLocalQueue(ProducerRecord<String, String> record) {
        // Implement local emergency storage logic for later resend (retry)
        System.out.printf("✓ Saving message with key '%s' to local emergency storage.%n", record.key());
    }
}

Summary #

• In-Sync Replicas (ISR) — The set of partition replicas (including the Leader) actively copying the latest data from the Leader and considered in sync by the system.
• Time Lag Criterion — A Follower is considered out of ISR if it fails to synchronize data to match the Leader LEO within the replica.lag.time.max.ms parameter window (default 30 seconds).
• Dynamic Alter ISR Process — The partition Leader detects follower lag directly and sends Alter Partition Replicas commands to the Controller to change ISR status in metadata.
• min.insync.replicas — The control parameter for the minimum number of active ISR replicas that must confirm data writes before the broker returns a success signal to the producer.
• acks=all — The producer parameter demanding write confirmation from all active ISR members, must be paired with min.insync.replicas=2 for zero data loss guarantees.
• Under-Replicated Partitions (URP) — The crucial JMX monitoring metric detecting cluster partitions that lost redundancy due to dead brokers or disk I/O issues.
• Failure Tolerance — The safest industry-standard production configuration is Replication Factor = 3, min.insync.replicas = 2, and acks = all.

← Previous: Leader & Follower Next: Zookeeper vs KRaft →