Replication Factor #

In modern distributed system architecture, physical infrastructure failure isn’t a matter of whether it will happen, but when. Hard disks will fail, network switches will malfunction, and data centers can suffer total power outages. To ensure systems keep operating reliably amid this barrage of failures, Apache Kafka relies on a data replication mechanism. The main component governing how many copies of our data exist across the broker cluster is called the Replication Factor (RF). Determining the ideal replica count is one of the most crucial capacity planning decisions because it directly impacts system reliability, disk storage capacity, and our network infrastructure costs.

Replication Basics and Mathematical Failure Tolerance Calculation #

The Replication Factor determines the total number of topic partition copies spread across all brokers in the Kafka cluster. These copies include one primary replica acting as the Leader (serving write and read operations by default) and the rest as Followers (passively copying data from the leader).

For example, if we create a topic with replication.factor=3, then for every partition in that topic, there will be 1 partition Leader and 2 partition Followers spread across different brokers.

Broker Failure Tolerance Formula #

The relationship between our data resilience level and the replica count can be formulated mathematically. Our cluster failure tolerance depends on the acks parameter configuration used by producers and the topic’s min.insync.replicas value.

Scenario A: Using acks=1 or acks=0 #

If producers don’t demand full replication before getting an ACK, data read availability stays maintained as long as at least one partition replica is alive.

  • Write & Read Failure Tolerance:

$$\text{Broker Crash Tolerance} = R - 1$$

Where $R$ is the replication.factor value. With $R=3$, we can tolerate up to $2$ brokers dying suddenly at once without losing data access.

Scenario B: Using acks=all (High Durability) #

If producers demand confirmation from the entire ISR quorum before declaring success, and the topic is configured with the minimum active replica limit min.insync.replicas = M, then:

  • Write Failure Tolerance (Write Availability):

$$\text{Broker Crash Tolerance} = R - M$$

If we set $R=3$ and $M=2$ (the industry-standard configuration), then:

$$\text{Tolerance} = 3 - 2 = 1\text{ Broker}$$

If only 1 broker dies, the cluster can still accept new data writes because the remaining $2$ live brokers still meet the minimum quorum limit $M$. However, if $2$ brokers die at once (leaving only 1 live broker), producer write operations are immediately blocked and replied with the NotEnoughReplicasException error, to prevent writing data that can’t be safely replicated.

How Does Kafka Determine Replica Placement? (Assignment Algorithm) #

When we create a new topic, the Kafka Controller automatically runs a replica placement algorithm to distribute partitions fairly across all available brokers. This algorithm aims to balance disk I/O and network load.

Default Placement Algorithm (Without Rack Awareness) #

Suppose we have $N$ brokers and want to place partition $P$ with replication factor $R$:

  1. Choosing the First Broker for the Leader: The Kafka Controller sorts the broker list randomly for initial initialization. For partition $i$, the leader broker is chosen with the formula:

$$\text{Leader Broker} = i \pmod N$$

  1. Spreading Follower Replicas: For replica $j$ of that partition (where $0 < j < R$), the follower broker is placed on the next broker sequentially with a shift:

$$\text{Follower Broker} = (i + j + \text{shift}) \pmod N$$

Where shift increments each time the broker list is fully circled once. This guarantees that if the first broker often becomes the leader for even-numbered partitions, it won’t be overloaded by follower replicas of other partitions.

Rack Awareness Architecture (broker.rack) #

Setting replication.factor=3 alone doesn’t guarantee data safety if all our brokers are in the same physical server rack or same virtual infrastructure (virtual hypervisor). If that server rack suffers a power supply failure, all our data replicas die simultaneously.

To prevent this disaster scenario, Kafka provides the Rack Awareness feature.

flowchart TD
    subgraph DC["Cloud Data Center (Region: ap-southeast-3)"]
        direction LR
        
        subgraph AZ_A["Availability Zone A <br/>'(broker.rack=zone-a)'"]
            B1["Broker 1 <br/>Partition 0 (Leader)"]
        end

        subgraph AZ_B["Availability Zone B <br/>'(broker.rack=zone-b)'"]
            B2["Broker 2 <br/>Partition 0 (Follower)"]
        end

        subgraph AZ_C["Availability Zone C <br/>'(broker.rack=zone-c)'"]
            B3["Broker 3 <br/>Partition 0 (Follower)"]
        end
    end

    B1 -. "Cross-AZ Replication" .-> B2
    B1 -. "Cross-AZ Replication" .-> B3
    style B1 stroke:#0288d1,stroke-width:2px
    style B2 stroke:#2e7d32,stroke-width:2px
    style B3 stroke:#2e7d32,stroke-width:2px

How Does Rack Awareness Work? #

  1. We configure the broker.rack property on each broker’s configuration file to identify its physical location (for example, the physical rack name in a local data center, or the Availability Zone name on cloud providers like us-east-1a, us-east-1b, etc.).
  2. When we create a new topic, the Kafka Controller reads the rack configuration from all brokers.
  3. The Controller intelligently spreads replicas of the same partition across brokers with different broker.rack values.
  4. Guarantee: No two replicas of the same partition are placed in the same rack/zone, unless the replica count exceeds the number of available racks.

With this architecture, if one Availability Zone suffers a massive total power outage, our Kafka cluster is guaranteed to stay alive and consistent because backup replicas are in different safe zones.

Cross-Availability Zone Network Cost Analysis #

Cross-Availability Zone (AZ) replica distribution is a mandatory practice for high reliability, but there’s a significant financial cost consequence we must anticipate carefully.

Cloud providers (like AWS, GCP, or Azure) charge for every gigabyte of data flowing out (egress) from one AZ to another in the same region (for example: $0.01 per GB).

Real Case Calculation #

Let’s estimate network traffic costs for a production cluster with the following specs:

  • Producer Throughput: 500 MB/second (equivalent to 1.8 TB/hour or 43.2 TB/day of clean data).
  • Replication Factor: 3 (distributed across 3 separate AZs).
  • Cross-AZ Network Cost: $0.01 per GB ($10 per TB) for AZ inbound and outbound data transfer.

Scenario 1: Without Follower Fetching Optimization (Classic Reading) #

  1. Replication Traffic: Messages are written to the Leader Broker in AZ-A, then pulled by Follower 1 in AZ-B (43.2 TB/day) and Follower 2 in AZ-C (43.2 TB/day). Total cross-AZ replication traffic: 86.4 TB/day.
  2. Consumer Traffic: If our consumers are deployed randomly (e.g., 50% of reads come from instances in AZ-B or AZ-C reading to the Leader in AZ-A), there’s additional outbound data transfer of:

$$43.2 \text{ TB/day} \times 66.6% \approx 28.8 \text{ TB/day}$$

  1. Total Cost:
    • Total cross-AZ traffic: $86.4 + 28.8 = 115.2 \text{ TB/day}$.
    • Daily cost: $115.2 \text{ TB} \times $10 = $1,152 \text{ per day}$.
    • Monthly cost: $34,560 per month (~Rp518,000,000) just for internal network costs!

Scenario 2: With Follower Fetching Optimization #

By enabling Follower Fetching, consumer applications in AZ-B only read from Follower 1 in AZ-B (local AZ), and consumers in AZ-C read from Follower 2 in AZ-C.

  • Cross-AZ replication traffic stays 86.4 TB/day.
  • Cross-AZ consumer traffic drops to 0 TB/day (because all consumers read locally in their respective AZs).
  • New Total Cost: $86.4 \text{ TB} \times $10 = $864 \text{ per day}$ ($25,920 per month).
  • Financial Savings: We save $8,640 per month (~Rp130,000,000) with just one simple configuration change.

Physical Log Write Mechanism to Disk: Page Cache vs fsync #

The durability level of the replication factor also depends on how brokers write data to disk storage.

By default, when a Kafka broker receives a new message (whether a leader from a producer or a follower from a leader), the broker only writes that data to the OS Page Cache (RAM) using standard JVM system calls, without invoking the fsync instruction to force a direct write to the physical disk platter.

flowchart LR
    Producer["Producer Client"] --> Socket["Network Socket"] --> Cache["Broker OS Page Cache (RAM)"] --> ACK["ACK Response"]

Why Does Kafka Recommend Refusing Synchronous fsync? #

Many traditional databases force synchronous fsync after every transaction to guarantee safety. In Kafka, we can configure this through the log.flush.interval.messages and log.flush.interval.ms properties. However, Kafka strongly advises leaving these parameters at their default values (unlimited / fully delegated to the operating system).

The main reasons are:

  1. Performance Degradation: Synchronous fsync calls turn very fast sequential memory writes into very slow blocking disk I/O operations, dropping Kafka throughput by up to 90%.
  2. Safety Through Cross-Broker Replication: Kafka shifts the data safety guarantee from the single-node physical hardware level to the multi-node active replication level. Power loss on one broker stays safe because 2 other brokers hold the same data in their memory. The chance of 3 separate brokers losing power simultaneously is very small (especially with Rack Awareness).

Step-by-Step Simulation of Failure and Replication Sync #

Let’s chronologically simulate how the cluster manages replication during a physical disruption.

Phase 1: The Disruption Happens #

  1. Broker 1 (Partition 0 Leader) crashes from a power supply failure.
  2. The Controller quorum detects the loss of Broker 1’s heartbeat.
  3. The Controller appoints Broker 2 (longest-serving ISR Follower) as the new Leader. The Leader Epoch rises from 1 to 2.
  4. The ISR membership for Partition 0 shrinks to: [2, 3].

Phase 2: Re-synchronization During Recovery #

  1. The operations team replaces Broker 1’s hardware component and powers it back on.
  2. Broker 1 starts and re-registers with the Controller.
  3. Broker 1 realizes it’s no longer the leader. It sends an OffsetsForLeaderEpochRequest to Broker 2 asking for the end offset of Epoch 1.
  4. Broker 2 replies that Epoch 1 ended at offset 10,000. Broker 1 checks its own log and detects it has data up to offset 10,050 (there are 50 messages written locally before the crash that were never replicated).
  5. Broker 1 truncates its log back to offset 10,000 to realign, then starts pulling data from Broker 2 starting at offset 10,001.
  6. After Broker 1’s LEO successfully catches up and stays within the time difference under replica.lag.time.max.ms, Broker 1 is automatically re-added to the ISR quorum: [1, 2, 3].

Alignment for Multi-Region Disaster Scenarios (Disaster Recovery) #

So far, we’ve discussed replication within one cluster (intra-cluster replication). For systems with extra-high failure tolerance levels (like core banking services), we must anticipate the destruction of an entire geographic region (region outage).

For that, we must implement inter-cluster replication across regions:

1. MirrorMaker 2 (MM2) #

MM2 is an asynchronous replication tool based on Kafka Connect that reads data from the primary cluster in Region-A and republishes it to the backup cluster in Region-B.

  • Weakness: Asynchronous replication means there’s replication latency (RPO — Recovery Point Objective isn’t zero; there’s a risk of losing the last few seconds of data if Region-A is totally destroyed).

2. Confluent Cluster Linking #

Modern technology enabling direct inter-broker replication across regions without needing an external Kafka Connect engine. Replication happens at the broker-to-broker protocol level directly, keeping message offsets identical in both regions dynamically (active-passive failover).

Operational CLI: Creating & Changing the Replication Factor #

Here are practical command-line instructions for managing replication configuration at the production level.

1. Creating a New Topic with a Specific Replication Factor #

# Creating a payment topic with 6 partitions and replication 3
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create \
  --topic payment-transactions \
  --partitions 6 \
  --replication-factor 3 \
  --config min.insync.replicas=2

2. Changing the Replication Factor of a Running Topic with Throttling #

Large replica movement processes can clog cluster network. We must apply bandwidth throttling during the migration process.

Step A: Create the Reassignment JSON File (reassignment.json) #

Create a configuration file defining the new partition mapping to target brokers manually:

{
  "version": 1,
  "partitions": [
    {
      "topic": "customer-clicks",
      "partition": 0,
      "replicas": [1, 2, 3],
      "log_dirs": ["any", "any", "any"]
    }
  ]
}

Step B: Run Reassignment with Bandwidth Throttling #

We limit the maximum data replication speed to 50 MB/s (52,428,800 bytes/s) so it doesn’t disturb active producer data traffic:

# Run the execution command with a 50MB/s throttle
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --throttle 52428800 \
  --execute

Step C: Verify and Clean Up Throttle #

After the verification status shows success, we must remove the throttle so broker bandwidth configuration returns to unlimited mode:

# Verify migration status and automatically remove the throttle after success
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Summary #

  • Replication Factor: Determines the total number of topic partition copies across the Kafka broker cluster to anticipate physical infrastructure failures.
  • Mathematical Tolerance: With RF = $R$ and Min ISR = $M$, our write failure tolerance is $R - M$ broker crashes.
  • Production Standard: Use replication.factor=3 combined with min.insync.replicas=2 as the standard configuration for valuable transactional data.
  • Rack Awareness: The crucial feature for spreading partition replicas evenly across different Availability Zones using the broker.rack parameter.
  • Under-Replicated Partitions: Continuously monitor the JMX UnderReplicatedPartitions metric to detect replication failures in our cluster.
  • Follower Fetching: The modern solution for cutting cross-AZ network traffic costs by allowing consumers to read from the nearest follower replica in the same zone.
← Previous: Data Loss
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact