Multi-Cluster Strategy & MirrorMaker 2.0: Building Cross-Region Architectures #

When the systems we build grow into the main foundations of company business operations, relying on one single Apache Kafka cluster in one data center becomes a big risk. Natural disasters, regional network failures from cloud providers, or total power outages in data centers can instantly cripple all our producer and consumer applications. To guarantee business continuity, we must step to the next level: designing multi-cluster architectures spread across different geographic regions.

Building multi-cluster architectures in Apache Kafka isn’t as easy as doing ordinary relational database replication. Kafka’s characteristics relying on high throughput and offset-based data consumption demand we deeply understand how data is synchronized, how consumer offsets are diverted during disasters, and how to prevent endless replication loops. In this guide, we’ll dissect various multi-cluster strategies, MirrorMaker 2.0 architectures, operational configurations, consumer offset translation, and Disaster Recovery Failover strategies.

Multi-Cluster Strategy Needs in Production #

Before reviewing technical aspects, let’s map the main business and operational scenarios underlying our organizations’ needs to use more than one Kafka cluster:

1. Disaster Recovery (DR) #

This is the most common reason. If our main cluster in the Jakarta Region (Primary) experiences total failure, we must have a backup cluster in the Singapore Region (Disaster Recovery) ready to replace the main cluster’s role with minimal data loss (low RPO) and the shortest possible recovery times (low RTO).

2. Data Localization and Regulation Compliance #

Many legal jurisdictions stipulate that citizens’ personal data must not leave those countries’ territorial boundaries. With multi-cluster strategies, we can operate local clusters in every country to handle sensitive data locally, then send masked or aggregated data to central clusters at headquarters.

3. Edge-to-Cloud Ingestion (Hub-and-Spoke) #

In modern IoT or retail architectures, we might have thousands of edge devices (like physical store branches or telecommunication towers) each running small Kafka clusters to guarantee offline operations. Data from these edge clusters is then asynchronously synchronized to giant central cloud clusters for batch analytics processing.

4. Reducing User Latency #

Putting all Kafka brokers on one continent slows response times for users in other parts of the world. By placing clusters near users (e.g., one in the United States, one in Europe, and one in Asia), local producer applications can write data with very low latency, while cross-region synchronization is done in the background through asynchronous replication pipelines.

Multi-Cluster Replication Topologies #

There are three main logical topology patterns for geographically connecting our Kafka clusters. Each has its own trade-offs between operational complexity and system resilience.

1. Active-Passive Topologies (DR Hot Standby) #

In this topology, all producer and consumer applications only write and read from one main cluster (Active Cluster). Meanwhile, backup clusters (Passive Clusters) only receive replication result data from main clusters in the background.

flowchart TB
    subgraph Active["Active Cluster (Jakarta)"]
        direction BT
        Producer["Producers"] --> |"Write"| ClusterA["Active Cluster"]
        Consumer["Consumers"] --> |"Read"| ClusterA
    end
    
    subgraph Passive["Passive Cluster (Singapore)"]
        direction BT
        Backup["(Backup)"] -. "Sleeping" .-> ClusterB["Passive Cluster"]
    end
    
    ClusterA -- "Replication (Asynchronous)" --> ClusterB

• RPO (Recovery Point Objective): Very low (depends on asynchronous replication lag, usually below a few seconds).
• RTO (Recovery Time Objective): Slightly slow because it requires manual or automatic switchover processes of client applications via DNS changes.
• Advantages: Simple, no write conflict collision risks, and data consistency is easy to maintain.

2. Active-Active Topologies (Dual Ingest & Cross-Replication) #

In this model, both clusters are active simultaneously. Local producers in Region A write to Cluster A, and local producers in Region B write to Cluster B. Both clusters replicate data to each other bidirectionally.

flowchart TD
    subgraph ClusterA["Cluster A (Jakarta)"]
        direction BT
        ProdA["Producer A"] --> |"Write"| CA["Cluster A"]
        ConsA["Consumer A"] <--- |"Read"| CA
    end
    
    subgraph ClusterB["Cluster B (Singapore)"]
        direction BT
        ProdB["Producer B"] --> |"Write"| CB["Cluster B"]
        ConsB["Consumer B"] <--- |"Read"| CB
    end
    
    CA -- "Replication A to B" --> CB
    CB -- "Replication B to A" --> CA

• Advantages: Maximally utilizes server resources in both regions and provides very low local write latency for global users.
• Disadvantages: Very complex. We risk endless replication loops (where data from A replicates to B, then replicates back from B to A as new data). Additionally, global event ordering can’t be guaranteed because inter-region network latency is fluctuating.

3. Hub-and-Spoke Topologies (Data Aggregation) #

This topology uses many satellite clusters (Spokes) tasked with locally collecting data, then replicating all that data to one central cluster (Hub) for data consolidation, data warehousing, or large-scale AI analytics needs.

flowchart TD
    A["Cluster A (Store 01)"] -- "Replication" --> Center["Central Cluster (Headquarters)"]
    B["Cluster B (Store 02)"] -- "Replication" --> Center

• Advantages: Reduces WAN network burdens because local clusters act as buffers. If WAN connections to centers disconnect, local stores keep operating normally and re-sync data after connections recover.
• Disadvantages: Central clusters must be designed with very large storage capacities and throughput to accommodate aggregations from all satellite clusters.

Data Replication Using MirrorMaker 2.0 (MM2) #

To realize the multi-cluster topologies above, Apache Kafka provides a robust built-in replication tool named MirrorMaker 2.0 (MM2).

MirrorMaker 2.0 Internal Architecture #

MirrorMaker 1.0 (the old version) was just simple consumers and producers wrapped together, which often lost data, didn’t replicate topic configurations, and easily became bottlenecks. MirrorMaker 2.0 solves these problems by being built entirely on the Kafka Connect framework.

MM2 consists of several internally synergistic connectors:

1. MirrorSourceConnector: Reads data from topics in source clusters and writes it to target clusters. This connector also replicates topic configurations (like partition counts and retention configurations).
2. MirrorCheckpointConnector: Periodically maps consumer offsets from source clusters to target clusters. This is very crucial for consumer application failover processes.
3. MirrorHeartbeatConnector: Periodically sends heartbeat messages to target clusters to verify that inter-cluster connectivity paths are actively running.

Cross-Cluster Replication Loop Prevention #

How does MM2 prevent endless loops in Active-Active topologies? MM2 introduces dynamic topic namespace naming concepts.

By default, when MM2 replicates a topic named orders from a cluster named jakarta to the singapura cluster, that topic is written on the singapura cluster with the name jakarta.orders. If another MM2 replicates back from singapura to jakarta, MM2 sees the jakarta. prefix on that topic name and intelligently refuses to replicate it back because it knows that data originated from the jakarta cluster itself.

MirrorMaker 2.0 Production Configuration Example #

MM2 can run in three modes: Dedicated (running a standalone Connect cluster specially for MM2), Connect Cluster (deployed as ordinary connectors in existing Kafka Connect clusters), or Legacy (using wrapper scripts). The most stable way for production is using Dedicated Mode.

Here’s a complete mm2.properties configuration file example for replicating data from the jakarta cluster (Primary/Source) to the singapura cluster (Backup/Target) with Active-Passive topology:

# Defining alias names for our clusters
clusters = jakarta, singapura

# Connection configuration for each cluster
jakarta.bootstrap.servers = broker-jkt-1:9092,broker-jkt-2:9092,broker-jkt-3:9092
singapura.bootstrap.servers = broker-sg-1:9092,broker-sg-2:9092,broker-sg-3:9092

# Configuring replication directions (from jakarta -> singapura)
jakarta->singapura.enabled = true

# Determining topic name patterns to replicate
# We use regex to replicate all topics except internal system topics
jakarta->singapura.topics = .*
jakarta->singapura.topics.blacklist = _.*, .*\\.internal, __consumer_offsets

# Enabling consumer group and offset synchronization
jakarta->singapura.emit.checkpoints.enabled = true
jakarta->singapura.sync.group.offsets.enabled = true
jakarta->singapura.sync.group.offsets.interval.seconds = 5

# Heartbeats for connectivity monitoring
jakarta->singapura.emit.heartbeats.enabled = true
jakarta->singapura.emit.heartbeats.interval.seconds = 1

# Determining whether to use cluster name prefixes on target topics
# For pure Active-Passive, we can turn off prefixes so topic names stay the same
# WARNING: Don't turn this off if we're using Active-Active topologies!
jakarta->singapura.rename.topics = false

# Internal Kafka Connect performance parameters for MM2
tasks.max = 8
jakarta.consumer.auto.offset.reset = earliest
singapura.producer.acks = all
singapura.producer.max.in.flight.requests.per.connection = 1
singapura.producer.compression.type = zstd

To run MirrorMaker 2.0 in the background using that configuration file, use the following terminal command:

# Run the MM2 daemon on our dedicated Connect server
connect-mirror-maker.sh /etc/kafka/mm2.properties > /var/log/kafka/mm2.log 2>&1 &

Offset Synchronization and Consumer Failover Mechanisms #

The biggest challenge in multi-cluster disaster recovery is how to move our consumer applications from main clusters to backup clusters without re-reading data from scratch (duplicate processing) or skipping unread data (data loss).

Why Can’t Offsets Be Replicated Raw? #

Offsets in Kafka are local to every partition inside one specific cluster. Because of topic creation time differences, log compaction, or broker failures during writes, offset 1000 in the jakarta cluster is never guaranteed to contain exactly the same message as offset 1000 in the singapura cluster.

Therefore, raw-replicating offset numbers directly from __consumer_offsets damages consumer read positions after failovers.

Solution: Offset Translation via Checkpoints #

MirrorMaker 2.0 solves this problem by periodically writing offset mapping metadata to an internal topic named singapura.checkpoints.internal on target clusters. This topic contains information like:

• Consumer group names.
• Topic names and partition numbers.
• Source offsets.
• Translated target offsets.
• Time metadata (timestamps).

When consumer applications are diverted to the singapura cluster, consumer libraries or our failover helper tools read this checkpoint metadata to find equivalent offset positions on new clusters, then do seek() operations before starting to read new messages.

Cross-Region Data Replication and Failover Flow #

Here’s a visualization of how data replicates from active regions to passive regions, and how client applications are diverted when total outage incidents happen in active regions:

flowchart TD
    subgraph Active_Region["ACTIVE REGION (PRIMARY - JAKARTA)"]
        direction TB
        ProdA["Producer Application A"]
        ClientG["Consumer Application B"]
        ClusterA["Main Kafka Cluster"]
        ProdA -->|1. Send Data| ClusterA
        ClientG -->|2. Consume Data| ClusterA
    end

    subgraph Replicator_Layer["MIDDLE LAYER REPLICATION PIPELINE"]
        direction TB
        MM2["MirrorMaker 2.0 (Kafka Connect)"]
    end

    subgraph Passive_Region["PASSIVE REGION (DR - SINGAPORE)"]
        direction TB
        ClusterB["Backup Kafka Cluster"]
        CheckpointTopic["Internal Checkpoint Topic"]
    end

    %% Main Replication Flow
    ClusterA -->|"3. Pull Data & Checkpoints"| MM2
    MM2 -->|"4. Push Data & Translate Offsets"| ClusterB
    MM2 -. "5. Write Checkpoint Metadata" .-> CheckpointTopic

    %% Failover Flow during Disasters
    Active_Region -.->|"6. Disaster Happens (Total Outage)"| FailoverEvent{"DISASTER FAILOVER"}
    FailoverEvent -->|"7. Divert DNS / Connections"| ProdA_Backup["Producer Application A (Singapore)"]
    FailoverEvent -->|"8. Fetch Translated Offsets"| ClientG_Backup["Consumer Application B (Singapore)"]
    
    ProdA_Backup -->|9. Write New Data| ClusterB
    CheckpointTopic -.->|"10. Read Offset Positions"| ClientG_Backup
    ClientG_Backup -->|11. Consume Without Duplication| ClusterB

    style FailoverEvent stroke:#d32f2f,stroke-width:3px

Disaster Mitigation & DNS Failover Strategies #

To smoothly do cluster switchovers during disasters (Disaster Recovery Switchovers), we must arrange mitigation strategies at application and network infrastructure levels.

1. Client Routing Strategies Using DNS Failover #

We must not hardcode physical broker IP addresses inside producer or consumer application code. Instead of writing broker-jkt-1:9092, use internal DNS domain names (CNAMEs) managed by load balancers or dynamic DNS servers (like Route 53 or Consul):

• kafka-bootstrap.company.internal

When the Jakarta cluster totally dies, our SRE teams just redirect the kafka-bootstrap.company.internal CNAME to point at the Singapore cluster bootstrap address (broker-sg-1:9092). When client applications experience TCP disconnections, they automatically re-resolve DNS and connect to backup clusters without needing to restart application pods.

Pay attention to DNS TTL (Time to Live) settings on our application OSes and JVMs. If JVMs are set to cache DNS resolutions forever (networkaddress.cache.ttl=-1), our applications never detect those DNS failover address changes before pods are manually restarted. Set this value to a maximum of 10 seconds in Java security configuration files (java.security).

2. Producer Switchover Strategies #

To prevent message loss during failovers, producers must be configured with high tolerance for temporary connection breaks:

• retries: Set to maximum values (Integer.MAX_VALUE).
• delivery.timeout.ms: Set to sufficiently long durations (e.g., 120000 or 2 minutes) to give SRE teams time to finish DNS failovers without triggering message delivery failures on application sides.
• acks: Must be set to all so data is truly safely stored before moves happen.

3. Consumer Switchover Strategies #

When switching to backup clusters, consumers must track the last successfully processed offset positions. If we use MM2 with sync.group.offsets.enabled=true, MM2 automatically synchronizes our consumer groups on backup clusters.

However, because offset alignment happens periodically (asynchronously), there’s a possibility consumers will re-read some last messages processed right before outages happen. Therefore, our consumer applications must be Idempotent Consumers so they can handle duplicate data processing without damaging our operational database integrity.

Replication Technology Alternatives Besides MirrorMaker 2.0 #

Although MM2 is a very reliable open source built-in standard, there are several alternative technologies in the industry offering specific advantages:

1. Confluent Replicator #

A proprietary commercial tool from Confluent tightly integrated with their ecosystem.

• Advantages: Replicator supports automatic, dynamic Schema Registry data replication across regions, and can replicate ACLs (Access Control Lists) instantly.
• Disadvantages: Requires commercial licenses (Enterprise).

2. Uber uReplicator #

An open source project developed by Uber to overcome MirrorMaker 1.0 limitations on giant-scale clusters before MM2 was released.

• Advantages: Uses Apache Helix for dynamically dividing replication tasks and has very stable task rebalance management on millions of messages per second throughput.
• Disadvantages: More complex to deploy and most of its features are now well adopted by MM2.

Multi-Cluster Disaster Recovery Readiness Audit Checklist #

Do the following operational audit steps periodically to guarantee our backup clusters are truly ready for real-world disasters:

Summary #

• Choose Suitable Topologies — Use Active-Passive for safe data consistency and operational convenience. Use Active-Active only if organizations have critical global local write latency needs and are ready to manage data conflicts.
• Use Dedicated MirrorMaker 2.0 — Run MM2 on dedicated Kafka Connect clusters for stable, failure-tolerant replication task scalability.
• Apply CNAMEs for Failovers — Avoid hardcoding broker IPs. Use internal DNS domains with low TTLs to speed up transparent client connection switchover processes.
• Guarantee Application Idempotency — Anticipate minor data duplication post-failover from asynchronous offset synchronization lag by making our consumer logic idempotent.
• Use Checkpoints for Transitions — Rely on MirrorMaker Checkpoint Connectors to dynamically track cross-cluster offset mappings rather than raw-copying offset numbers.

← Previous: Centralized vs Decentralized Clusters Next: Anti-Pattern in Production →