Centralized vs Decentralized Clusters: Determining Cluster Topology Models #

When our organizations grow and adopt event-driven architectures broadly, the number of producer and consumer applications interacting with Apache Kafka increases drastically. At this point, we face infrastructure governance dilemmas determining cost efficiency and operational convenience: “Should we build one giant centralized Kafka cluster shared by all departments (Centralized Shared Cluster), or provide separate dedicated clusters for every developer team (Decentralized Dedicated Clusters)?”

Making these decisions without mature analysis often leads to regrets. Hurriedly choosing centralized models can trigger noisy neighbor incidents where one problematic application consumes all broker bandwidth and damages other applications’ performance. Conversely, choosing uncontrolled decentralized models triggers wild infrastructure cost spikes, hard-to-synchronize data silos, and exhausting maintenance workloads for our SRE teams.

In this guide, we’ll dissect the advantages and disadvantages of centralized vs decentralized cluster models, discuss safe multi-tenancy management techniques in centralized clusters using Client Quotas and ACLs, arrange decision-making matrices, and provide practical guides for choosing the right cluster topologies.

Kafka Cluster Topology Model Introduction #

To lay clear context, let’s define the characteristics of the following two cluster topology models:

1. Centralized Clusters (Centralized Shared Cluster / Multi-Tenant) #

In this model, there’s only one giant Kafka cluster (or several large clusters divided by environment classifications like Dev, Staging, Prod) managed by one central platform team. All developer teams (tenants) share their data logs into the same cluster.

2. Decentralized Clusters (Decentralized Dedicated Clusters) #

In this model, every division, micro team, or project has their own small, independent dedicated Kafka clusters. Cross-cluster data usage must be bridged using external data replicators.

Centralized Cluster Advantages & Disadvantages #

Centralized models offer maximum efficiency at the cost of absolute isolation.

Advantages: #

• Cost Optimization (CAPEX & OPEX): We only pay metadata controller machine overhead costs (KRaft/ZooKeeper) for one cluster. Broker server resource usage (CPU/RAM/Storage) is also far more efficient because fluctuating workloads from various applications balance each other out.
• Data Governance Convenience: Security management (SSL/SASL), ACL audit policies, centralized data schemas (Schema Registry), and operational metric monitoring are focused in one place, easing industry regulation compliance.
• Data Integration Convenience: Because all data topics are in one cluster, cross-department data consumption processes can happen instantly without needing to build additional data replication pipelines.

Disadvantages: #

• Noisy Neighbor Effects: A producer application owned by Team A experiencing infinite loop bugs can bombard brokers with millions of messages per second, consuming RAM page caches, saturating broker network cards, and slowing down data processing belonging to innocent Team B.
• Large Blast Radius: If centralized clusters experience major outages, all company business activities immediately become totally paralyzed.
• Customization Difficulties: We can’t do specific performance tuning for one particular application (e.g., different log segment sizes) if those settings disturb other tenant applications on the same brokers.

Decentralized Cluster Advantages & Disadvantages #

Decentralized models prioritize security and freedom at the cost of cost efficiency.

Advantages: #

• Absolute Resource Isolation: No noisy neighbor risks. One dedicated cluster’s performance is fully controlled by its owner application.
• Minimal Blast Radius: If the Payment team’s cluster dies, the Logistics team’s package tracking systems keep running at $100%$ normal.
• Configuration Freedom: Developer teams freely choose Kafka versions, independently do binary upgrades, and tune broker parameters according to their own application workload needs without negotiating with central platform teams.

Disadvantages: #

• Ballooning Infrastructure Costs: We must pay replication and minimum quorum overhead costs for every small cluster (remember the $\text{Broker Count} \ge \text{Replication Factor}$ constraint). Our cloud server bills soar high.
• Data Silos: When Marketing teams need transaction data from Payment teams, they can’t access it directly. We must build, maintain, and monitor asynchronous replication pipelines (like MirrorMaker) across clusters, adding architecture complexity.
• High Operational Burdens: SRE teams must do maintenance, operating system patching, SSL certificate rotations, and version upgrades for dozens of scattered independent clusters.

Safe Multi-Tenancy Implementation in Centralized Clusters #

If we decide to choose Centralized Cluster models to save operational costs, we must configure the following multi-tenancy security features to prevent noisy neighbor disasters and data leaks.

1. Topic Naming Standardization (Namespaces) #

We must forbid random topic creation without rules. Apply namespace-based topic naming conventions reflecting department or application names: [department].[environment].[application-name].[topic-name]

Examples: finance.prod.payment-service.orders logistics.staging.tracking.driver-location

Use the auto.create.topics.enable=false broker property to prevent clients from automatically creating topics outside these naming standards.

2. Applying Client Quotas to Limit Throughput #

To isolate broker network card usage from one naughty tenant’s domination, we must set Client Quotas. These quotas limit data transfer rates (in bytes per second) for every specific Principal (SSL/SASL Users) or Client ID.

Run the following CLI command to limit Team A producer write rates to a maximum of $10 \text{ MB/s}$ and consumer read rates to a maximum of $20 \text{ MB/s}$ on brokers:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --alter \
  --entity-type users \
  --entity-name User:CN=tenant-a-app,O=MyCorp \
  --add-config producer_byte_rate=10485760,consumer_byte_rate=20971520

How Quotas Work in Kafka: #

If Team A producers try sending data exceeding $10 \text{ MB/s}$, brokers don’t roughly reject client connections (reject requests). Instead, brokers calculate how long the delay needed to normalize data rates back to quota limits, then deliberately hold TCP handshake responses (injecting delay / throttling) to those producers before sending success confirmations. This delay technique naturally slows producer thread rates in client applications without triggering crashes.

Technically, Kafka uses Sliding Window algorithms with token bucket metrics to calculate average throughput within certain time windows (usually set via quota.window.num=11 and quota.window.size.seconds=1). When limits are exceeded, brokers hold responses inside internal broker DelayQueues for penalty durations (e.g., $450 \text{ ms}$) before releasing them back to client network threads. This measurably increases response latency on producer sides, triggering natural backpressure mechanisms on producer client libraries.

3. Advanced Compute and Network Quotas #

Besides byte rate limits, large-scale centralized clusters must also apply compute quotas and connection creation limits to anticipate unintentional denial-of-service (DoS) attacks:

• request_percentage: Percentage limits of I/O thread (num.io.threads) and network thread (num.network.threads) time one user may consume (e.g., set request_percentage=20). This prevents clients from sending metadata queries in fast loops that can jam request handlers.
• connection_creation_rate: Limits new TCP connection creation counts per second from specific IP addresses (e.g., set a maximum of 50 connections per second) to secure brokers from file descriptor exhaustion from botnet attacks or client application reconnection bugs.

# Dynamically limit TCP connection creation rates from specific IPs
kafka-configs.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --alter \
  --entity-type ips \
  --entity-name 10.120.4.52 \
  --add-config connection_creation_rate=50

4. Schema Compliance Enforcement (Schema Registry Governance) #

In centralized clusters, if one producer changes payload data structures without coordination, all consumer applications belonging to other tenants reading those topics crash from parsing errors (deserialization errors).

• Action: Must integrate Schema Registry (like the Confluent Schema Registry).
• Compatibility Policies: Set compatibility levels to BACKWARD or FULL. This ensures producers can only register new schema versions if those schemas are safe to read by consumers still using old-version code, maintaining smooth centralized data operations.

Here’s the Schema Registry compatibility matrix table our platform teams must understand for determining schema evolution rules:

5. Securing Access Rights Using Strict ACLs and SASL Authentication #

Don’t let one tenant freely read data from other tenants’ topics. Apply Zero-Trust security architectures by limiting Write and Read access rights using Kafka ACL policies based on SSL certificates (SSL Principals) or SASL/SCRAM accounts.

First, we must configure JAAS authentication for SASL/SCRAM users on brokers. Add the following lines to the broker JAAS configuration file (/etc/kafka/kafka_server_jaas.conf):

KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"
    password="admin-secret-password"
    user_admin="admin-secret-password"
    user_tenant_a="tenant-a-secret-pass"
    user_tenant_b="tenant-b-secret-pass";
};

Next, run the following CLI command to enable ACLs limiting cross-tenant topic read/write access rights:

# Give Write access only to Finance producers on their own topics
kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/client.properties \
  --add \
  --allow-principal User:tenant_a \
  --operation Write \
  --topic finance.prod.

To complete the security, we must also enable the authorizer.class.name=kafka.security.authorizer.AclAuthorizer parameter inside every broker’s server.properties file. By default, if this parameter is active, brokers reject all client access (deny by default) unless those accesses have been explicitly permitted through ACLs.

Main Monitoring Metrics in Multi-Tenancy #

When managing centralized clusters inhabited by many tenants, detecting which tenants overburden systems is the key to maintaining operational stability. We must collect the following JMX metrics through Prometheus/Jolokia agents on our brokers:

1. Quota Throttling Metrics (Client Throttle Time) #

These metrics show penalty durations (in milliseconds) brokers give to clients for violating byte rate limits.

• MBean Names (Producer): kafka.server:type=Produce,user={user},client-id={client-id}
• Attributes: throttle-time-average and throttle-time-max
• Alert Actions: If average throttle-time values are above $0 \text{ ms}$, those tenants are actively limited. SREs must prepare if developers complain about their application latency increases.

2. CPU Thread Usage Metrics (Request Percentage Quota) #

Shows the percentage of broker I/O thread time spent serving requests from specific users.

• MBean Names: kafka.server:type=Request,user={user},client-id={client-id}
• Attributes: request-time-fraction
• Alert Actions: If a user consumes request-time-fraction more than $0.20$ ($20%$), limit that user’s activities or ask them to optimize delivery batching (e.g., raising batch.size and linger.ms on producers).

3. Connection Creation Metrics (Connection Creation Rate) #

Monitors new TCP connection rates per second created by clients to detect connection leak bugs.

• MBean Names: kafka.server:type=SocketServer,name=ConnectionCreationRate
• Attributes: MeanRate and OneMinuteRate
• Alert Actions: Sharp increases above 100 new connections per second on one broker indicate client applications experiencing failed-connection loops continuously trying to create new TCP sockets, risking broker server OS file descriptor capacity exhaustion.

Logical Topology Comparison: Centralized vs Decentralized #

Here’s a visualization of data structure and access differences between shared centralized cluster models vs dedicated decentralized clusters:

flowchart TD
    subgraph Centralized["CENTRALIZED CLUSTER MODEL (Multi-Tenant)"]
        direction TB
        subgraph SharedCluster["Giant Centralized Kafka Cluster"]
            TopicA["Topic: tenant-a.orders"]
            TopicB["Topic: tenant-b.orders"]
        end
        ClientA1["Producer A"] -->|10MB/s Write Quota| TopicA
        ClientB1["Producer B"] -->|5MB/s Write Quota| TopicB
        ClientA2["Consumer A"] -->|ACL-Limited Access| TopicA
        ClientB2["Consumer B"] -->|ACL-Limited Access| TopicB
    end

    subgraph Decentralized["DECENTRALIZED CLUSTER MODEL (Dedicated)"]
        direction TB
        subgraph ClusterA["Kafka Cluster A (Division A)"]
            TopicA_Ded["Topic: orders"]
        end
        subgraph ClusterB["Kafka Cluster B (Division B)"]
            TopicB_Ded["Topic: orders"]
        end
        ClientA_Ded["Producer & Consumer A"] <--> ClusterA
        ClientB_Ded["Producer & Consumer B"] <--> ClusterB
        
        ClusterA -.->|"Data Replication Pipelines (MirrorMaker 2.0)"| ClusterB
    end

Decision-Making Matrices: Choosing the Right One #

Use the evaluation matrix table below to guide our organizations in choosing cluster topology models best matching real business conditions:

Cluster Topology Model Selection Audit Checklist #

Do the following audit steps to verify that the cluster model we’re currently running is equipped with adequate governance instruments:

Summary #

• Share Loads with Centralized — Choose centralized cluster models if the organization’s main priorities are infrastructure cost efficiency and one-stop data governance convenience.
• Isolate with Decentralized — Choose dedicated decentralized cluster models for systems with high security compliance levels (like bank regulations) or critical blast radius needs.
• Apply Client Quotas — Prevent noisy neighbor disasters on centralized clusters by configuring producer_byte_rate and consumer_byte_rate limits for every tenant.
• Lock Access via ACLs — Apply zero-trust security architectures using strict ACL authorization to isolate cross-department topic read-write access rights.

← Previous: Rolling Upgrade & Maintenance Next: Multi-Cluster Strategy →