Scaling & Resource Allocation #

Operating Apache Kafka Connect in large-scale production environments demands deep understanding of how performance is distributed in parallel across clusters. When data volumes grow from gigabytes to terabytes per day, our integration pipelines must be able to scale up without burdening target systems or Kafka brokers. To achieve this efficiency, we must optimize parallelism limits through tasks.max parameter tuning, manage worker JVM memory and Garbage Collector allocation, and design elastic scaling architecture in Kubernetes. This article will comprehensively dissect scaling and resource allocation strategies for Kafka Connect so our data pipelines operate with maximum performance and minimal latency.

Determining the Main Parallelism Limit (tasks.max) #

The first pillar in Kafka Connect scaling is the tasks.max parameter. This parameter is set in every connector configuration and limits the maximum number of parallel tasks the Connect cluster may run for that connector.

How Does tasks.max Relate to Scalability? #

When we set tasks.max, the leader worker in the Connect cluster tries dividing the work into several independent parts (tasks) and spreading them evenly across all active workers. However, actual parallelism is often physically limited by the source system (for Source) or topic partitions (for Sink):

flowchart TD
    subgraph KafkaTopic["Kafka Topic: orders (4 Partitions)"]
        P0["Partition 0"]
        P1["Partition 1"]
        P2["Partition 2"]
        P3["Partition 3"]
    end
    
    subgraph ConnectCluster["Connect Cluster (tasks.max = 6)"]
        T0["Sink Task 0"]
        T1["Sink Task 1"]
        T2["Sink Task 2"]
        T3["Sink Task 3"]
        T4["Sink Task 4 (IDLE)"]
        T5["Sink Task 5 (IDLE)"]
    end
    
    P0 --> T0
    P1 --> T1
    P2 --> T2
    P3 --> T3

1. Sink Connector Parallelism Rules #

Sink Connector parallelism is absolutely limited by the number of Kafka topic partitions it reads.

• If we set "tasks.max": "6", but the Kafka topic being read only has 4 partitions, Kafka Connect only creates 4 active tasks. The other two tasks are idle because there are no remaining topic partitions to allocate to them.
• Ideal Ratio: Set tasks.max equal to the number of consumed Kafka topic partitions for maximum throughput.

2. Source Connector Parallelism Rules #

Source Connector parallelism is limited by the data unit division logic in the external source system.

• JDBC Source: If we poll 10 tables, we can set tasks.max up to 10 (each task copies 1 table). If there’s only 1 table, setting tasks.max=5 is useless because 1 table can’t be split across tasks for reading.
• Debezium CDC: The database transaction log (WAL/Binlog) is linear and can only be read by one single reader thread to maintain transaction event order. Therefore, the Debezium CDC Connector always forces execution of only 1 task (tasks.max=1), no matter how large we set that parameter value.

Threading Model Details in Kafka Connect Workers #

To understand CPU bottlenecks in a Connect cluster, we need to unpack how threads are allocated inside the Connect worker JVM. Each worker instance manages several different execution thread categories:

1. Leader Coordinator Thread #

This thread is responsible for coordinating with the Kafka broker (Group Coordinator) for Connect cluster membership. This thread manages the heartbeat process and listens for rebalancing signals. If a GC pause stops this thread too long, the worker gets kicked from the cluster.

2. Task Execution Threads #

Every task (both Source Tasks and Sink Tasks) executes in an isolated dedicated thread.

• On Source Tasks, this thread runs the poll() method in an infinite loop.
• On Sink Tasks, this thread takes data from the internal consumer and executes the put() method.
• CPU Spikes: If we run 50 tasks in one worker node with only 8 CPU cores, severe thread contention happens (CPU context switching). This causes drastic performance degradation.

3. Client Network Threads #

The internal Kafka Producer (on Source) and internal Kafka Consumer (on Sink) employ additional network threads to send and receive data bytes via TCP/IP sockets.

• Thread Leaks: Poorly written custom connectors often don’t close database connections, HTTP clients, or file descriptors properly when tasks stop (stop()). This causes the thread count to keep growing and triggers the java.lang.OutOfMemoryError: unable to create new native thread error.

Connect Cluster Topology Design and Workload Isolation #

When designing enterprise-scale data pipeline infrastructure, one of the most important architectural decisions is determining how many physical Connect clusters to deploy.

The Danger of Using One Giant Connect Cluster (Unified Cluster) #

Combining all connector types (for example Debezium CDC for ERP databases, S3 Sink for audit logs, and Elasticsearch Sink for product search) into one single logical Connect cluster is a very dangerous anti-pattern in production:

• Wide Blast Radius: If one Sink task experiences OutOfMemory from a data load surge, the entire JVM Connect worker where that task runs crashes. This disrupts other innocent tasks.
• Rebalance Storms: Adding or changing one connector’s configuration triggers rebalancing for all connectors in that cluster. If our cluster runs 50 connectors with 200 total tasks, the rebalancing process takes a long time and causes global latency degradation.
• group.id Collisions: If two Connect clusters deployed separately on different VMs accidentally use the same group.id, they join into one unstable giant Connect cluster. They fight over tasks and trigger endless rebalancing loops.

Topology Recommendation: Workload-Based Cluster Isolation #

We’re strongly advised to separate Connect clusters into several independent physical clusters based on workload characteristics:

flowchart LR
    DB["Database Subnet"] --> CDC["Connect CDC Cluster (group.id=connect-cdc)"] --> Kafka["Kafka Cluster"]
    Kafka --> Sink["Connect Sink Cluster (group.id=connect-sink)"] --> Storage["Target Storage"]

1. CDC Source Cluster: Dedicated specifically for Debezium/JDBC Connectors. This cluster has the highest CPU/Memory resource priority because it captures main business transactions.
2. Bulk Sink Cluster: Dedicated to S3, GCS, or Snowflake Sinks. This cluster is specifically tuned for large throughput (large buffer memory) and latency tolerance.
3. Real-time Search/Cache Cluster: Dedicated to Elasticsearch and Redis Sinks. Tuned for low latency.

Network Zoning and Cross-Zone Cost Optimization (Cross-AZ Traffic) #

For security, Connect clusters must be placed on special subnets (DMZ/private subnets) with strict firewall rules:

• The REST API port 8083 may only be opened to trusted internal management subnets.
• Limit outbound database port connections (like 5432 PostgreSQL or 3306 MySQL) only to operational database server IPs.
• Cross-AZ Data Transfer Costs (Cross-AZ Egress Cost): Cloud providers charge for data transfers between Availability Zones. If the database is in AZ-a, make sure the Connect worker pods reading that database are also deployed in AZ-a. Streaming gigabytes of raw CDC data across zone boundaries before compression can produce unexpected network cost bloat.

JVM Worker Resource Allocation: Memory & CPU Optimization #

Because Kafka Connect runs on the Java Virtual Machine (JVM), its performance depends heavily on heap memory allocation configuration, system thread allocation, and Garbage Collector (GC) settings.

1. Optimizing Heap Memory #

Connect Workers handle many temporary objects during data deserialization and transformation (SMT) processes. If the JVM heap memory allocation is too small, the JVM frequently runs intensive Garbage Collection processes triggering Stop-The-World (STW) conditions. During STW, workers can’t send heartbeats to Kafka brokers, so brokers consider Connect workers dead and trigger disruptive rebalancing processes.

• Production Recommendation: Use a minimum heap setting of 4 GB to 8 GB for medium distributed worker clusters, and ensure the minimum value (-Xms) equals the maximum value (-Xmx) to avoid runtime memory resizing overhead:

  export KAFKA_HEAP_OPTS="-Xms8G -Xmx8G"
  

2. Using the G1 Garbage Collector (G1GC) #

Avoid the old Java built-in GC (Parallel GC) because it’s prone to long pauses on large memories. Use the G1 Garbage Collector designed for short, predictable pauses:

export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent"

3. Off-Heap Memory Usage #

Converters (especially Avro and Parquet formats) often allocate byte buffers outside the JVM heap memory (off-heap direct buffers) to speed up disk and network I/O. Make sure our server operating system has enough physical memory capacity beyond the Connect JVM heap setting so it doesn’t get process-killed by the Linux kernel (Out Of Memory / OOM Killer).

4. Jetty HTTP Thread Pool Tuning #

The worker’s HTTP REST API is served by the embedded Jetty server. By default, Jetty allocates a dynamic HTTP reader thread pool. If our team frequently runs cluster status queries or there are automation systems aggressively monitoring performance via the REST API, we must configure the Jetty thread limit so it doesn’t steal CPU from main data processing:

# Written in connect-distributed.properties
# Maximum Jetty thread limit for handling REST API requests
rest.threads.max=100

JMX Metrics Analysis for Scalability Monitoring #

Reliable elastic scaling can only be achieved if we have accurate monitoring metrics. Kafka Connect exposes various internal metrics via JMX (Java Management Extensions).

Here are the crucial JMX metrics we must monitor and use as alert or scaling triggers:

Horizontal Worker Scaling Strategies #

Horizontal scaling is done by adding new Connect worker instance nodes to an existing distributed cluster. This process runs very smoothly:

1. We run a new Docker container or VM with the same group.id configuration.
2. The new worker detects the group coordinator in Kafka and sends a join signal (JoinGroup).
3. The Connect cluster triggers an incremental rebalance using the Incremental Cooperative Rebalancing protocol.
4. Some tasks from old workers are asynchronously moved to the new worker without stopping other healthy tasks.

Autoscaling in Kubernetes Using KEDA #

In modern cloud environments (like Amazon EKS or Google GKE), we can automate the horizontal scaling process of Connect worker pods using KEDA (Kubernetes Event-driven Autoscaling).

KEDA acts as a smart autoscaler that can monitor metrics outside Kubernetes (like consumer lag directly from Kafka brokers) and automatically adjust the Connect deployment’s replica pod count.

flowchart LR
    subgraph KubernetesCluster["Kubernetes Cluster"]
        KEDAAgent["KEDA Controller"]
        ConnectDeployment["Kafka Connect Pods (Replica: 2 to 10)"]
    end
    
    KafkaBroker[("Kafka Broker Cluster")] -->|Monitor Consumer Lag| KEDAAgent
    KEDAAgent -->|Trigger Horizontal Scaling| ConnectDeployment

Example KEDA ScaledObject Manifest for Kafka Connect #

Below is a custom Kubernetes YAML manifest example for configuring KEDA to scale Connect worker pods (between a minimum of 2 pods to a maximum of 10 pods) based on the accumulated consumer lag on the orders-topic:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-connect-autoscaler
  namespace: data-pipeline
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kafka-connect-worker-deployment
  minReplicaCount: 2
  maxReplicaCount: 10
  cooldownPeriod: 300
  pollingInterval: 30
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker-1:9092,kafka-broker-2:9092
        consumerGroup: connect-elasticsearch-sales-sink
        topic: mysql-db-orders
        # Trigger adding 1 new pod for every 5000 consumer lag increment
        lagThreshold: "5000"
        activationLagThreshold: "100"

With KEDA, if the Elasticsearch target system experiences slowness causing Kafka lag to spike past 5,000 messages, KEDA instantly triggers adding new Connect worker pods to help speed up the data consumption process. After the lag clears, KEDA slowly lowers the replica pod count back to the minimum limit.

Summary #

• Parallel Limit — The tasks.max property sets the task execution parallelism limit. Sink tasks are capped by the Kafka topic partition count, while Debezium Source CDC is limited to only 1 task.
• Worker Threading — Every task runs on its own thread. Avoid thread leaks by ensuring file descriptor and target database connection cleanup when tasks stop.
• Workload Isolation — Don’t mix all connector types in one same Connect cluster. Split clusters into special groups (e.g., CDC-specific, Sink-specific) for safety and rebalance storm prevention.
• JVM Tuning — Avoid STW GC pauses triggering false rebalances by allocating a minimum of 4GB-8GB heap memory and enabling the G1 Garbage Collector (-XX:+UseG1GC).
• JMX Metrics — Monitor put-batch-time-ms and poll-batch-time-ms to detect performance degradation in external systems early.
• Kubernetes KEDA — KEDA enables elastic Connect worker pod autoscaling in Kubernetes based on consumer lag metrics on Kafka brokers without manual intervention.

← Previous: External System Integration Next: Error Handling & DLQ →