Connector Anti-Pattern #

Adopting Apache Kafka Connect as a data integration bridge does promise operational ease because of its declarative configuration approach without code writing. However, this ease often makes developer and system administrator teams complacent about the underlying distributed architecture rules. In real production environments, small configuration errors or Connect feature misuses can result in CPU/Memory resource waste, data leaks, massive data pipeline downtime, and even damaged database system consistency. To avoid these operational failures, we need to recognize the common error patterns frequently happening in the industry. This article will critically dissect nine fatal mistakes (connector anti-patterns) in designing and operating Kafka Connect in production, provide their fixes, and present a comprehensive production readiness checklist.

1. Developing Custom Producers/Consumers for Standard Integration Scenarios (Re-inventing the Wheel) #

Many organizations newly adopting Apache Kafka rush to write custom producer and consumer applications (for example using Java SDK, Go, or Python) just to copy PostgreSQL database tables to Kafka, or move Kafka events to AWS S3.

• Why Is This an Anti-Pattern? Writing custom code for standard integration cases means we must figure out source offset management logic, transient network error handling, parallel load balancing, cluster failover, and metric instrumentation ourselves. This code is bug-prone, hard to maintain across teams, and wastes valuable development time.
• The Correct Solution: Always check the Confluent Hub directory before writing any integration code. More than 90% of popular storage systems (PostgreSQL, MySQL, S3, Elasticsearch, Redis, Snowflake, etc.) already have professionally developed, reliability-tested ready-made connectors. Leverage the Kafka Connect framework to save time and guarantee data pipeline reliability.

2. Setting tasks.max Higher than the Input Partition Count on Sink Connectors (Idle Tasks) #

In an effort to increase data processing speed (throughput), some cluster administrators aggressively set "tasks.max" to very high numbers on Sink Connectors.

// ANTI-PATTERN: Setting tasks.max = 10 to read a topic with 4 partitions
{
  "name": "elasticsearch-sink",
  "config": {
    "tasks.max": "10",
    "topics": "orders-topic" // This topic only has 4 partitions in Kafka
  }
}

• Why Is This an Anti-Pattern? On Sink Connectors, one Kafka topic partition can only be consumed by one Connect task at a time. If we set tasks.max=10 on a topic with only 4 partitions, the Connect cluster creates 10 task thread instances, but 6 of those tasks are permanently IDLE. This wastes JVM heap memory resources, litters system logs, and adds cluster monitoring overhead without providing any performance improvement.
• The Correct Solution: Always coordinate the tasks.max setting with the Kafka topic partition count being read. To increase Sink Connector read parallelism, we must first increase the partition count on the Kafka broker side (for example raising partitions from 4 to 12), then set tasks.max to 12.

3. Letting Connect Internal Topic Cleanup Policies Get Deleted (connect-configs/offsets/status Cleanup Policy) #

By default, when a distributed Connect worker first starts, the worker tries creating three internal topics: connect-configs, connect-offsets, and connect-status. However, sometimes this automatic creation policy is disabled, and administrators create those topics manually with default broker configuration.

• Why Is This an Anti-Pattern? If internal topics are created with default broker configuration having time-based cleanup policies (cleanup.policy=delete with 7-day retention), Kafka automatically deletes old messages after 7 days. When connector configuration data in connect-configs is deleted, the Connect cluster loses all registered connector information. When offset data in connect-offsets is deleted, all Source Connectors start reading source system data from the beginning again (data duplication).
• The Correct Solution: All three Kafka Connect internal topics must be created with log compression-based cleanup policies (cleanup.policy=compact). This guarantees Kafka never deletes status data and last offset coordinates, but only compacts the log to retain the latest record per key.

4. Doing Heavy Data Processing in Single Message Transforms (SMT) (Heavy ETL inside SMT) #

Single Message Transforms (SMT) is a very practical feature for quickly changing data shapes while transiting in worker memory. However, this ease is often misused for complex data manipulations.

• Why Is This an Anti-Pattern? SMTs run synchronously inside the task execution’s single thread. If we use SMTs to run additional SQL queries to external databases for data enrichment, do high-level cryptographic encryption, or make external REST API calls, per-message latency spikes. The data pipeline experiences severe congestion, triggering task heartbeat delays (heartbeat timeout), and causing Connect worker instances to continuously crash and rebalance.
• The Correct Solution: Use SMTs only for lightweight stateless single-row modifications (like renaming fields, hiding columns, or inserting timestamp metadata). If we need data join operations, external queries, or heavy business calculations, do those in a separate stream processing layer using Kafka Streams or Apache Flink after data lands in Kafka.

5. Running Standalone Mode in Critical Production Environments (Standalone Mode in Production) #

Some developer teams keep the Standalone Mode deployment they used during local development when systems move to main production environments.

• Why Is This an Anti-Pattern? Standalone Mode runs in one single JVM process. This mode has no High Availability (HA) or automatic failover capability. If the VM where the standalone Connect worker runs dies or experiences a JVM error, the entire integration data flow is totally paralyzed until an administrator manually logs into the server to restart the service.
• The Correct Solution: Always use Distributed Mode for production environments, deploying at least 2 or 3 worker nodes spread across different Availability Zones. This guarantees high availability where tasks can automatically move if one node crashes.

6. Ignoring Schema Compatibility Settings (Schema Incompatibility) #

When using the Schema Registry to manage binary data type validation, developers often directly change column data types in source databases (for example from INT to VARCHAR) without thinking about the impact on downstream consumers.

• Why Is This an Anti-Pattern? Direct column modifications without strict schema compatibility rules make the Debezium CDC Connector send new schemas violating the Schema Registry compatibility rules. As a result, schema registration is rejected, and the Connect task immediately enters permanently FAILED status, stopping the entire data pipeline.
• The Correct Solution: Apply BACKWARD or FULL schema compatibility rules on the Schema Registry. To drastically change database column data types, do a gradual migration process (add new columns, slowly migrate data, then safely delete old columns after consumers are upgraded).

7. Not Enabling Dead Letter Queues (DLQ) on Sink Connectors (DLQ Negligence) #

Preventing Sink tasks from easily crashing when encountering corrupted messages is often done by setting the errors.tolerance=all property.

• Why Is This an Anti-Pattern? Setting errors.tolerance=all without defining a Dead Letter Queue (DLQ) topic makes the Connect worker silently throw corrupted messages into the trash. We never know a message was lost from the pipeline, only realizing it months later when data lake audits show mismatched transaction report numbers.
• The Correct Solution: Always combine "errors.tolerance": "all" with defining a DLQ topic through the "errors.deadletterqueue.topic.name" property. Also enable context headers so error causes are neatly recorded for debugging needs.

8. Hard-Deleting CDC Data Without Sending Tombstone Records #

When operating Source CDC (Debezium), we often clean up old data in source databases using SQL DELETE commands.

• Why Is This an Anti-Pattern? By default, when a data row deletion happens in an RDBMS, Debezium emits a record to Kafka with the op: d metadata value. Right after that record, Debezium emits a second empty message with value null and the same key. This second null message is called a Tombstone Record. Tombstones are very important for downstream consumers (like Kafka topics with compact cleanup policies, or compressed target databases) as a signal to remove that key from physical memory. If we disable tombstones (tombstones.on.delete=false), disk space in downstream systems keeps bloating because old data already deleted in the main database is never cleaned in Kafka.
• The Correct Solution: Always keep the default setting "tombstones.on.delete": "true" active on Debezium configuration. Also make sure our downstream consumer systems are designed to handle null-valued messages without triggering NullPointerExceptions.

9. Placing Plugin JAR Files in the Global Java Classpath (Dependency Hell) #

When installing new custom connectors on Connect servers, some administrators put plugin JAR files in the Kafka installation’s built-in library folder (/opt/kafka/libs/) or merge them into the global system CLASSPATH environment variable.

• Why Is This an Anti-Pattern? Putting all connector JAR files in the global JVM classpath triggers Dependency Hell (Library Collisions). Two different connectors often need different third-party library versions (for example, Connector A needs guava-20.0.jar while Connector B needs guava-32.0.jar). When placed in the global classpath, the JVM only loads one version, causing one connector to crash at runtime from method call failures (java.lang.NoSuchMethodError).
• The Correct Solution: Leverage Kafka Connect’s built-in classloader isolation feature using the plugin.path parameter. We must place each connector in its own isolated subdirectory folder under the plugin path:

/opt/connectors/
  ├── debezium-connector-postgres/
  │     ├── debezium-core-2.1.jar
  │     └── postgresql-42.5.jar
  └── confluent-connector-s3/
        ├── connect-s3-10.3.jar
        └── aws-java-sdk-s3-1.12.jar

In the worker properties, we just define the parent folder: plugin.path=/opt/connectors

Each connector is loaded in its own isolated Classloader without any library collision risk.

Production Readiness Review Checklist #

Before deploying a new connector to main production environments, use the checklist below to verify its safety and reliability:

Category 1: Infrastructure & Worker Cluster #

• Connect Workers are deployed in Distributed Mode with at least 2 instances in different Availability Zones.
• Worker JVM Heap Memory allocation is set to at least 4 GB with the -XX:+UseG1GC option active.
• The Connect cluster subnet is isolated and the HTTP REST API port 8083 is secured from external public access.
• Workers are set in the same AZ as source databases to minimize Cross-AZ egress costs.

Category 2: Connector Configuration #

• The tasks.max parameter for Sink Connectors is set according to the Kafka topic partition count.
• CDC Source Connectors (Debezium) are explicitly set with tasks.max=1 to prevent overhead.
• All database passwords and API Keys are encrypted and called using Config Provider placeholders (like FileConfigProvider or Vault).
• SMTs only do lightweight stateless operations; no network I/O calls inside transformations.

Category 3: Internal Topics & Schema Validation #

• The three internal topics (connect-configs, connect-offsets, connect-status) are created with replication.factor=3 and min.insync.replicas=2.
• The connect-configs internal topic is configured with exactly 1 partition.
• All three internal topics use the cleanup.policy=compact cleanup policy.
• Data validation uses Avro/Protobuf with the Schema Registry active using BACKWARD compatibility.

Category 4: Error Handling & Observability #

• Sink Connectors are configured with a Dead Letter Queue (DLQ) and the context.headers.enable=true property active.
• The errors.retry.timeout setting is set to at least 3-5 minutes to handle transient network failures.
• JMX metric monitoring for put-batch-time-ms, poll-batch-time-ms, and total-record-errors is connected to Prometheus/Grafana.
• An auto-recovery script is active to automatically restart failed tasks from short network errors.

Summary #

• Re-inventing the Wheel — Always use industry-standard connectors listed on Confluent Hub instead of writing custom producer/consumer code for standard database/storage integration cases.
• tasks.max Tuning — The number of active Sink tasks is physically limited by the Kafka topic partition count. Setting tasks.max beyond partitions only wastes JVM memory.
• Compacted Internal Topics — Make sure the three distributed Connect internal topics (configs, offsets, status) use the cleanup.policy=compact policy so configuration data isn’t lost after the default retention period passes.
• Classloader Isolation — Use the plugin.path setting by placing each connector in its own subdirectory to prevent library version collisions (dependency hell) on the JVM.
• Cluster Isolation — Avoid combining all connectors into one giant Connect cluster. Isolate clusters by workload characteristics (CDC Source vs Bulk Sink) to prevent rebalance storms.