What is Kafka Connect? #

In modern distributed system architectures, Apache Kafka is often positioned as the central nervous system streaming data between various platforms. However, having a fast and reliable Kafka broker isn’t enough if we still have to write complex custom code every time we want to move data from a relational database to Kafka, or from Kafka to object storage. Connecting various external systems manually often ends up creating custom producer and consumer applications that are failure-prone, hard to scale, and consume significant development time. To solve this integration challenge in a standardized way, Apache Kafka provides a very powerful data integration framework called Kafka Connect. This article will thoroughly unpack the philosophy behind Kafka Connect, why we need it, its internal architecture covering Connectors, Tasks, and Workers, plus data processing mechanisms through Converters and Single Message Transforms (SMT).

Why Do We Need Kafka Connect? #

Before we dive into Kafka Connect’s internal workings, let’s first review why writing manual integration code using the standard Kafka Producer API and Consumer API often becomes a nightmare for developer teams. Imagine we have a common scenario where we must copy every new row from a PostgreSQL transaction database to a Kafka topic, then stream that data from Kafka to Elasticsearch for fast search needs, and also to AWS S3 for long-term archiving.

1. State and Offset Management Complexity #

If we write a custom producer for PostgreSQL, we must track which rows have already been successfully sent to Kafka. We need to handle the scenario where the producer application dies mid-way: where should we start reading again when the application restarts? This problem is even more complex on the consumer side (Elasticsearch and S3). Consumers must consistently commit offsets to Kafka only after data is truly written to the target system. If Elasticsearch experiences downtime, our consumer must hold back reads (backpressure) to avoid data loss.

2. Repeated Boilerplate Code #

Every time there’s a new database, new file system, or new cloud service that needs connecting to Kafka, developer teams are forced to write boilerplate code with similar patterns: connecting to the target system, reading data, serializing, sending to Kafka, handling network failures, and doing monitoring. This wastes compute time and team energy that should be allocated to solving core business logic.

3. Scalability and Fault Tolerance Problems #

Writing a single instance integration application may feel easy at first. But what if the data load suddenly surges tenfold? We must think about how to divide database table read workloads in parallel across multiple processes. If one machine running the integration process dies, how does the task transfer (failover) happen to another machine without causing massive data duplication or message loss?

Kafka Connect is designed to solve all the challenges above out-of-the-box. With Kafka Connect, we no longer write imperative programming code; we simply define declarative configuration (usually in JSON format) to run reliable, fault-tolerant, horizontally scalable data pipelines.

Kafka Connect Philosophy: Integration Without Code #

The core philosophy of Kafka Connect is data integration without coding. This means we separate data integration logic from our business application logic. Kafka Connect provides a standardized runtime framework where anyone can write an integration module (connector) once, and reuse it repeatedly in various runtime environments just by changing configuration parameters.

When to Use Kafka Connect vs Kafka Streams? #

It’s very important for us not to confuse the roles of Kafka Connect with Kafka Streams, because both have completely different architectural purposes:

Simply put, if we want to move data from a PostgreSQL database to Kafka, use Kafka Connect Source. If we want to transform raw transaction data in Kafka into hourly revenue aggregate reports, use Kafka Streams. If we want to move those aggregate results from Kafka to Elasticsearch, use Kafka Connect Sink.

Three Main Pillars of Kafka Connect Architecture #

Kafka Connect’s internal architecture is modularly designed to separate integration logic definitions, parallel work division, and execution infrastructure provisioning. The three main pillars composing this architecture are the Connector, Task, and Worker.

flowchart TD
    subgraph WorkerCluster["Kafka Connect Worker (Distributed Cluster)"]
        direction TB
        subgraph Worker1["Worker Instance 1"]
            C1["PostgreSQL Source Connector"]
            T1["Source Task 1"]
            T2["Source Task 2"]
        end
        subgraph Worker2["Worker Instance 2"]
            T3["Source Task 3"]
            T4["Sink Task 1"]
        end
    end
    
    SourceDB["("PostgreSQL Database")"] -->|Read via JDBC/CDC| C1
    C1 -->|Divide Work| T1 & T2 & T3
    
    T1 & T2 & T3 -->|Send Events| KafkaCluster{"Apache Kafka Cluster"}
    
    KafkaCluster -->|Fetch Events| T4
    T4 -->|Write| TargetS3["AWS S3 Bucket"]

1. Connector #

A Connector is the component defining interaction logic with a specific external system. There are two types of Connectors:

• Source Connector: Responsible for pulling data from external systems (like relational databases, other message queues, or IoT sensors) and sending it to Apache Kafka as records.
• Sink Connector: Responsible for taking data from Apache Kafka and writing it to external target systems (like Elasticsearch, NoSQL databases, or cloud object storage).

Connectors don’t execute the data movement process directly. Their main job is defining configuration, inspecting the target system (for example, detecting which tables exist in a database), and determining how that large job is divided into small parts executable in parallel.

2. Task #

A Task is the actual work unit in Kafka Connect data processing. The Connector divides the large workload into one or several Tasks. For example, a PostgreSQL Source Connector can divide the work of reading 10 database tables into 3 Tasks, where Task 1 reads tables 1-3, Task 2 reads tables 4-6, and Task 3 reads tables 7-10.

Because Tasks don’t store operational state locally (they’re stateless), they can run anywhere in the cluster. Tasks receive data from the Connector (for Source) or from Kafka (for Sink) then process it. Horizontal scalability in Kafka Connect is achieved directly by increasing the number of Tasks allowed to run in parallel through the tasks.max configuration.

3. Worker #

A Worker is the actual JVM runtime process running Connectors and Tasks. Workers act like execution containers. There are two Worker modes we can run:

• Standalone Worker: All Connectors and Tasks run in one single JVM process on one machine. This mode is very simple to configure but has no fault tolerance or horizontal scalability. Perfect for local development or simple ETL on edge nodes.
• Distributed Worker: Several Workers run distributedly on several different machines forming a Connect cluster. They coordinate automatically using Apache Kafka’s internal features. If one Worker crashes, the remaining Workers detect that Worker’s loss and automatically move hanging Tasks to healthy Workers (auto-failover).

Internal Data Models: SourceRecord and SinkRecord #

One of Kafka Connect’s main strengths is its ability to integrate any system with any system without creating tight coupling. This is achievable because Kafka Connect defines an abstract internal data model.

When data is pulled from PostgreSQL by a JDBC Source Connector, that data isn’t directly converted into JSON or Avro bytes. Instead, the Connector converts that database data representation into an internal Kafka Connect Java object called a SourceRecord. This SourceRecord object consists of two main parts:

1. Schema: The data structure defining column names, data types, and whether a column may be null.
2. Value: The actual record value matched to the defined schema.

Conversely, on the Sink side, Kafka Connect takes data bytes from Kafka, converts them into an internal object called a SinkRecord (which also has Schema and Value), then hands them to the Sink Connector to be written to external systems like S3 or Elasticsearch.

By separating the internal data format from the physical storage format in Kafka, the same connector can write data to Kafka in JSON, Avro, Protobuf, or even plain text format without changing a single line of code inside that connector. The physical format writing task is fully delegated to a component called the Converter.

The Role of Converters and Schema Registry #

A Converter is the component responsible for translating data between the Kafka Connect internal data model (schema and value) and the serialized binary representation stored in Kafka topics. Converters are configured independently from Connectors.

flowchart LR
    subgraph SourcePipeline["Source Pipeline"]
        SourceSystem["Source System"] -->|Raw Data| SourceConn["Source Connector"]
        SourceConn -->|"SourceRecord (Internal)"| SrcConverter["Converter (e.g., Avro)"]
        SrcConverter -->|Send Schema| SchemaReg["Schema Registry"]
        SrcConverter -->|Serialized Byte Data| KafkaTopic("Kafka Topic")
    end
    
    subgraph SinkPipeline["Sink Pipeline"]
        KafkaTopic -->|Serialized Byte Data| SnkConverter["Converter (e.g., Avro)"]
        SchemaReg -.->|Download Schema| SnkConverter
        SnkConverter -->|"SinkRecord (Internal)"| SinkConn["Sink Connector"]
        SinkConn -->|Raw Data| TargetSystem["Target System"]
    end

There are several built-in Converter types we often use in production:

1. StringConverter: Used if our data is simple text strings (for example raw application logs).
2. JsonConverter: Converts the internal data model into binary JSON strings. We can set the schemas.enable=true parameter to include the data schema directly in every JSON message, although this produces very large message size overhead.
3. AvroConverter: Converts the internal data model into a very compact Avro binary format. AvroConverter integrates tightly with the Confluent Schema Registry to store schemas centrally, so records sent to Kafka only contain the schema ID (5 bytes) and a very small binary payload.
4. ProtobufConverter: Similar to Avro, using the Protobuf binary format and leveraging Schema Registry for schema compatibility management.

Configuring Converters in Practice #

Below is an example of how we define Converter configuration in the distributed worker properties file (connect-distributed.properties):

# Using the Avro Converter for message keys and values for binary efficiency
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081

value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081

# ANTI-PATTERN: Don't use JsonConverter with schemas.enable=true in large-scale production
# because it duplicates large schema metadata into every message record.
# value.converter=org.apache.kafka.connect.json.JsonConverter
# value.converter.schemas.enable=true

If we’re forced to use JSON without a Schema Registry but still want to preserve schema information for downstream systems (for example target SQL databases), we can enable schemas.enable=true. However, for high-throughput production systems, switching to AvroConverter or ProtobufConverter is highly recommended.

Single Message Transforms (SMT) #

In the data pipeline journey from source systems to Kafka, or from Kafka to target systems, we often need simple-level data modifications. For example, we want to mask customer credit card numbers before writing to Kafka, add dynamic timestamps to records, or rename columns to match target database standards.

For these lightweight single-row data modification needs, Kafka Connect provides the Single Message Transforms (SMT) feature. SMT operates directly in worker memory before the record is serialized by the Converter (on Source) or right after the record is deserialized by the Converter (on Sink).

How SMT Works #

SMTs can be chained sequentially to form simple data transformation pipelines. Because SMTs operate at the single record level, they have no local state storage overhead and very low execution latency (microsecond range).

flowchart LR
    Source["Source Record"] --> T1["Transform 1: MaskField"] --> T2["Transform 2: InsertField"] --> Conv["Converter"] --> Byte["Kafka Byte"]

Some very useful built-in SMT modules include:

• MaskField: Replaces a specific field’s value with a static value or null (very important for privacy data regulation compliance like GDPR or PCI-DSS).
• InsertField: Inserts additional metadata like the Kafka topic name, system timestamp, or partition ID into the record.
• ReplaceField: Renames fields or filters certain fields so they aren’t sent to the target system.
• Cast: Changes a field’s data type (for example converting string "123" to integer 123).
• ValueToKey / RegExRouter: Restructures records or dynamically maps messages to different topic names based on regular expressions.

SMT Configuration Example on a Source Connector #

Below is a real example of how we configure SMT in a Source Connector configuration JSON file to mask the password column and insert a unique transaction UUID into the record:

{
  "name": "mysql-source-connector",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "tasks.max": "1",
    "connection.url": "jdbc:mysql://mysql-db:3306/shop",
    "connection.user": "exporter",
    "connection.password": "secret",
    "mode": "incrementing",
    "incrementing.column.name": "id",
    "topic.prefix": "mysql-",
    
    "//": "Defines a transformation chain named maskPassword and addUUID",
    "transforms": "maskPassword,addUUID",
    
    "//": "Transformation 1 configuration: Hides the password column content",
    "transforms.maskPassword.type": "org.apache.kafka.connect.transforms.MaskField$Value",
    "transforms.maskPassword.fields": "password",
    "transforms.maskPassword.replacement": "[HIDDEN]",
    
    "//": "Transformation 2 configuration: Inserts the Kafka topic name into the record",
    "transforms.addUUID.type": "org.apache.kafka.connect.transforms.InsertField$Value",
    "transforms.addUUID.topic.field": "kafka_topic_origin"
  }
}

Critical SMT Limitations #

Although SMTs are very practical, we must be careful not to misuse them. SMTs must not be used for:

• Data aggregation across multiple messages (for example calculating total sales per minute).
• Dynamically joining data with other topics or databases.
• Complex business logic calculations consuming network I/O time.

If our data pipeline needs the complex operations above, we must move that logic to an actual stream processing layer using Kafka Streams or Apache Flink.

Summary #

• Kafka Connect — A declarative configuration-based data integration framework for moving data in (Source) and out (Sink) of Apache Kafka without writing custom code.
• Connector vs Task — Connectors define configuration and logically divide workloads, while Tasks are parallel execution units running on workers.
• Workers — Kafka Connect’s runtime engine that can run in Standalone mode (for dev) or Distributed mode (for production with high scalability and automatic fault tolerance).
• Converters — The important component bridging the internal schema-value data model with physical representations (like Avro, Protobuf, or JSON) when interacting with Kafka brokers.
• Single Message Transforms (SMT) — Lightweight single-row transformations operating inside JVM memory for cleaning, type manipulation, or sensitive data masking before serialization.