Encryption in Transit: Securing Kafka Data Traffic with SSL/TLS #

When we stream data through Apache Kafka, messages flow in plaintext format by default across network infrastructure. Without encryption in transit, anyone with physical or logical access to network cables, switches, routers, or even hypervisors in cloud environments can intercept that data traffic (packet sniffing). For sensitive data like financial transaction data, personally identifiable information (PII), or medical records, this negligence can be fatal for regulatory compliance (like GDPR, PCI-DSS, or local data protection laws) and business reputation.

Encryption in transit ensures data sent by producers to brokers, replicated between brokers, and consumed by consumers is cryptographically encoded using the Transport Layer Security (TLS/SSL) protocol. This way, data stays safe from potential third-party interception while on network cables.

In this practical, in-depth guide, we’ll discuss TLS encryption architecture in Kafka, step-by-step building of a self-managed Public Key Infrastructure (PKI) for broker certificates, configuring brokers and Java clients, optimizing Cipher Suites for high performance, and doing certificate rotation without causing production cluster downtime.

SSL/TLS Architecture & Handshake Mechanisms in Kafka #

The key exchange and data traffic encryption mechanism in Apache Kafka relies on the TLS protocol (often still called SSL by Java/Kafka APIs). The TLS Handshake happens at the transport layer before the Kafka protocol (Request/Response API) starts exchanging data.

Let’s study the Mutual TLS (mTLS) handshake flow between Kafka clients and brokers through the diagram below:

sequenceDiagram
    autonumber
    actor Client as Kafka Client (Java SDK)
    participant Broker as Kafka Broker

    Note over Client, Broker: TLS Handshake Initiation (mTLS)
    Client->>Broker: ClientHello (Carries supported Cipher Suites & TLS versions)
    Broker->>Client: ServerHello (Selects the Cipher Suite & TLS version)
    Broker->>Client: Broker Certificate (Broker SSL Certificate)
    Broker->>Client: Certificate Request (Requests client certificates if ssl.client.auth=required)
    
    Note over Client: Verify the Broker Certificate
    Client->>Client: Validates the Broker Certificate with the Local Truststore
    
    Client->>Broker: Client Certificate (Client SSL Certificate)
    Client->>Broker: ClientKeyExchange & CertificateVerify (Symmetric keys & cryptographic signatures)
    
    Note over Broker: Verify the Client Certificate
    Broker->>Broker: Validates the Client Certificate with the Broker Truststore
    
    Broker->>Client: Finished (Active Encrypted Channel)
    Client->>Broker: Finished (Active Encrypted Channel)
    
    Note over Client, Broker: Secure Channel Established (Symmetric Encrypted Kafka Message Exchange)

This handshake process guarantees three information security pillars:

Confidentiality: All Kafka message payloads are encoded using strong symmetric encryption (like AES or ChaCha20).
Integrity: Every transmitted data packet is equipped with a Message Authentication Code (MAC) to detect mid-way data modifications.
Authentication: Both parties prove their identities to each other using digital certificates signed by trusted Certificate Authorities (CAs).

Step-by-Step Guide: Building a Self-Managed PKI for Keystores & Truststores #

For enterprise production environments, we should use certificates issued by company internal CAs (like Active Directory Certificate Services or HashiCorp Vault PKI). However, for deep understanding needs, we’ll build a self-managed CA using the openssl utility and Java keytool directly on the shell.

1. Creating an Internal Certificate Authority (CA) #

This CA acts as the single trusted entity that will sign all broker and client certificates in our cluster.

First, create the CA Private Key and the Root Certificate:

# Creating the CA private key (protected with AES-256 encryption)
openssl genrsa -aes256 -out ca-key.pem 4096

# Creating the CA certificate with a 10-year validity period (3650 days)
openssl req -new -x509 -key ca-key.pem -days 3650 -out ca-cert.pem \
  -subj "/CN=MyCompany Kafka Root CA/OU=IT Security/O=MyCompany/L=Jakarta/C=ID"

The ca-cert.pem file is the public certificate that must be imported into the Truststore of every broker and client so they can validate each other.

2. Creating the Kafka Broker Keystore #

Every Kafka broker needs its own unique certificate stored in a Keystore file (usually JKS or PKCS12 format).

Run this command on every broker server (for example for broker-1):

# 1. Creating a new Keystore and a public/private key pair for broker-1
keytool -keystore kafka.broker1.keystore.jks \
  -alias broker1 \
  -validity 365 \
  -genkey -keyalg RSA -keysize 2048 \
  -dname "CN=broker1.kafka.mycompany.local,OU=Infrastructure,O=MyCompany,L=Jakarta,C=ID" \
  -storepass VerySecret123 \
  -keypass VerySecret123

Note: The CN (Common Name) value must be set to the same as the broker server’s FQDN (Fully Qualified Domain Name) so endpoint identity verification succeeds.

3. Creating a Certificate Signing Request (CSR) for the Broker #

To get a signature from the internal CA, we must export the broker’s public key as a CSR file:

# Exporting the CSR from the broker-1 Keystore
keytool -keystore kafka.broker1.keystore.jks \
  -alias broker1 \
  -certreq \
  -file broker1-cert-file-request.csr \
  -storepass VerySecret123

4. Signing the Broker Certificate Using the CA #

Bring the broker-1 CSR file to the CA server, then sign that certificate:

# Signing the broker-1 CSR with the CA private key
openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem \
  -in broker1-cert-file-request.csr \
  -out broker1-cert-signed.pem \
  -days 365 -CAcreateserial \
  -passin pass:OurCAPassword

5. Importing Certificates into the Broker Keystore #

So the Certificate Chain forms correctly inside the Keystore, we must import the Root CA certificate first, then the signed broker certificate.

# A. Import the Root CA certificate into the broker-1 Keystore (as the trust anchor)
keytool -keystore kafka.broker1.keystore.jks \
  -alias CARoot \
  -import -file ca-cert.pem \
  -storepass VerySecret123 -noprompt

# B. Import the broker-1 certificate signed by the CA
keytool -keystore kafka.broker1.keystore.jks \
  -alias broker1 \
  -import -file broker1-cert-signed.pem \
  -storepass VerySecret123

6. Creating Truststores for Brokers and Clients #

A Truststore is a container only containing trusted public CA certificates. Both brokers and clients only need a copy of the Root CA certificate inside it.

# Creating a new Truststore and importing the Root CA certificate
keytool -keystore kafka.truststore.jks \
  -alias CARoot \
  -import -file ca-cert.pem \
  -storepass TruststoreSecret123 -noprompt

SSL Configuration on the Kafka Broker Side #

After the Keystore (kafka.broker1.keystore.jks) and Truststore (kafka.truststore.jks) are ready on broker servers, we need to configure the broker’s server.properties file to listen on encrypted SSL ports.

Here’s a secure production configuration for Kafka brokers:

# ==============================================================================
# BROKER SSL/TLS CONFIGURATION (server.properties)
# ==============================================================================

# 1. Define the Network Listeners
# We separate client listeners (CLIENT) from internal replication (REPLICATION)
listeners=CLIENT://0.0.0.0:9093,REPLICATION://0.0.0.0:9092
advertised.listeners=CLIENT://broker1.kafka.mycompany.local:9093,REPLICATION://broker1.kafka.mycompany.local:9092

# 2. Associate Security Protocols with Listeners
listener.security.protocol.map=CLIENT:SASL_SSL,REPLICATION:SSL
security.inter.broker.protocol=REPLICATION

# 3. Configure the Keystore and Truststore Locations
# It's better to use a ConfigProvider, but below is the parameter representation
ssl.keystore.location=/var/private/ssl/kafka.broker1.keystore.jks
ssl.keystore.password=VerySecret123
ssl.key.password=VerySecret123

ssl.truststore.location=/var/private/ssl/kafka.truststore.jks
ssl.truststore.password=TruststoreSecret123

# 4. Client Authentication Policy (Mutual TLS)
# - required: Clients must present valid SSL certificates (mTLS).
# - requested: The broker requests certificates, but connections continue if clients don't have them.
# - none: One-way authentication (only clients verify the broker).
listener.name.replication.ssl.client.auth=required
listener.name.client.ssl.client.auth=required

# 5. Additional Security
# Requiring broker FQDNs to be verified against their certificate Common Names
ssl.endpoint.identification.algorithm=HTTPS

SSL Configuration on the Client Side (Producer & Consumer) #

Java client applications (like Spring Boot producers, Kafka Streams, or CLI scripts) must be configured to successfully do SSL handshakes with the cluster.

Here are client configuration properties for mutual authentication (mTLS):

# ==============================================================================
# KAFKA CLIENT - SSL CONFIGURATION (client.properties)
# ==============================================================================

# Define the security protocol
security.protocol=SSL

# Configure the truststore location (clients validate brokers)
ssl.truststore.location=/var/private/ssl/kafka.truststore.jks
ssl.truststore.password=TruststoreSecret123

# Configure the keystore location (clients send certificates to brokers for mTLS)
ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
ssl.keystore.password=ClientStoreSecret123
ssl.key.password=ClientKeySecret123

# Enable broker hostname validation (highly recommended to prevent DNS spoofing)
ssl.endpoint.identification.algorithm=HTTPS

Cipher Suite Choices and Encryption Performance Optimization #

Many organizations hesitate to enable full encryption because they fear performance penalties (CPU overhead and increased latency). However, with the right protocol and Cipher Suite choices, plus modern hardware optimization, this encryption performance cost can be suppressed to below 3-5%.

1. Use TLSv1.3 Instead of TLSv1.2 #

The TLSv1.3 protocol cuts handshake time from 2 round-trips (in TLSv1.2) to just 1 round-trip (1-RTT). Additionally, TLSv1.3 removes old slow and insecure cryptographic algorithms, leaving only AEAD (Authenticated Encryption with Associated Data)-based cipher suites that are fundamentally fast and secure.

In server.properties, limit the allowed TLS versions:

ssl.enabled.protocols=TLSv1.3

2. Choose GCM (Galois/Counter Mode)-Based Cipher Suites #

GCM-based algorithms are much faster because they’re designed to run in parallel at the processor level. If our infrastructure runs on modern x86_64 CPUs, those processors almost certainly have AES-NI (Advanced Encryption Standard New Instructions) hardware instructions. With AES-NI active, AES encryption happens directly at the CPU silicon circuit level, not in JVM software, so speeds multiply.

Here are high-performance Cipher Suite recommendations for our cluster:

# TLSv1.3 Cipher Suite Recommendations
ssl.cipher.suites=TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_CHACHA20_POLY1305_SHA256

Note: Use TLS_CHACHA20_POLY1305_SHA256 if our clients are low-power mobile or IoT devices whose CPUs don’t have AES-NI hardware acceleration.

3. Tuning JVM Parameters & Socket Buffers #

To compensate for the extra packet size load (payload overhead) from TLS encryption, adjust the TCP socket buffer parameters on brokers:

# Increasing socket buffer sizes to handle large encrypted throughput
send.buffer.bytes=1048576
receive.buffer.bytes=1048576

Zero-Downtime SSL Certificate Rotation #

One of the biggest fatal mistakes is letting clusters go down just to rotate expired SSL certificates. Apache Kafka has supported Dynamic SSL Reloading since version 1.1.0 without needing to restart broker JVM processes.

Dynamic Keystore Reload Working Mechanism #

When we trigger dynamic reload configuration, the Kafka Broker reloads the JVM SSL Engine instance, reopens the new keystore and truststore files located on disk, and applies them to new connection handshakes instantly. Previously established encrypted TCP connections (existing connections) keep running with old parameters until those connections naturally close or disconnect.

Practical Certificate Rotation Steps: #

Update the broker’s physical keystore JKS/PKCS12 certificate files in the same disk location (e.g., overwriting the /var/private/ssl/kafka.broker1.keystore.jks file with the new certificate file).
Run the kafka-configs.sh command to trigger reloads on the target broker:

# Triggering a dynamic SSL Keystore reload on broker-1 (node 1)
kafka-configs.sh --bootstrap-server broker1.kafka.mycompany.local:9093 \
  --command-config /etc/kafka/client.properties \
  --entity-type brokers \
  --entity-name 1 \
  --alter \
  --add-config "listener.name.client.ssl.keystore.type=JKS"

Tip: Changing keystore support properties (like changing the keystore type, or re-referencing the same property names) forces the broker to detect file modification date changes on disk and immediately reload new certificates into memory.

Verify whether the new certificate is active using the openssl command:

echo | openssl s_client -connect broker1.kafka.mycompany.local:9093 -servername broker1.kafka.mycompany.local 2>/dev/null | openssl x509 -noout -dates

Check whether the notAfter date has shifted into the future according to the new certificate we installed.

Encryption in Transit Security Audit Checklist #

Use the following compliance checklist to verify our cluster’s data traffic encryption implementation:

No	Security Audit Criteria	Verification Method	Status
1	Disable Unencrypted Ports	Scan port 9092 from outside the VPC; the port must be closed or reject connections.	[ ]
2	Use TLSv1.3	Try forcing TLSv1.1/TLSv1.2 handshakes via openssl: `openssl s_client -connect broker:9093 -tls1_1`. Connections must fail.	[ ]
3	Endpoint Validation (Ident Algorithm)	Make sure `ssl.endpoint.identification.algorithm` is set to `HTTPS` in client configs to prevent Man-in-the-Middle attacks.	[ ]
4	Inter-Broker Port Separation	The inter-broker protocol must run on a separate internal listener (not the client listener).	[ ]
5	Active Client Authentication (mTLS)	If using mTLS, try connecting a Java client without a keystore. Connections must be rejected with a `bad_certificate` message.	[ ]
6	Certificate Validity Audits	Install remaining certificate expiry day monitoring on internal alert systems.	[ ]

Summary #

Disable PLAINTEXT — Always use SSL or SASL_SSL in production environments to protect data confidentiality.
Apply mTLS — Enable mutual authentication (ssl.client.auth=required) to authenticate client identities using digital certificates before they can connect.
Use TLSv1.3 & GCM — Leverage TLSv1.3 performance efficiency and CPU AES-NI hardware acceleration to minimize encryption latency overhead.
Dynamic Rotation — Never shut down brokers just to update SSL certificates. Use the alter configs command in the Kafka Configs CLI to trigger dynamic keystore reloads.

← Previous: Common Security Mistakes Next: Encryption at Rest →