Certificate & Key Management: Managing Keystores and Truststores in Kafka #

Public Key Infrastructure (PKI)-based cryptographic security is the main foundation of data traffic encryption and Mutual TLS authentication in Apache Kafka. In distributed cluster environments, every broker and client must be able to cryptographically prove their identity using standard x509 digital certificates signed by a trusted party (Certificate Authority / CA).

However, managing digital certificates in the Java Virtual Machine (JVM)—the runtime where Apache Kafka runs—is often confusing for both developers and operations teams. We must deal with the Keystore and Truststore concepts, the complex Java keytool commands, and certificate rotation strategies so clusters don’t suddenly die when certificate validity periods expire.

This article will thoroughly unpack the conceptual differences between Keystores and Truststores, provide practical step-by-step guides for creating internal Certificate Authorities (CAs) using OpenSSL and keytool, and explain production certificate rotation strategies without causing cluster downtime.

Understanding the Conceptual Differences: Keystore vs Truststore #

In the Java Security world, certificates and private keys are stored in special repositories in encrypted binary file forms. The two storage file types used in Kafka are the Keystore and the Truststore. Technically, both files can use the same format (like .jks for the standard Java KeyStore, or .p12 for the industry-standard PKCS12 format). However, their functional purposes in the security architecture are very different.

flowchart LR
    subgraph JVM["JVM MEMORY ARCHITECTURE"]
        direction LR
        Keystore["KEYSTORE (Who am I?)<br/>- Broker Private Key<br/>- Broker Public Certificate<br/>- Full Certificate Chain"]
        Truststore["TRUSTSTORE (Who do I trust?)<br>- Public Root CA Certificate<br>- Intermediate CA Certificates"]
    end

1. Keystore: Proving Your Own Identity (“Who Am I?”) #

Keystores store the personal identity credentials of the entity where they run (whether brokers or clients).

  • Keystore Contents: The private key that must be kept tightly secret, the entity’s public key certificate, and the certificate chain connecting it to the Root CA.
  • Function: When a broker or client wants to open a secure connection, it uses the Keystore contents to prove its identity to the other party.
  • Analogy: A Keystore is like Our Personal Passport. The passport contains our identity data and only we have the right to hold it (because our private key is inside).

2. Truststore: The Trust List (“Who Do I Trust?”) #

Truststores are used to verify the validity of certificates presented by external entities trying to connect with us.

  • Truststore Contents: Only contains public certificates from trusted third parties, especially the Root CA certificate and Intermediate CA certificates. Truststores must not store any private keys.
  • Function: When an external entity presents its certificate to prove itself, we match the signature on that certificate against the list of trusted CA certificates stored in our Truststore. If that certificate was signed by one of the CAs in our Truststore, the connection is allowed.
  • Analogy: A Truststore is like the List of Official Government Institutions recognized by our country to validate whether a foreigner’s passport entering our territory is valid or fake.

Step-by-Step Guide to Creating an Internal CA and Broker Certificates #

Because Apache Kafka clusters are usually inside private corporate internal networks and aren’t directly accessed by the general public via the internet, we don’t need to buy certificates from public CAs (like DigiCert or GlobalSign). We also can’t easily use free services like Let’s Encrypt because DNS/HTTP verification requires internet exposure.

The industry-standard approach is to create our own internal Certificate Authority (self-signed Root CA), then use that root CA to sign all broker and client certificates in our environment.

Here’s the complete certificate creation workflow using the openssl program and Java keytool (recommended using the modern .p12 / PKCS12 format):

flowchart TD
    subgraph CA_Server["1. Certificate Authority (CA Server)"]
        CA_Key["Root CA Private Key (ca-key)"]
        CA_Cert["Root CA Certificate (ca-cert)"]
        CA_Key --> CA_Cert
    end

    subgraph Broker_Server["2. Kafka Broker Server"]
        B_Keystore["Broker Keystore (kafka.server.keystore.p12)"]
        B_CSR["Certificate Signing Request (CSR)"]
        B_Keystore -->|Generate Keypair| B_CSR
    end

    B_CSR -->|3. Send CSR| CA_Cert
    CA_Cert & CA_Key -->|4. Sign the Certificate| B_Cert["Signed Certificate (cert-signed)"]
    
    B_Cert -->|5. Import| B_Keystore
    CA_Cert -->|6. Import Trust Chain| B_Keystore
    CA_Cert -->|7. Create| B_Truststore["Broker Truststore (kafka.server.truststore.p12)"]

Let’s execute the commands one by one in the terminal:

Step 1: Creating the Internal Root CA #

Run this command on a secure server functioning as the CA server (separate this server from production Kafka brokers for security):

# 1. Generate the Root CA Private Key (4096-bit key length)
openssl genrsa -out ca-key.pem 4096

# 2. Create the Root CA Public Certificate valid for 10 years (3650 days)
openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days 3650 \
  -subj "/CN=Kafka-Internal-Root-CA/OU=IT Security/O=Badri Creative Tech/C=ID"

Step 2: Creating the Keystore for the Kafka Broker #

Run this command on the Kafka broker server. We create a PKCS12-format binary keystore (deststoretype pkcs12 is the modern Java default):

# Generate a keypair (private & public key) directly inside the keystore file
keytool -keystore kafka.broker.keystore.p12 \
  -alias localhost \
  -validity 365 \
  -genkey -keyalg RSA -keysize 2048 \
  -dname "CN=kafka-broker-1.internal,OU=Data Platform,O=Badri Creative Tech,C=ID" \
  -storepass broker-keystore-secret-pass

Step 3: Creating a Certificate Signing Request (CSR) #

A CSR is a request file containing the broker’s public key and identity data, which we send to the CA server for signing.

keytool -keystore kafka.broker.keystore.p12 \
  -alias localhost \
  -certreq -file broker.csr \
  -storepass broker-keystore-secret-pass

Step 4: Signing the Broker Certificate using the Root CA #

Move the broker.csr file to the CA server, then run the certificate signing. We set the broker certificate validity period to 1 year (365 days):

openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem \
  -in broker.csr -out broker-cert-signed.pem \
  -days 365 -CAcreateserial

Step 5: Importing the Root CA Certificate into the Broker Keystore #

Before importing the signed broker certificate, the keystore must first recognize the root CA that signed it to form a trust chain.

keytool -keystore kafka.broker.keystore.p12 \
  -alias CARoot \
  -import -file ca-cert.pem \
  -storepass broker-keystore-secret-pass -noprompt

Step 6: Importing the Signed Broker Certificate into the Keystore #

Now, re-import the signed certificate we got from the CA into the keystore under the same alias where the keypair was created (localhost):

keytool -keystore kafka.broker.keystore.p12 \
  -alias localhost \
  -import -file broker-cert-signed.pem \
  -storepass broker-keystore-secret-pass

Step 7: Creating the Broker Truststore #

Finally, create a truststore file containing only the trusted root CA certificate so the broker can validate connections from clients:

keytool -keystore kafka.broker.truststore.p12 \
  -alias CARoot \
  -import -file ca-cert.pem \
  -storepass broker-truststore-secret-pass -noprompt

SSL Parameter Configuration on Kafka Brokers #

After the kafka.broker.keystore.p12 and kafka.broker.truststore.p12 files are successfully created and placed in the broker configuration folder (e.g., /etc/kafka/secrets/), we need to update the broker’s server.properties configuration file:

# Enabling the SSL listener on port 9093
listeners=PLAINTEXT://:9092,SSL://:9093
advertised.listeners=PLAINTEXT://kafka-broker-1.internal:9092,SSL://kafka-broker-1.internal:9093

# Determining the inter-broker communication protocol using SSL
security.inter.broker.protocol=SSL

# Keystore Configuration
ssl.keystore.location=/etc/kafka/secrets/kafka.broker.keystore.p12
ssl.keystore.password=broker-keystore-secret-pass
ssl.keystore.type=PKCS12

# Truststore Configuration
ssl.truststore.location=/etc/kafka/secrets/kafka.broker.truststore.p12
ssl.truststore.password=broker-truststore-secret-pass
ssl.truststore.type=PKCS12

# Enabling Mutual Authentication (mTLS) so the broker demands certificates from clients
ssl.client.auth=required

Zero-Downtime Certificate Rotation Strategies in Production #

One of the most common mistakes in Kafka operations is letting certificates expire. When certificate validity periods end (for example after 365 days), Kafka brokers reject new SSL connections and terminate ongoing connections. The cluster experiences a total outage.

To prevent this, we must design periodic certificate rotation strategies. The good news: since Apache Kafka version 2.0.0, we can rotate SSL certificates without restarting Kafka broker processes (zero-downtime certificate rotation).

Dynamic SSL Throttling & Reloading Mechanisms #

Kafka provides advanced configurations that can be dynamically updated while the cluster is running (runtime). We can use the alter configs API (via the kafka-configs.sh CLI command) to force brokers to reload new keystore and truststore files from disk.

Here’s the operational runbook for doing zero-downtime broker certificate rotation:

Step 1: Prepare the New Keystore on the Broker Server #

Create a new PKCS12 keystore containing the updated certificate (for example, the validity period extended 1 year into the future). Place that file in the broker folder with a different name or overwrite the old file. It’s highly recommended to overwrite the old keystore file in the same location for simplicity, e.g., /etc/kafka/secrets/kafka.broker.keystore.p12.

Step 2: Run the Dynamic Reload Command #

After the new keystore file is ready on the broker server disk, run the following CLI command to trigger an instant certificate reload in the broker JVM memory:

# Forcing broker ID 1 to reload the SSL keystore configuration from disk
kafka-configs.sh --bootstrap-server kafka-broker-1.internal:9093 \
  --entity-type brokers --entity-name 1 \
  --alter --add-config 'listener.name.SSL.ssl.keystore.location=/etc/kafka/secrets/kafka.broker.keystore.p12'

[!TIP] When the command above executes, the network threads on the Kafka broker gradually close existing SSL connections and load the new keystore into memory. Producer and consumer clients automatically reconnect and do SSL handshakes using the new certificate transparently without data processing failures.

Step 3: Verify the New Certificate using OpenSSL #

To ensure the broker is truly presenting the new updated certificate, we can scan the broker’s SSL port using the OpenSSL s_client client tool:

openssl s_client -connect kafka-broker-1.internal:9093 -showcerts | openssl x509 -noout -dates

The command above displays the certificate validity date ranges (notBefore and notAfter). Make sure the expiration date has shifted into the future according to our new certificate.

Troubleshooting & Diagnosing SSL Handshake Errors in Kafka #

Operating SSL-based Kafka often brings us face-to-face with SSL Handshake failures. When connections fail, clients usually only receive generic error messages like Connection closed by peer without any details. To diagnose problems, we must log into broker servers and inspect broker logs (server.log) or enable JVM debugging.

1. Enabling Detailed JVM SSL Logging #

To see the byte-by-byte SSL handshake flow, we can add the following JVM option to the Kafka broker startup script or client applications:

# Register the KAFKA_OPTS environment variable before running Kafka
export KAFKA_OPTS="-Djavax.net.debug=ssl,handshake"

When this option is active, the JVM prints all exchanged certificate contents, offered cipher suites, and the exact failure point of the SSL handshake to Standard Output (stdout).

2. Three Classic Kafka SSL Errors & Their Solutions #

A. Custom Key Password Errors (UnrecoverableKeyException) #

  • Symptoms: Brokers fail to start and display a stack trace containing java.security.UnrecoverableKeyException: Cannot recover key.
  • Cause: In the classic Java Keystore (JKS) format, the password for the keystore file (ssl.keystore.password) and the password for the individual private key inside it (key password) can differ. If both differ and we don’t set ssl.key.password specifically, the JVM can’t open the broker’s private key.
  • Solution: Add the ssl.key.password parameter in the server.properties file with your private key password, or when creating modern PKCS12 keystores, make sure the keystore password and private key password are the same.

B. Subject Alternative Name Errors (SAN Missing) #

  • Symptoms: Java clients fail to connect to brokers with the error message: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-broker-1.internal found.
  • Cause: By default, Kafka clients verify whether the target broker hostname matches the identity written in the broker certificate. If broker certificates only use CN=localhost or CN=kafka-broker-1.internal without defining it in the Subject Alternative Name (SAN) extension section, Java’s hostname matching (endpoint identification) rejects it.
  • Solution: We can disable this verification on the client side (highly not recommended for production) by setting ssl.endpoint.identification.algorithm=. The best solution is recreating the broker CSR with complete SAN extension options using keytool:
    keytool -keystore kafka.broker.keystore.p12 -alias localhost \
  -genkey -keyalg RSA -keysize 2048 \
  -ext SAN=dns:kafka-broker-1.internal,ip:192.168.1.10

C. Unknown Trust Chain Errors (Unknown CA / CertificateUnknown) #

  • Symptoms: Broker logs display javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown.
  • Cause: Brokers or clients present certificates signed by a CA not registered in the receiving party’s Truststore. This often happens if we forget to import the Root CA certificate into the client’s truststore file, or conversely when mTLS is active, client certificates aren’t recognized by broker truststores.
  • Solution: Inspect the truststore file contents using the keytool list command:
    keytool -keystore kafka.broker.truststore.p12 -list -v
    Make sure the issuing CA certificate (issuer) of the client/broker certificate is listed inside with trustedCertEntry status.

Summary #

  • Keystore — Contains the private key and public certificate belonging to the broker/client itself. Used as proof of self identity.
  • Truststore — Only contains trusted public root/intermediate CA certificates. Used to validate the authenticity of other entities’ certificates.
  • PKCS12 Format — The .p12 binary format is the modern industry standard highly recommended to replace the old Java .jks built-in format.
  • Internal CA — Internal clusters must use self-created self-signed Root CAs for cost efficiency and full control over cluster authentication.
  • Dynamic Rotation — Never restart brokers just to rotate certificates. Leverage dynamic config reload features via kafka-configs.sh to guarantee 100% system availability.

← Previous: PLAINTEXT vs SSL vs SASL Next: Client Authentication →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact