Client Authentication: Securing Client Access to Kafka Brokers #

Client authentication is the first defense gate in securing Apache Kafka clusters. Without clear authentication mechanisms, Kafka brokers never know who the entity sending or pulling messages really is. As a result, we can never implement advanced authorization policies (Access Control Lists or ACLs) because we can’t match caller identities with their access rights.

Apache Kafka provides two main methods for authenticating clients: Mutual TLS (mTLS) based on digital certificates, and SASL (Simple Authentication and Security Layer) supporting various verification mechanisms like usernames/passwords, Kerberos tickets, and OAuth2 tokens.

This article will deeply dissect the technical configuration of both authentication methods, explain how Principal Mapping Rules work in mTLS, dissect JAAS (Java Authentication and Authorization Service) configuration files, and provide ready-to-use properties file examples for Java SDK clients.

Mutual TLS (mTLS)-Based Authentication #

When we enable SSL/TLS encryption in Kafka, the default handshake is one-way. Brokers present their certificates to clients to prove their identity and encrypt the communication path. However, the broker doesn’t ask who the client is.

To turn this into Mutual TLS (mTLS), we set the following broker security parameter in server.properties:

# Setting the broker to mandatorily request and verify certificates from clients
ssl.client.auth=required

1. Forming Client Principals from x509 Certificates #

When an mTLS handshake succeeds, the Kafka broker extracts the subject name from the client’s x509 certificate (Distinguished Name / DN) and uses it as the Principal username. By default, Kafka uses the full subject DN.

For example, if our client certificate was created with the command: keytool -dname "CN=payment-service,OU=Finance,O=Badri Creative Tech,C=ID"

Then, the Kafka broker automatically recognizes that client as the principal: User:CN=payment-service,OU=Finance,O=Badri Creative Tech,C=ID

2. Simplifying Principals via Principal Mapping Rules #

The full DN format above is very long and troublesome when we must write ACL (Access Control Lists) regulations. We surely prefer concise principal names like User:payment-service.

To convert it dynamically, Kafka provides the ssl.principal.mapping.rules parameter. This parameter uses Regex-based syntax to extract specific parts from certificate DNs.

Here’s an example mapping rule in the broker’s server.properties:

# Translating Distinguished Names (DN) into single usernames (simple principals)
ssl.principal.mapping.rules=RULE:^CN=([^,]*)(,.*|$)/$1/,DEFAULT

How the Rule Above Works:

The rule matches certificate DN strings starting with CN=.
The first capturing group ([^,]*) takes the character values after CN= until the first comma , (in our example, it extracts the string payment-service).
That value is replaced by $1 (the extraction result), so the final principal recognized by the Kafka Authorizer is: User:payment-service.
If the DN doesn’t match that rule, it uses the built-in DEFAULT rule (full DN).

3. Complex Mapping Rules #

Kafka supports arranging multiple mapping rules separated by commas, evaluated sequentially from left to right (or top to bottom) until the first match is found. The general mapping rule syntax is: RULE:pattern/replacement/[L|U]

pattern: A Java regular expression used to match certificate subject DN strings.
replacement: The replacement string supporting capturing group reference expressions (like $1, $2).
L / U: Optional modifiers to change the final result to all lowercase (L) or all uppercase (U).

Here’s a multi-rule configuration example commonly used in enterprise environments:

# Rule 1: Extract CN and OU, combine both, and change to lowercase (L)
# Input format: CN=Payment-App,OU=Finance,O=Badri Creative Tech,C=ID
# Output format: User:payment-app-finance
ssl.principal.mapping.rules=RULE:^CN=([^,]*),OU=([^,]*),O=.*$/$1-$2/L,\
                            RULE:^CN=([^,]*)(,.*|$)/$1/L,\
                            DEFAULT

By applying the layered rules above:

Connections with certificates containing the Finance organizational unit (OU) are neatly mapped to User:payment-app-finance automatically in lowercase, minimizing case-sensitivity.
Connections containing only CN are mapped to User:<cn> in lowercase.
Other connections not matching the regex patterns use the full DN identity as a fallback (DEFAULT).

SASL (Simple Authentication and Security Layer)-Based Authentication #

If our organization prefers credential-based authentication over distributing client SSL certificates to hundreds of application servers, then SASL is the right choice.

In Java environments (including Kafka), SASL authentication is configured through the JAAS (Java Authentication and Authorization Service) module. This module defines which libraries are used to verify credentials.

Let’s dissect the configuration details for each main SASL mechanism:

1. SASL/PLAIN: Static Username & Password #

The SASL/PLAIN mechanism is the simplest. We define credentials in the broker JAAS configuration file.

Broker JAAS File (kafka_server_jaas.conf):

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-secure-password"
    user_producer_app="producer-app-secret-pass"
    user_consumer_app="consumer-app-secret-pass";
};

In the configuration above, the admin user acts as the internal user used by brokers to coordinate with each other, while producer_app and consumer_app are credentials external clients can use to connect to brokers.

2. SASL/SCRAM: Dynamic Hash Challenges #

The SASL/SCRAM mechanism is far safer because passwords aren’t sent to brokers (using the Salted Challenge Response scheme). Credentials aren’t hardcoded in files, but stored in cluster metadata.

Broker JAAS File (kafka_server_jaas.conf):

KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required;
};

To register new users at runtime without broker restarts, run the following CLI command:

kafka-configs.sh --bootstrap-server localhost:9093 \
  --entity-type users --entity-name transaction_service \
  --alter --add-config 'SCRAM-SHA-512=[password=tx_secure_pass_123]'

3. SASL/GSSAPI (Enterprise Kerberos) #

For enterprise-scale centralized authentication integration (Single Sign-On), Kerberos is the standard.

Broker JAAS File (kafka_server_jaas.conf):

KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/kafka/secrets/kafka.keytab"
    principal="kafka/[email protected]";
};

useKeyTab=true: Tells the JVM to use the encrypted keytab credential file rather than asking for interactive passwords.
principal: The broker identity principal name on the Kerberos KDC server.

The SASL/SCRAM Authentication Handshake Flow in Kafka #

To clarify the authentication handling differences on networks, here’s the information exchange (handshake) flow when clients try to log in using the SASL/SCRAM-SHA-256 mechanism:

sequenceDiagram
    autonumber
    participant Client as "Kafka Client"
    participant Broker as "Kafka Broker"
    participant Metadata as "KRaft / ZooKeeper Metadata"

    Note over Client, Broker: Prerequisite: Secure SSL/TLS connection path active (SASL_SSL)
    Client->>Broker: Initial connection & SASL Mechanism negotiation (SCRAM-SHA-256)
    Broker-->>Client: Mechanism approval confirmation
    
    Client->>Broker: SASL First Message (Send Username: user_app & client_nonce)
    Note over Broker: Broker reads user_app's salt & iterations from Metadata
    Broker->>Metadata: Fetch salt & iterations for user_app
    Metadata-->>Broker: Return salt & hash iterations
    
    Broker->>Client: SASL Server First Message (Send server_nonce, salt, iterations)
    Note over Client: Client calculates SaltedPassword & ClientProof using the original password
    Client->>Broker: SASL Client Final Message (Send ClientProof)
    
    Note over Broker: Broker cryptographically validates ClientProof
    Broker-->>Client: SASL Server Final Message (Success, Send ServerSignature)
    Note over Client: Client verifies the ServerSignature (Mutual validation)

Ready-to-Use Configuration Files for Java Client Applications #

Here are complete configuration property examples that must be installed on Java SDK client application code (both Producer and Consumer) for each protocol:

1. Client Properties for Mutual TLS (mTLS) #

Client applications need their own keystore files containing certificates signed by the cluster root CA.

# Setting the mTLS security protocol
security.protocol=SSL

# Truststore location configuration (root CA certificates to verify brokers)
ssl.truststore.location=/var/private/ssl/client.truststore.p12
ssl.truststore.password=client-truststore-pass
ssl.truststore.type=PKCS12

# Keystore location configuration (client identity certificates to be verified by brokers)
ssl.keystore.location=/var/private/ssl/client.keystore.p12
ssl.keystore.password=client-keystore-pass
ssl.keystore.type=PKCS12

2. Client Properties for SASL/SCRAM (Salted Password) #

Client applications send password credentials securely using SCRAM hashing. Credentials are configured through the inline JAAS parameter sasl.jaas.config.

# Combining SASL authentication with SSL data encryption
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512

# Truststore Location (Required for SSL encryption)
ssl.truststore.location=/var/private/ssl/client.truststore.p12
ssl.truststore.password=client-truststore-pass
ssl.truststore.type=PKCS12

# Client JAAS Configuration inline (without using a separate file)
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="transaction_service" \
  password="tx_secure_pass_123";

3. Client Properties for SASL/PLAIN #

If your cluster is forced to use the PLAIN mechanism, make sure the sasl.jaas.config parameter references PlainLoginModule.

security.protocol=SASL_SSL
sasl.mechanism=PLAIN

ssl.truststore.location=/var/private/ssl/client.truststore.p12
ssl.truststore.password=client-truststore-pass
ssl.truststore.type=PKCS12

# Client JAAS using PlainLoginModule
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="producer_app" \
  password="producer-app-secret-pass";

Advanced SASL/OAUTHBEARER & Kerberos Integration #

For enterprise-scale environments prioritizing seamless automation, authentication using OAuthbearer or Kerberos offers high security without manual intervention. However, implementing them demands precise JVM runtime configuration and callback handlers.

1. Token Refresh on SASL/OAUTHBEARER #

JWTs (JSON Web Tokens) issued by authentication servers (IdPs) are usually very short-lived (e.g., 15 minutes to 1 hour) to minimize security risks if tokens are stolen. If tokens expire, client connections to Kafka are disconnected.

To handle this, Java Kafka clients have internal mechanisms to automatically refresh tokens before their validity ends. This is configured using the Login Callback Handler class.

Since Kafka 3.0.0, Kafka provides a robust built-in implementation to automate this cycle:

# Using the OAUTHBEARER protocol
security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER

# Registering Kafka's built-in Callback Handler for OAuth2
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler

# Parameters for communicating directly to the OAuth2 Identity Provider (IdP) server
sasl.oauthbearer.token.endpoint.url=https://keycloak.internal/auth/realms/kafka/protocol/openid-connect/token
sasl.oauthbearer.client.id=payment_service_app
sasl.oauthbearer.client.secret=my-client-secret-key-from-keycloak

# Client JAAS Configuration
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

How It Works: The OAuthBearerLoginCallbackHandler contacts the OAuth2 Token endpoint URL, sends client_id and client_secret to obtain a new JWT token, then sends it to the Kafka broker. As the token approaches expiration, this handler automatically re-requests the IdP for a new token and refreshes the session on the broker without disconnecting the active TCP connection (zero disconnect).

2. Kerberos Ticket Rotation & JVM Differences #

When implementing SASL/GSSAPI (Kerberos), we often face compatibility issues from different JVM vendors used on client application servers.

Kerberos login module libraries differ between Oracle JDK / OpenJDK and IBM JDK (often used on WebSphere/AIX enterprise servers):

Oracle JDK / OpenJDK:

# Using the Krb5LoginModule class from Oracle/Sun
sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
  useKeyTab=true \
  storeKey=true \
  keyTab="/etc/security/keytabs/client.keytab" \
  principal="[email protected]";

IBM JDK:

# Using the Krb5LoginModule class from IBM
sasl.jaas.config=com.ibm.security.auth.module.Krb5LoginModule required \
  useKeytab=true \
  credsType=both \
  keytab="/etc/security/keytabs/client.keytab" \
  principal="[email protected]";

B. Automatic Ticket Rotation Parameters (Ticket Renewal) #

Kerberos tickets have limited validity periods (e.g., 24 hours). If tickets expire, clients can’t make new connections. Kafka clients provide parameters for configuring how background threads refresh tickets before expiration:

sasl.kerberos.ticket.renew.window.factor: Determines the percentage of remaining ticket lifetime before the thread starts trying to renew it (Default: 0.80, meaning if a ticket lasts 10 hours, the renewal process starts at hour 8).
sasl.kerberos.ticket.renew.jitter: Adds random values (jitter) to renewal times to prevent all client instances on thousands of servers from attacking Kerberos KDC servers simultaneously (thundering herd problem, Default: 0.05).

Summary #

mTLS vs SASL — mTLS is highly recommended if your infrastructure has adopted certificate management automation (Kubernetes/Vault). SASL suits dynamic integration without distributing public/private key files to client servers.
Encryption Is Mandatory — SASL doesn’t encrypt data traffic binaries. SASL operations in production must always be combined with SSL (SASL_SSL) to protect against data leaks.
Dynamic SCRAM Config — Use SASL/SCRAM rather than SASL/PLAIN so credential management is dynamic via cluster metadata without requiring periodic broker restarts.
Principal Mapping — Set the ssl.principal.mapping.rules parameter on brokers so principal names extracted from x509 certificates are neat and easy to manage in ACL regulations.

← Previous: Certificate & Key Management Next: Access Control Lists (ACL) →

Client Authentication: Securing Client Access to Kafka Brokers #

Mutual TLS (mTLS)-Based Authentication #

1. Forming Client Principals from x509 Certificates #

2. Simplifying Principals via Principal Mapping Rules #

3. Complex Mapping Rules #

SASL (Simple Authentication and Security Layer)-Based Authentication #

1. SASL/PLAIN: Static Username & Password #

2. SASL/SCRAM: Dynamic Hash Challenges #

3. SASL/GSSAPI (Enterprise Kerberos) #

The SASL/SCRAM Authentication Handshake Flow in Kafka #

Ready-to-Use Configuration Files for Java Client Applications #

1. Client Properties for Mutual TLS (mTLS) #

2. Client Properties for SASL/SCRAM (Salted Password) #

3. Client Properties for SASL/PLAIN #

Advanced SASL/OAUTHBEARER & Kerberos Integration #

1. Token Refresh on SASL/OAUTHBEARER #

2. Kerberos Ticket Rotation & JVM Differences #

A. JAAS Login Module Differences #

B. Automatic Ticket Rotation Parameters (Ticket Renewal) #

Summary #