Access Control Lists (ACL): Managing Authorization in Apache Kafka #

After we successfully authenticate clients using mTLS or SASL, the next step is limiting what those clients can do inside our Kafka cluster. Without access restrictions (authorization), authenticated clients can freely read sensitive data belonging to other teams, publish junk messages to financial topics, or even accidentally delete important topics.

In Apache Kafka, authorization is natively managed using Access Control Lists (ACL). ACLs act as security policy matrices precisely defining “Who” may do “What Actions” to “Which Resources” from “Which Hosts”.

This article will thoroughly unpack the structural components forming ACL rules in Kafka, how to enable the Zero-Trust (Deny-by-Default) security policy, practical ACL management guides using the kafka-acls.sh CLI command-line tool for producers, consumers, and transactions, plus the ACL rule evaluation priority logic on brokers.

Dissecting Kafka ACL Policy Structures #

One ACL rule in Apache Kafka is declaratively defined using six main forming components. Let’s dissect those components:

+-----------------------------------------------------------------------------+
|                           ONE ACL RULE ENTRY                                |
|                                                                             |
|   1. PRINCIPAL    : Who is the user? (e.g., User:payment-service)           |
|   2. HOST         : Where is the connection from? (e.g., 192.168.1.50 or *) |
|   3. OPERATION    : What action? (e.g., Read, Write, Describe)              |
|   4. RESOURCE TYPE: What target type? (e.g., Topic, Group)                  |
|   5. PATTERN TYPE : How to match names? (Literal vs Prefixed)               |
|   6. PERMISSION   : Allowed or denied? (Allow vs Deny)                      |
+-----------------------------------------------------------------------------+

1. Principal (User Identity) #

The verified username from the authentication phase. The writing format is Type:Name (e.g., User:payment-service or User:CN=producer-app,O=Corp if using mTLS without mapping). We can use the User:* wildcard to apply rules to all registered users.

2. Host (Connection Origin Address) #

Limits which client IP addresses this rule applies to. This is very useful for limiting critical applications to only be accessible from certain network segments (e.g., 192.168.10.50). If we don’t want to limit by physical IP, we can set it to the * wildcard (meaning from any host).

3. Resource Type #

Determines the asset categories inside Kafka we want to secure. Commonly used categories include:

  • Topic: Kafka topics where data is stored.
  • Group: Consumer Group IDs used to track consumer read offsets.
  • Cluster: Represents the Kafka cluster itself (used for administrative actions like creating topics or monitoring internal metrics).
  • TransactionalId: IDs used by transactional producers to guarantee Exactly-Once Semantics.

4. Pattern Type (Name Matching Method) #

Determines how Kafka matches the resource name strings we define against real resources in the cluster:

  • Literal: Names must match exactly (e.g., a rule for the finance-transactions topic only applies to that topic).
  • Prefixed: Rules apply to all resources starting with a specific string (e.g., the dept-sales- prefix covers the dept-sales-invoice, dept-sales-leads topics, etc.).
  • Wildcard (*): Applies to all resources of that type.

5. Operation (Action) #

Defines the specific operations allowed or forbidden. Example operations include:

  • Read and Write (for reading/writing data).
  • Describe (for viewing topic metadata, like partition counts).
  • Create and Delete (for creating/deleting topics).
  • Alter (for changing topic configurations).
  • All (covers all operations above).

6. Permission Type #

Only has two values: Allow (permitting access) or Deny (absolutely forbidding access).

Enabling the Zero-Trust Policy: Deny-by-Default #

By default, if we just enable the authorization module in Kafka, brokers have the built-in parameter allow.everyone.if.no.acl.found=true. That means, if a topic has no ACL rules at all, all clients (including anonymous users) are allowed to access that topic. This is certainly not good security practice for production.

We must implement the Zero-Trust (Deny-by-Default) policy, where all access is denied unless explicitly permitted by ACLs.

Broker Configuration (server.properties) #

To enable it in KRaft-based cluster environments (Kafka 3.x+), add the following lines to the broker configuration:

# Enabling the built-in KRaft Standard Authorizer
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer

# Locking the cluster: If no ACL exists, access is absolutely denied
allow.everyone.if.no.acl.found=false

# Registering cluster admin users (super users) immune to ACL rules
# (Super users always have full access to all administrative operations)
super.users=User:admin;User:kafka-broker

Note: If your cluster still uses ZooKeeper, replace the authorizer class with kafka.security.authorizer.AclAuthorizer.

Practical ACL Management Guides with the kafka-acls.sh CLI #

Kafka provides a command-line tool called kafka-acls.sh for manipulating ACL entries directly in cluster metadata.

Here are operational scenarios often encountered in production along with their CLI execution commands:

flowchart TD
    subgraph ClientApps["Client Applications"]
        Prod["Producer App (User: payment-prod)"]
        Cons["Consumer App (User: payment-cons)"]
    end

    subgraph KafkaBroker["Kafka Broker (ACL Rules)"]
        StandardAuth{"Standard Authorizer"}
        
        Rule1["ACL Allow Write/Describe on Topic 'payment-logs'"]
        Rule2["ACL Allow Read/Describe on Topic 'payment-logs'"]
        Rule3["ACL Allow Read on Group 'payment-group'"]
        
        StandardAuth --> Rule1 & Rule2 & Rule3
    end

    Prod -->|1. Write Record| StandardAuth
    Cons -->|2. Read Record| StandardAuth
    
    Rule1 -.->|Allow payment-prod| Prod
    Rule2 -.->|Allow payment-cons| Cons
    Rule3 -.->|Allow Group Access| Cons

1. ACL Configuration for Standard Producers #

Producers need Write and Describe permissions on target topic names to send messages and read partition metadata.

kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config client-ssl.properties \
  --add \
  --allow-principal User:payment-prod \
  --operation Write \
  --operation Describe \
  --topic payment-logs \
  --resource-pattern-type literal

2. ACL Configuration for Standard Consumers #

Consumers need Read and Describe permissions on topics, plus Read permission on the consumer group ID used to store offsets.

# Granting read permission on the 'payment-logs' topic
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config client-ssl.properties \
  --add \
  --allow-principal User:payment-cons \
  --operation Read \
  --operation Describe \
  --topic payment-logs \
  --resource-pattern-type literal

# Granting permission to join the 'payment-analytics-group' Consumer Group
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config client-ssl.properties \
  --add \
  --allow-principal User:payment-cons \
  --operation Read \
  --group payment-analytics-group \
  --resource-pattern-type literal

3. ACL Configuration for Transactional Producers (Exactly-Once Semantics) #

Transactional producers need additional authorization on the TransactionalId they use so brokers can verify secure transactions.

kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config client-ssl.properties \
  --add \
  --allow-principal User:payment-prod-tx \
  --operation Write \
  --operation Describe \
  --transactional-id tx-payment- \
  --resource-pattern-type prefixed

Note: Using --resource-pattern-type prefixed above allows transactional producers to use any transaction ID starting with tx-payment- (like tx-payment-001, tx-payment-002) without needing to create new ACLs for every transaction ID.

4. ACL Configuration for Administrative Tasks #

Data operations teams often need to do cluster maintenance like adding partitions, doing partition reassignment, or monitoring cluster metrics without being full super.users, to comply with the principle of least privilege.

To run those commands, admin users need permissions on the high-level Cluster resource:

# Granting administrative permissions to the 'ops-engineer' user
# to change cluster configurations and do partition rebalancing
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config client-ssl.properties \
  --add \
  --allow-principal User:ops-engineer \
  --operation Alter \
  --operation ClusterAction \
  --operation Describe \
  --cluster
  • --cluster: Marks that the target resource is at the global cluster level (named kafka-cluster).
  • ClusterAction: Special rights for doing replica balancing commands and inter-broker quorum monitoring.
  • Alter: Allows dynamic broker configuration manipulation at runtime.

5. Removing ACL Rules #

If an application is decommissioned, we must remove its ACL rules to keep cluster metadata clean.

kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config client-ssl.properties \
  --remove \
  --allow-principal User:payment-prod \
  --operation Write \
  --topic payment-logs \
  --force

ACL Evaluation Logic on Brokers (Precedence Rules) #

When a client sends a request to a Kafka broker (e.g., a message write request), the Broker Authorizer evaluates the existing ACL list to make a decision: Allow or Deny.

This evaluation logic runs following the priority rules below strictly:

flowchart TD
    A["Client Sends a Request"] --> B{"Is the Principal a SuperUser?"}
    B -- "YES" --> ALLOW1["ALLOW"]
    B -- "NO" --> C{"Is there a matching DENY rule?"}
    C -- "YES" --> DENY1["DENY (Absolute)"]
    C -- "NO" --> D{"Is there a matching ALLOW rule?"}
    D -- "YES" --> ALLOW2["ALLOW"]
    D -- "NO" --> E["Deny Access (Default)"] --> DENY2["DENY"]

The Three Golden Rules of ACL Evaluation: #

  1. Super Users Always Pass: If the user is registered in the broker’s super.users parameter, the authorization process immediately ends with an Allow decision, ignoring all other ACL rules.
  2. Deny Beats Allow (Deny Precedence): If there’s a specific Deny rule matching the client’s principal, host, and operation, access is immediately denied (Deny), even if there’s simultaneously an Allow rule permitting it.
  3. Literal Matching Beats Prefixed: When searching for matching rules, exact (Literal) name matches are evaluated first before switching to prefix (Prefixed) matching, and finally matching the * wildcard.

ACL Authorization Error Auditing & Diagnostics #

In daily operations, ACL configuration errors often trigger errors on client application sides. Without clear logs, developer teams struggle to know whether connection failures come from network issues, authentication errors (wrong passwords), or authorization failures (ACLs not registered).

1. Enabling Broker Authorization Audit Logging #

By default, Kafka brokers separate authorization logs into a special file called kafka-authorizer.log so it doesn’t flood the main log (server.log). We can configure this log’s verbosity level through the log4j.properties file on each broker:

# Separating authorization logs into a special appender
log4j.logger.kafka.authorizer.logger=INFO, authorizerAppender
log4j.additivity.kafka.authorizer.logger=false

# Setting the daily rolling file for authorization logs
log4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.authorizerAppender.File=${kafka.logs.dir}/kafka-authorizer.log
log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

2. Reading Authorization Failure Log Lines #

When a client is denied access, the kafka-authorizer.log file records entry lines like this:

[2026-06-08 20:42:00,123] INFO Principal = User:payment-cons is Denied Operation = Read from Host = 192.168.1.50 on Resource = Topic:LITERAL:payment-logs (kafka.authorizer.logger)

Log Entry Dissection:

  • Principal = User:payment-cons: The detected client identity.
  • is Denied: The authorizer’s final decision (denied).
  • Operation = Read: The action type the client attempted (reading data).
  • Host = 192.168.1.50: The physical IP of the client application server.
  • Resource = Topic:LITERAL:payment-logs: The target asset being a topic named payment-logs with exact (literal) name matching.

By reading this log, cluster administrators can quickly identify that user payment-cons needs a new ACL for the Read operation on the payment-logs topic.

3. Exceptions on Client SDKs #

On Java client application sides, authorization failures are thrown as specific exception classes:

  • TopicAuthorizationException: Thrown when producers try to send() to a topic without Write permission, or consumers try to poll() from a topic without Read permission.
  • GroupAuthorizationException: Thrown when consumers try to join a Consumer Group (e.g., during initialization) but the user doesn’t have Read permission on the Group resource for that group ID.
  • ClusterAuthorizationException: Happens when applications try to do administrative operations (like creating topics via AdminClient or making configuration changes) without cluster-level administrative permissions (Cluster resource).

Summary #

  • ACL Authorization — The gate determining client actions. Must be configured after authentication is enabled so the cluster is protected granularly.
  • Zero-Trust — Set allow.everyone.if.no.acl.found=false on production brokers so all access is denied by default unless there’s valid permission.
  • Prefixed Patterns — Leverage prefix matching (prefixed) to minimize the number of ACL entries on topics or transaction IDs with standardized name patterns.
  • Deny Precedence — Deny is absolute. If a client is hit by one Deny rule, it can’t access the resource even with an Allow rule.
  • Consumer Group Auth — Remember that consumers don’t only need Read access on Topics, but also Read access on Group so it doesn’t trigger GroupAuthorizationException errors.

← Previous: Client Authentication Next: Top-Level Security →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact