Common Security Mistakes: 8 Fatal Kafka Security Mistakes in Production #

Building secure distributed systems demands very thorough conceptual attention. In Apache Kafka, one small gap in network configuration, certificates, or Access Control Lists (ACLs) can collapse the entire defense we’ve painstakingly built. In production environments, these mistakes don’t only cause potential sensitive data leaks, but are also often the main cause of unplanned cluster downtime.

Many operations teams feel their clusters are already “secure” just because they’ve enabled SSL and SASL. However, after deeper inspection, we often find serious security gaps like leaving JMX ports open without authentication, giving overly loose full wildcard ACL permissions, or forgetting to monitor broker certificate validity periods.

In this in-depth guide, we’ll thoroughly unpack 8 fatal mistakes (security anti-patterns) in securing Kafka clusters in production, analyze the technical consequences behind those failures, provide bad config vs good config comparisons, build automatic expired certificate detection scripts, and provide a comprehensive Production Readiness Security Checklist.

In-Depth Analysis: 8 Fatal Kafka Security Mistakes #

Here are details of the 8 most critical security mistakes often happening in production Kafka clusters along with their failure scenarios:

+---------------------------------------------------------------------------------+
|                        8 FATAL KAFKA SECURITY MISTAKES                         |
|                                                                                 |
|   1. PLAINTEXT on Open Networks  ==>  Data intercepted & manipulated            |
|   2. Hardcoded Passwords in Config ==>  Credentials leak via Git                 |
|   3. Excessive Wildcard ACL (*)  ==>  Over-privileged access rights             |
|   4. Forgetting SSL Certificate Rotation ==>  Cluster dies when cert expires    |
|   5. Mixed Client-Broker Ports   ==>  Internal port resources drained by clients|
|   6. Open Metadata Quorum        ==>  KRaft metadata vulnerable to manipulation |
|   7. No Connection Rate Limit    ==>  Vulnerable to DoS / Connection Leaks      |
|   8. allow.everyone...=true      ==>  Disables the Zero-Trust principle (Default)|
+---------------------------------------------------------------------------------+

1. Letting the PLAINTEXT Protocol Run on Open Networks #

• Mistake: Enabling the PLAINTEXT listener (port 9092) on public network interfaces (0.0.0.0) or letting it be accessible without encryption across inter-data-center (cross-DC) networks. This is often done so developer teams can connect quickly without the hassle of managing TLS certificates.
• Impact: Third parties on the same network path can do packet sniffing using tools like Wireshark or tcpdump. Because data is sent in plain text, confidential information like transaction payloads, user personal data (PII), and even less-secure authentication credentials can be easily read. More dangerously, attackers can launch Man-in-the-Middle (MitM) attacks to inject or modify message payloads sent to crucial topics, leading to downstream app logic damage.
• Mitigation: Completely disable the PLAINTEXT listener in production environments. Configure all external and inter-broker listeners to use encrypted protocols like SSL or SASL_SSL.

2. Hardcoding Credentials in Configuration Files #

• Mistake: Writing SASL usernames/passwords, keystore/truststore passwords, or encryption keys directly (hardcoded) in server.properties configuration files, client JAAS files, or JSON configuration files for Kafka Connect.
• Impact: These configuration files are often accidentally pushed into internal or public company Git repositories. Attackers use automatic scanners (Git scanning tools like GitGuardian or Trufflehog) to detect these secrets within seconds. Once these credentials leak, attackers can infiltrate the cluster from anywhere.
• Mitigation: Use Apache Kafka’s built-in Config Providers library. We can configure Kafka to securely read secrets from environment variables, isolated encrypted local files, or third-party secret management services like AWS Secrets Manager, HashiCorp Vault, or Google Secret Manager at cluster startup.

3. Using the * Wildcard Excessively in ACL Rules #

• Mistake: To save time when developer teams request access, administrators give full wildcard permissions (for example, giving All access on the * topic to specific principals).
• Impact: This policy violates the principle of least privilege. If one client application (e.g., payment-service) is successfully compromised by attackers, those attackers inherit the wildcard access rights. They can freely read data from other sensitive topics (like hr-records), manipulate cluster configurations, create thousands of empty topics to consume memory, or delete important production topics.
• Mitigation: Implement granular per-topic ACLs with specific (Literal) names. If flexibility is needed, use Prefixed Pattern matching disciplinedly (e.g., only allowing access to topics with the payment- prefix) instead of giving the global * wildcard.

4. Ignoring SSL Certificate Lifecycles and Forgetting Rotation #

• Mistake: Creating broker SSL certificates with standard validity periods (e.g., 1 year), but not installing monitoring systems and not designing certificate rotation runbooks.
• Impact: When certificates expire, the broker JVM rejects SSL handshakes with other brokers and clients. Inter-broker data replication immediately stops, triggering under-replicated partitions, and Java SDK clients throw massive SSLHandshakeException errors. The cluster experiences sudden total functional failure.
• Mitigation: Install automatic monitoring systems (like openssl-based diagnostic scripts) to track remaining broker certificate validity periods. Leverage the dynamic SSL reloading feature via the alter configs command in the Kafka Admin API to update keystores/truststores without shutting down broker JVM processes.

5. Mixing Broker Communication Ports (Inter-Broker) with Client Ports #

• Mistake: Using one same SSL/SASL listener port (e.g., port 9093) to serve connections from external client applications and internal inter-broker replication traffic.
• Impact: If the client port is flooded by millions of new connections from client applications experiencing client connection leaks (or connection storms after downtime), all broker network threads are consumed serving external client SSL handshakes. As a result, internal inter-broker replication threads don’t get thread allocations (socket queue starvation). Brokers lose connections to each other, partitions become out-of-sync, and controller watchdog nodes consider those busy brokers dead, triggering cascading failover processes worsening cluster performance degradation.
• Mitigation: Always separate listener ports for internal inter-broker coordination (e.g., port 9092 via SSL) from listener ports for external client applications (e.g., port 9093 via SASL_SSL).

6. Leaving the Metadata Path (KRaft Quorum) Without Encryption and Authentication #

• Mistake: Assuming KRaft Controller Quorum traffic is safe because it’s on a private VPC network, thus leaving the controller port (e.g., port 9094) running without SSL encryption and without authentication.
• Impact: KRaft controls all cluster metadata like partition leadership, topic configurations, and new broker additions. If the controller port is left open without authentication on local networks, attackers who successfully enter one server in the same VPC can send fake metadata packets to the controller quorum. Attackers can seize partition leadership, change replica allocations, or cause split-brain in our Kafka cluster.
• Mitigation: Enable strict SSL encryption and mutual authentication (mTLS) specifically on the controller listener (controller.listener.names=CONTROLLER). Limit controller port access only to fellow KRaft controller members through strict firewall rules (Security Groups).

7. Absence of Connection Rate Limiting #

• Mistake: Not setting new-connection-per-second creation limits at the broker level.
• Impact: Broker JVMs can run out of heap memory (Out of Memory - OOM) and OS file descriptor allocations if attacked by millions of consecutive new TCP connections from client servers that mismanage their producer/consumer connection lifecycles (e.g., instantiating a new producer for every sent message). This is an unintentional Denial of Service (DoS) attack form.
• Mitigation: Strictly configure the max.connection.creation.rate parameter on brokers (especially on listeners connected to external clients) to withstand connection storms. Also set maximum connection limits per IP address using max.connections.per.ip.

8. Leaving allow.everyone.if.no.acl.found=true Active in Production #

• Mistake: Enabling the authorization module (Authorizer) on brokers but leaving Kafka’s default authorization parameter at true.
• Impact: Every new topic without registered ACL rules is left open for anyone to access (free read/write). This policy disables the fundamental Zero-Trust (Deny-by-Default) defense principle. Our cluster becomes insecure because protection only applies to topics explicitly given ACLs.
• Mitigation: Absolutely set allow.everyone.if.no.acl.found=false in production server.properties files. With this setting, if no matching ACL rule exists, access is instantly denied.

Architecture Comparison: Vulnerable vs Secured #

Let’s visualize the architecture design differences between badly configured (Vulnerable) clusters and securely configured (Secured) clusters:

flowchart TD
    subgraph VulnerableCluster["VULNERABLE ARCHITECTURE (NOT SECURE)"]
        direction TB
        ClientV["Client Applications"] -->|Port 9092: PLAINTEXT| BrokerV["Kafka Broker"]
        BrokerV -->|Hardcoded Passwords in Git| GitV[("Git Repository")]
        JMXV["JMX Port 9999 (No Auth)"] -->|Freely Open| InternetV["Internet / Local Network"]
        ACL_V{"allow.everyone... = true"} -.->|Unlimited Access| BrokerV
    end

    subgraph SecuredCluster["SECURED ARCHITECTURE (SAFE & ROBUST)"]
        direction TB
        ClientS["Client Applications"] -->|Port 9093: SASL_SSL / SCRAM| BrokerS["Kafka Broker"]
        BrokerS -->|"ConfigProvider (Secrets)"| SecretStore[("Vault / Secure Local Disk")]
        JMXS["JMX Exporter (HTTP 7071)"] -->|Only Allow Scraping IPs| Prometheus["Prometheus Server"]
        ACL_S{"allow.everyone... = false"} -->|Deny by Default| AuthCheck{"Standard Authorizer (ACL)"}
        AuthCheck -->|Only Authorized Principals| BrokerS
    end

Bad Config vs Good Config #

To help us audit configuration files, here are direct comparison examples between bad practices we often encounter and how to fix them with secure configurations:

1. Credential Handling: Hardcoded Passwords vs FileConfigProvider #

Storing sensitive passwords in server.properties is very dangerous because this file often enters version control systems (Git).

Bad Practice (Hardcoded Credentials) #

# server.properties - BAD CONFIGURATION
ssl.keystore.password=VerySecret123!
ssl.key.password=VerySecret123!
ssl.truststore.password=TrustPasswordSecure99

# JAAS configuration for SASL SCRAM-SHA-512
listener.name.client.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="admin" \
  password="SuperSecretAdminPassword123";

Good Practice (Using FileConfigProvider) #

To fix it, we must enable Kafka’s built-in FileConfigProvider. This service reads password values from isolated external files with strict operating system access rights (e.g., chmod 600 /etc/kafka/secrets.properties).

First, register the config provider in server.properties:

# server.properties - GOOD CONFIGURATION
config.providers=file
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider

Then, reference that external file in the main configuration using the ${file:path:key} syntax:

# server.properties - GOOD CONFIGURATION (Continued)
ssl.keystore.password=${file:/etc/kafka/secrets.properties:keystore_password}
ssl.key.password=${file:/etc/kafka/secrets.properties:key_password}
ssl.truststore.password=${file:/etc/kafka/secrets.properties:truststore_password}

# Reference the JAAS config stored in the external secrets file
listener.name.client.scram-sha-512.sasl.jaas.config=${file:/etc/kafka/secrets.properties:jaas_client_config}

Contents of the external /etc/kafka/secrets.properties file (make sure it can only be read by the kafka process user):

keystore_password=VerySecret123!
key_password=VerySecret123!
truststore_password=TrustPasswordSecure99
jaas_client_config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="SuperSecretAdminPassword123";

2. Authorization Configuration: Wildcard ACL vs Prefixed & Specific ACLs #

Using the global * wildcard gives access to all cluster resources. We must define access rules strictly based on topic names or topic name prefixes (Prefixed).

Bad Practice (Global Wildcard ACL) #

The command below gives the payment-service user full access rights to do any operation on any topic in the cluster.

# BAD CONFIGURATION - Giving unlimited permissions
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config /etc/kafka/client.properties \
  --add \
  --allow-principal User:payment-service \
  --operation All \
  --topic *

Good Practice (Granular & Prefixed ACLs) #

Instead, we must limit the payment-service user to only read from topics with the payment- prefix and only publish messages to the specific transactions-incoming topic.

# GOOD CONFIGURATION - Limit to only reading from topics prefixed with "payment-"
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config /etc/kafka/client.properties \
  --add \
  --allow-principal User:payment-service \
  --operation Read \
  --topic payment- \
  --resource-pattern-type prefixed

# GOOD CONFIGURATION - Limit to only publishing to the "transactions-incoming" topic specifically
kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config /etc/kafka/client.properties \
  --add \
  --allow-principal User:payment-service \
  --operation Write \
  --topic transactions-incoming \
  --resource-pattern-type literal

3. Metrics Exposure: Remote JMX Without Authentication vs Prometheus JMX Exporter #

Enabling raw remote JMX on public ports is a huge security threat because it opens opportunities for Remote Code Execution (RCE) through Java deserialization vulnerabilities.

Bad Practice (Open Remote JMX) #

# BAD CONFIGURATION - Setting JVM parameters in kafka-run-class.sh / environment
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
                       -Dcom.sun.management.jmxremote.port=9999 \
                       -Dcom.sun.management.jmxremote.authenticate=false \
                       -Dcom.sun.management.jmxremote.ssl=false"

Good Practice (Using the Prometheus JMX Exporter Agent) #

To securely monitor internal Kafka metrics, we should use the Prometheus JMX Exporter Java Agent. This agent runs inside the same Kafka JVM process and exports internal metrics in Prometheus-friendly HTTP text format on an isolated local port.

Fix steps:

1. Download the jmx_prometheus_javaagent.jar file.
2. Create the /etc/kafka/jmx_exporter.yaml configuration file to filter only the needed metrics.
3. Configure Kafka to run this agent on the localhost interface (127.0.0.1) on port 7071:

# GOOD CONFIGURATION - Run the JMX Exporter bound to localhost
export KAFKA_OPTS="-javaagent:/usr/share/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/etc/kafka/jmx_exporter.yaml"

With this configuration, the internal JVM JMX port doesn’t need to be opened to external networks. Prometheus services can safely fetch metrics through the HTTP 7071 port protected by firewalls.

Automatic Diagnostic Script: SSL Certificate Expiry Monitoring #

To prevent Kafka cluster outages from expired SSL certificates, we can use the automatic Bash script below. This script has two capabilities:

1. Checking broker certificates remotely through TLS network connections (openssl s_client).
2. Checking the contents of local Java JKS Keystore files (keytool) directly on broker servers.

Bash Script: check-kafka-certs.sh #

We can create this script file on monitoring servers or broker servers, set the appropriate environment variables, and schedule it to run periodically using cron jobs.

#!/usr/bin/env bash
# ==============================================================================
# check-kafka-certs.sh
# Apache Kafka SSL Certificate Validity Monitoring Automation Script
# Alert Criteria: Remaining certificate validity less than or equal to 30 days.
# ==============================================================================

# Alert Threshold Day Configuration
ALERT_DAYS=30

# Mode 1: Remote Network Inspection (Remote TLS Inspection)
CHECK_REMOTE=true
REMOTE_HOST="localhost"
REMOTE_PORT="9093"

# Mode 2: Local JKS Keystore Inspection (Local Keystore Inspection)
CHECK_LOCAL_KEYSTORE=false
KEYSTORE_PATH="/var/private/ssl/kafka.server.keystore.jks"
KEYSTORE_PASS="VerySecret123!"
KEY_ALIAS="caroot"

# Initialize the error status
EXIT_STATUS=0

echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting Kafka certificate inspection..."

# --- MAIN FUNCTION: Evaluate the certificate's remaining days ---
evaluate_expiry() {
    local expiry_date_str="$1"
    local source_desc="$2"

    # Convert the expiry date to epoch seconds format
    # Supports date formats from openssl and keytool
    if [[ "$OSTYPE" == "darwin"* ]]; then
        # macOS OS
        expiry_epoch=$(date -j -f "%b %d %T %Y %Z" "$expiry_date_str" "+%s" 2>/dev/null)
        if [ -z "$expiry_epoch" ]; then
            expiry_epoch=$(date -j -f "%Y-%m-%d" "$expiry_date_str" "+%s" 2>/dev/null)
        fi
    else
        # Linux OS
        expiry_epoch=$(date -d "$expiry_date_str" "+%s" 2>/dev/null)
    fi

    if [ -z "$expiry_epoch" ]; then
        echo "[ERROR] Failed to parse the expiry date: '$expiry_date_str' on $source_desc"
        EXIT_STATUS=1
        return
    fi

    current_epoch=$(date '+%s')
    diff_seconds=$((expiry_epoch - current_epoch))
    diff_days=$((diff_seconds / 86400))

    echo "Certificate [$source_desc]:"
    echo "  - Expiry Date      : $expiry_date_str"
    echo "  - Remaining Validity: $diff_days days"

    if [ "$diff_days" -le 0 ]; then
        echo "[CRITICAL] THE CERTIFICATE HAS EXPIRED ON $source_desc!"
        EXIT_STATUS=2
    elif [ "$diff_days" -le "$ALERT_DAYS" ]; then
        echo "[WARNING] THE CERTIFICATE WILL EXPIRE IN $diff_days DAYS ON $source_desc!"
        # Here we can add Slack/Discord/PagerDuty webhook integrations
        # curl -X POST -H 'Content-type: application/json' --data '{"text":"Warning: Cert expired in '$diff_days' days!"}' https://hooks.slack.com/services/...
        EXIT_STATUS=1
    else
        echo "[OK] Certificate is safe. Validity is more than $ALERT_DAYS days."
    fi
}

# --- ACTION 1: Remote SSL Inspection ---
if [ "$CHECK_REMOTE" = true ]; then
    echo "Inspecting the remote TLS port $REMOTE_HOST:$REMOTE_PORT..."
    
    # Fetch the expiry date using openssl s_client
    cert_info=$(echo | openssl s_client -connect "$REMOTE_HOST:$REMOTE_PORT" -servername "$REMOTE_HOST" 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null)
    
    if [ -n "$cert_info" ]; then
        # Extract the date string after "notAfter="
        raw_date=$(echo "$cert_info" | cut -d= -f2)
        evaluate_expiry "$raw_date" "Remote Port $REMOTE_HOST:$REMOTE_PORT"
    else
        echo "[ERROR] Failed to connect to the remote TLS port $REMOTE_HOST:$REMOTE_PORT or the port doesn't respond to SSL handshakes."
        EXIT_STATUS=1
    fi
fi

# --- ACTION 2: Local JKS Keystore Inspection ---
if [ "$CHECK_LOCAL_KEYSTORE" = true ]; then
    echo "Inspecting the local keystore file $KEYSTORE_PATH..."
    
    if [ ! -f "$KEYSTORE_PATH" ]; then
        echo "[ERROR] Keystore file not found at path: $KEYSTORE_PATH"
        EXIT_STATUS=1
    else
        # Use keytool to read the expiry date of the specific alias
        # Grab lines containing 'Valid from' and filter the 'until' expiry date
        cert_info=$(keytool -list -v -keystore "$KEYSTORE_PATH" -storepass "$KEYSTORE_PASS" -alias "$KEY_ALIAS" 2>/dev/null | grep "Valid from")
        
        if [ -n "$cert_info" ]; then
            # Extract the date after the word "until:"
            raw_date=$(echo "$cert_info" | sed -E 's/.*until: (.*)/\1/')
            # Example keytool format: Mon Aug 17 14:32:01 WIB 2026
            # We clean up local timezone formats (like WIB/WIT/WITA) if they complicate parsing
            clean_date=$(echo "$raw_date" | sed -E 's/ (WIB|WIT|WITA|WET|CET|EST|PST|UTC|GMT) / /')
            evaluate_expiry "$clean_date" "Local Keystore Alias [$KEY_ALIAS]"
        else
            echo "[ERROR] Failed to read alias '$KEY_ALIAS' inside the keystore or the password is wrong."
            EXIT_STATUS=1
        fi
    fi
fi

echo "[$(date '+%Y-%m-%d %H:%M:%S')] Inspection finished with exit code: $EXIT_STATUS"
exit $EXIT_STATUS

Automation Schedule Configuration (Cron Job) #

To run this script automatically every day at 3:00 AM and record results to log files, we can add it to the monitoring server’s cron table (crontab):

1. Run the crontab edit command:

   crontab -e
   

2. Add the following schedule line:

   0 3 * * * /usr/local/bin/check-kafka-certs.sh >> /var/log/kafka/cert-check.log 2>&1
   

Production Readiness Security Checklist #

Before releasing Apache Kafka clusters to production environments, administrator and IT Security teams must do a compliance audit based on the checklist below to make sure no security gaps are missed:

Summary #

• Apply Zero-Trust — Always disable the PLAINTEXT listener and change the built-in authorization setting to allow.everyone.if.no.acl.found=false to adopt the deny-by-default defense principle.
• Use Config Providers — Avoid putting secrets/passwords in Git. Leverage FileConfigProvider to reference strictly isolated external password files on the operating system.
• Monitor SSL Certificates — Install openssl or keytool-based automatic monitoring scripts to detect broker certificates expiring within less than 30 days.
• Use the JMX Exporter — Close JVM remote JMX ports from networks and use Prometheus JMX Exporter agents bound to local hosts (127.0.0.1) so internal metrics can be safely scraped.

← Previous: Top-Level Security Next: Encryption in Transit →