Top-Level Security: Securing the Kafka Cluster Network and Infrastructure #

When securing distributed data systems like Apache Kafka, software-level protections (like client authentication and ACLs) are useless if the gates at the network infrastructure level are left wide open. Physical infrastructure, network port settings, and data traffic division are important pillars in Kafka cluster Top-Level Security.

Without network-level traffic restrictions, attackers can directly scan internal inter-broker communication ports, try to intercept controller metadata coordination, or launch Denial of Service (DoS) attacks flooding broker ports until total failure.

This article will deeply dissect Kafka cluster network security architecture. We’ll review the crucial differences between the listeners and advertised.listeners parameters, explain how to secure internal coordination ports (inter-broker and KRaft controller quorum), configure connection rate limiting to deflect DoS attacks, and design network isolation topologies using firewalls.

Configuration Dissection: Listeners vs Advertised Listeners #

One of the biggest confusion sources for new Kafka administrators is client connection failures with TimeoutException or Connection reset error messages, even though authentication feels correctly configured. This problem almost always originates from misconfigured listeners and advertised.listeners parameters in the server.properties file.

flowchart TD
    Client["External Client (Internet)"] -- "Access via FQDN: kafka.badri.com:9093" --> Broker["KAFKA BROKER SERVER<br/>- LISTENERS: binds to the internal interface IP (e.g., 10.0.1.5:9093)<br/>- ADVERTISED LISTENERS: returns the public domain name (e.g., kafka.badri.com:9093)"]

1. Listeners (Internal Bind Addresses) #

The listeners parameter defines the list of physical interface IP addresses and ports on the server where the Kafka broker JVM process will bind its network sockets.

  • Function: Determines which ports the broker listens on locally.
  • Format: protocol://interface_ip:port (Example: listeners=PLAINTEXT://10.0.1.5:9092,SSL://10.0.1.5:9093).

2. Advertised Listeners (Public Address Metadata) #

The advertised.listeners parameter defines the host and port addresses the broker writes into cluster metadata (and reports to KRaft or ZooKeeper).

  • Function: When clients first contact Kafka (through one of the initial bootstrap server brokers), the broker responds by returning the advertised listeners list of all brokers in the cluster. Clients then use this address list to make actual connections to partition leaders.
  • Format: protocol://dns_public_name:port (Example: advertised.listeners=PLAINTEXT://kafka-node-1.internal:9092,SSL://kafka.badri.com:9093).

Why Is This Separation Very Important for Security? #

If we run Kafka in Cloud environments (like AWS), brokers usually have private IPs (e.g., 10.0.1.5) and public IPs (or connect to a Load Balancer with the kafka.badri.com domain).

  • If we fill advertised.listeners with a private IP (10.0.1.5), external clients from the internet successfully contact the bootstrap port, but when receiving metadata, clients try contacting the private IP 10.0.1.5 which can’t be routed from the internet. Connections fail.
  • Conversely, if we bind listeners directly to the public IP 0.0.0.0 (listening on all interfaces) without port protection, internal inter-broker coordination ports are exposed to the wide internet, inviting cyber attacks.

Securing Inter-Broker Communication and the KRaft Quorum #

Inside multi-node clusters, brokers must continuously communicate with each other to replicate partition data, elect new leaders (leader election), and synchronize cluster status. Additionally, in modern KRaft (Kafka Raft)-based Kafka, brokers must actively coordinate with controller nodes (controller quorum) to manage metadata.

This internal coordination traffic is very sensitive and must not be mixed with the traffic ports used by general clients.

1. Inter-Broker Port Isolation #

We must dedicate one special listener that’s encrypted and authenticated exclusively for inter-broker communication.

# Defining the internal port 9092 for broker coordination, port 9093 for clients
listeners=INTERNAL://:9092,CLIENT_SSL://:9093
advertised.listeners=INTERNAL://kafka-broker-1.internal:9092,CLIENT_SSL://kafka.badri.com:9093

# Determining the security protocol mapping for each listener name
listener.security.protocol.map=INTERNAL:SSL,CLIENT_SSL:SASL_SSL

# Forcing inter-broker communication to use the SSL-encrypted internal listener
security.inter.broker.protocol=INTERNAL

2. Securing the KRaft Controller Quorum #

In KRaft architectures, controller nodes manage all cluster status. Securing controller ports is absolute because if these ports are compromised, attackers can take over full cluster leadership.

Here’s the security configuration for locking the KRaft Quorum port (9094) using the SSL protocol in the server.properties broker/controller:

# Registering the controller port in the listener map
listener.security.protocol.map=INTERNAL:SSL,CLIENT_SSL:SASL_SSL,CONTROLLER:SSL

# Controller quorum communication configuration
process.roles=broker,controller
controller.listener.names=CONTROLLER
controller.quorum.voters=[email protected]:9094

# Enabling mutual certificate verification (mTLS) on the controller port
listener.name.controller.ssl.client.auth=required

With the settings above, only entities with official SSL certificates signed by our internal root CA are allowed to send coordination commands to the controller quorum.

Protection Against DoS Attacks: Connection Rate Limiting & Quotas #

Production Kafka brokers are very vulnerable to failures from memory or port exhaustion if attacked with millions of continuously new connections. These attacks can be intentional Denial of Service (DoS) attacks, or from connection leaks in poorly written client applications (e.g., creating new producer instances on every incoming HTTP request).

To protect brokers from the scenarios above, we can configure network-level limits:

1. Limiting New Connection Creation Rates #

Kafka provides parameters for limiting the number of new TCP connections allowed to be created per second on each broker. If this limit is exceeded, the broker rejects new connections to protect itself from crashing.

# Limiting the broker to only accept a maximum of 100 new connections per second globally
max.connection.creation.rate=100

# Or limiting connection rates specifically per listener (e.g., the CLIENT_SSL listener)
# to keep the INTERNAL inter-broker port connecting smoothly
listener.name.client_ssl.max.connection.creation.rate=50

2. Limiting Total Active Connections per IP Address #

We can also limit the total number of active TCP connections from one specific client IP address so one broken client server doesn’t consume the entire broker connection quota.

# Limiting one IP address to a maximum of 50 simultaneous active connections
max.connections.per.ip=50

3. Applying Bandwidth Quotas for Clients #

Besides limiting physical connection counts, we must also prevent one client from absorbing the entire cluster network bandwidth (e.g., analytics teams doing unlimited massive data pulls). We can set byte-per-second quota limits for specific principals dynamically:

# Limiting the 'payment-app' producer bandwidth to a maximum of 10 MB/second (10485760 bytes/sec)
# and the 'payment-app' consumer bandwidth to a maximum of 20 MB/second (20971520 bytes/sec)
kafka-configs.sh --bootstrap-server localhost:9093 \
  --command-config client-ssl.properties \
  --entity-type users --entity-name payment-app \
  --alter --add-config 'producer_byte_rate=10485760,consumer_byte_rate=20971520'

Network Isolation Using Firewall Topologies (Security Groups) #

The best network protection is physically closing ports using network security systems like firewalls, AWS Security Groups, or VPC routing rules.

Here’s a secure Kafka cluster network port architecture design:

flowchart TD
    subgraph PublicNetwork["1. Public Network / Client Applications"]
        AppClients["Producer & Consumer Applications"]
    end

    subgraph VPC["2. Private Virtual Private Cloud (VPC)"]
        subgraph DMZ["DMZ Subnet / Load Balancer"]
            LB["Network Load Balancer (Port 9093)"]
        end
        
        subgraph PrivateSubnet["Isolated Internal Subnet"]
            Broker1["Kafka Broker 1"]
            Broker2["Kafka Broker 2"]
            Controller1["KRaft Quorum Controller"]
        end
    end

    AppClients -->|May only access Port 9093 via TLS| LB
    LB -->|Route to Brokers| Broker1 & Broker2
    
    Broker1 <-->|Port 9092: Inter-Broker SSL| Broker2
    Broker1 & Broker2 <-->|Port 9094: Controller Quorum SSL| Controller1
    
    style DMZ stroke:#0288d1,stroke-width:2px
    style PrivateSubnet stroke:#2e7d32,stroke-width:2px

Production Firewall Configuration Golden Rules: #

  1. External Client Port (9093): Only opened to client application server IPs or through a Network Load Balancer (NLB). Never expose this port to the wide internet (0.0.0.0/0) without mTLS/SASL protection.
  2. Inter-Broker Port (9092): Only allowed for communication between fellow broker internal IP addresses in the cluster. Block access to this port from outside the Kafka subnet.
  3. Controller Quorum Port (9094): Only allowed for communication between brokers and KRaft controller nodes.
  4. JMX Monitoring Port (9999): The JMX port used for metric collection (by Prometheus/Grafana) sends raw JVM data without standard authentication. This port must be tightly locked and only allowed to be accessed by your monitoring server IPs.

Securing the JMX Monitoring Port & DNS Hostname Resolution #

Two infrastructure security gaps often missed in audits are securing the Java Management Extensions (JMX) port and SSL handshake dependency on domain name (DNS) resolution.

1. Securing the JMX Monitoring Port #

JMX is a crucial built-in Java technology for monitoring internal Kafka broker metrics (like throughput, consumer lag, Under-Replicated Partitions, and Garbage Collection). By default, administrators often enable JMX without authentication using the following JVM properties so Prometheus can easily fetch data:

# VERY DANGEROUS IN PRODUCTION ENVIRONMENTS
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.port=9999"

JMX Security Risks Without Authentication #

JMX runs on the RMI (Remote Method Invocation) protocol. If the JMX port is open without protection, attackers with local network access can not only read internal metrics, but also execute Remote Code Execution (RCE) through the JVM, force Garbage Collection cycles to cripple broker performance, or forcibly kill the Kafka JVM process.

Solution 1: Enabling JMX Authentication #

We must enable authentication by creating a credential configuration file:

  1. Create the /etc/kafka/jmxremote.password file (fill with username and password):
    monitor-user  secureJmxPass123
admin-user    superJmxAdminPass
  2. Create the /etc/kafka/jmxremote.access file (fill with access rights):
    monitor-user  readonly
admin-user    readwrite
  3. Set Linux file permissions so only the Kafka process user can read them (OS security):
    chmod 400 /etc/kafka/jmxremote.password
chown kafka:kafka /etc/kafka/jmxremote.password
  4. Update the Kafka JMX startup options:
    export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.password.file=/etc/kafka/jmxremote.password \
  -Dcom.sun.management.jmxremote.access.file=/etc/kafka/jmxremote.access \
  -Dcom.sun.management.jmxremote.ssl=false"

Solution 2: Using the Prometheus JMX Exporter (Modern Recommendation) #

The best approach in modern cloud-native environments is to not open the remote JMX port at all. We can install the Prometheus JMX Exporter Java Agent (jmx_prometheus_javaagent.jar) directly inside the Kafka JVM at startup.

This agent reads JMX metrics internally in JVM memory and exposes them as a standard Prometheus plain-text HTTP endpoint on a specific port (e.g., port 7071):

# Running the Prometheus exporter internally inside the Kafka JVM
export KAFKA_OPTS="-javaagent:/opt/prometheus/jmx_prometheus_javaagent.jar=7071:/etc/prometheus/kafka-config.yml"

This way, the JMX RMI port (9999) stays off, and we just secure the HTTP 7071 port using firewall rules so it can only be accessed by Prometheus scraper server IPs.

2. SSL Dependency on DNS Hostname Resolution #

Many infrastructure teams try configuring advertised.listeners using IP addresses (e.g., 192.168.1.10) to avoid needing internal DNS server management. However, if we use SSL/TLS encryption, using IP addresses is a recipe for connection failures.

Why? #

The SSL/TLS protocol verifies whether the hostname clients contact matches the Subject or Subject Alternative Name (SAN) fields written in the digital certificate presented by brokers.

  • If clients contact 192.168.1.10, while the broker certificate was created for CN=kafka-broker-1.internal, the client security library (e.g., the Java SSL engine) cancels the handshake because of host identity mismatch (endpoint identification failure).
  • Creating IP address-based SSL certificates (using SAN IP fields) is very inflexible in dynamic environments like cloud or Kubernetes because broker IP addresses can change anytime nodes are scaled or moved.

Solution #

We must use FQDN (Fully Qualified Domain Name) domain addresses consistently in the advertised.listeners configuration (e.g., kafka-broker-1.internal) and make sure internal DNS servers (or CoreDNS in Kubernetes) resolve those names to the correct physical broker IP addresses from both client and broker perspectives.

Summary #

  • Listeners vs Advertised — Listeners manage locally bound ports, while Advertised Listeners manage the DNS addresses presented to clients for connections.
  • Port Isolation — Always separate client traffic ports (CLIENT_SSL / 9093) from inter-broker coordination communication ports (INTERNAL / 9092).
  • KRaft Security — The KRaft quorum port (9094) is the heart of cluster metadata. Secure this port using strict mutual mTLS.
  • DoS Protection — Enable new connection creation rate limits per second (max.connection.creation.rate) to protect brokers from connection floods from client leaks.
  • VPC Firewalls — Apply strict firewall rules to close all internal ports (JMX, inter-broker, KRaft) from external network access.

← Previous: Access Control Lists (ACL) Next: Common Security Mistakes →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact