Kafka Metrics: The Ultimate Guide to Monitoring Key Apache Kafka Metrics #

Operating an Apache Kafka cluster at production level without a capable monitoring system is like flying an airplane at night without panel instruments. Kafka is a high-performance distributed system handling millions of messages per second, but its internal complexity involves constant interactions between operating systems, disk I/O, JVM (Java Virtual Machine) memory allocation, network latency, and consensus coordination. When performance degradation or data replication disruptions happen, we must be able to instantly identify root causes before they impact application SLAs.

Apache Kafka monitoring relies on JMX (Java Management Extensions) metrics natively exposed by broker JVMs. Thousands of metrics are available, but not all have the same importance level. We must focus on key metrics (golden signals) reflecting the true health of partition replication, broker thread efficiency, Garbage Collection performance, and data I/O transfer rates.

In this in-depth guide, we’ll dissect JMX monitoring architecture in Kafka, detail the most critical cluster health metrics along with their original MBean names, analyze handler thread efficiency, review JVM/GC metrics with their tuning parameters, and provide production-ready Prometheus JMX Exporter configuration file examples.

Metric Collection Flow Architecture in Apache Kafka #

Before looking at the metric list in detail, let’s study how metrics from inside the Apache Kafka JVM system are extracted, processed, and visualized in modern architectures:

flowchart LR
    subgraph KafkaBrokerNode["KAFKA BROKER NODE"]
        KafkaApp["Apache Kafka (JVM)"] -->|Native JMX MBeans| JMXEngine["JMX Registry"]
        JMXAgent["JMX Exporter (Java Agent)"] -->|Reads Internal JMX| JMXEngine
    end

    Prometheus["Prometheus Server"] -->|HTTP Scraping via Port 7071| JMXAgent
    Prometheus -->|TSDB Storage / Querying| Grafana["Grafana Dashboard"]
    
    style KafkaBrokerNode stroke-width:2px

In this flow, the Prometheus JMX Exporter Java Agent runs inside the same Kafka JVM memory space. This agent reads MBeans locally, filters them according to regex rules, converts them into Prometheus-friendly HTTP time-series data series, and serves them on a local port (e.g., port 7071) to be scraped periodically by the Prometheus Server.

The Most Critical Cluster Health Metrics #

Here are the main metric groups showing whether our cluster is healthy or in critical condition. Alerts must be configured on these metrics.

1. Under-Replicated Partitions (URP) #

• JMX MBean Name: kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
• Description: Shows the number of partitions whose replicas aren’t fully in-sync (not fully in-sync). This means the current number of active synchronized replicas (ISR) is smaller than the Replication Factor (RF) value specified for that topic.
• Action: The normal value must always be 0. If the URP value is persistently > 0, that means a broker is dead, experiencing network problems (network partitions), or very slow disk I/O so followers can’t catch up with leader offsets. This is the most important metric for measuring data loss risk.

2. Offline Partitions Count #

• JMX MBean Name: kafka.server:type=KafkaController,name=OfflinePartitionsCount
• Description: Shows the number of partitions without an active leader. If a partition loses its leader and there are no followers in the ISR that can replace it, that partition becomes offline.
• Action: The normal value must be 0. If offline partitions > 0, producer applications experience delivery rejections (NotLeaderOrFollowerException) and consumers can’t read data from those partitions. This signals partial downtime in our applications.

3. Active Controller Count #

• JMX MBean Name: kafka.server:type=KafkaController,name=ActiveControllerCount
• Description: Shows whether that broker acts as the active controller (main coordinator) of the cluster.
• Action: Inside one Kafka cluster, there must be exactly 1 active controller. If the count is 0, the cluster loses its metadata coordinator and can’t process topic creation or partition failovers. If the count is > 1 (split-brain), immediately inspect our cluster’s internal network.

4. Unclean Leader Elections Per Second #

• JMX MBean Name: kafka.server:type=ControllerStats,name=UncleanLeaderElectionsPerSec
• Description: The frequency of replicas outside the ISR (whose data isn’t fully synchronized) being elected as new leaders after old leaders die when the unclean.leader.election.enable=true parameter is enabled.
• Action: The normal value must be 0. Every time an unclean leader election happens, we’re guaranteed to lose data (data loss) because lagging replicas are forced to become the main data reference for those partitions.

Network Efficiency and Thread Utilization Metrics #

Kafka uses Java NIO-based non-blocking network architecture with two main thread pool groups: Network Threads (receiving TCP socket requests) and Request Handler Threads (processing request logic like disk writes).

flowchart LR
    Client["Clients (SDK)"] ==> NT["Network Threads"]
    NT ==> RQ["Request Queue"]
    RQ ==> HT["Handler Threads"]
    NT --- Metric1["NetworkProcessorIdlePercent"]
    HT --- Metric2["RequestHandlerIdlePercent"]

1. Request Handler Idle Percent #

• JMX MBean Name: kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerIdlePercent
• Description: The percentage of idle time of the request processing thread pool (I/O threads).
• Action: Metric values range from 0.0 (0% idle / 100% busy) to 1.0 (100% idle). The healthy value is above 0.3 (30% idle). If this metric’s average value drops below 0.2 (20% idle), that means the broker is overwhelmed processing disk I/O requests. Our cluster starts experiencing drastic response latency increases (request queuing delay). The solution is adding handler threads (num.io.threads) or doing horizontal scaling (adding brokers).

2. Network Processor Idle Percent #

• JMX MBean Name: kafka.network:type=Processor,name=NetworkProcessorIdlePercent
• Description: The percentage of idle time of network threads (SSL encryption processors, network parsing).
• Action: Similar to handlers, the healthy value is above 0.3. If this value is very low, check whether the CPU is burdened from excessive SSL/TLS handshakes. We can add network threads through the num.network.threads parameter.

JVM and Garbage Collection (GC) Performance Metrics #

Because Apache Kafka is built on the Scala/Java language, its behavior is heavily influenced by JVM memory management. Garbage Collection metric monitoring is mandatory to avoid cluster hang disasters from overly long GC pauses.

1. GC Pause Duration (GC Time) #

• JMX MBean Name: java.lang:type=GarbageCollector,name=* (Metrics: CollectionTime and CollectionCount)
• Description: Tracks the frequency and duration (in milliseconds) of JVM garbage memory collection activities.
• Impact: If broker JVMs use standard memory cleanup algorithms and experience long “Stop-the-World” (STW) pauses (e.g., > 5-10 seconds), all broker activities are temporarily frozen. During this freeze, brokers don’t respond to KRaft/Zookeeper watchdog heartbeats. Other brokers consider the frozen broker dead, then trigger partition failovers. When the broker wakes from the GC pause, it’s shocked that the partitions it led were seized, triggering rebalance storms and cluster instability.

2. Optimal JVM Garbage Collector Configuration #

To minimize GC pauses in production, we’re highly recommended to use the Garbage-First Garbage Collector (G1GC) with the following tuning parameters on broker JVMs:

# Production JVM Garbage Collector settings for Apache Kafka
export KAFKA_GC_LOG_OPTS="-Xlog:gc*,gc+age=trace,gc+phases=debug:file=/var/log/kafka/kafka-gc.log:time,uptime,pid:filecount=10,filesize=100M"

export KAFKA_JVM_PERFORMANCE_OPTS="-server \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=20 \
  -XX:InitiatingHeapOccupancyPercent=35 \
  -XX:G1ReservePercent=15 \
  -XX:MinMetaspaceFreeRatio=50 \
  -XX:MaxMetaspaceFreeRatio=80"

Parameter Explanations:

• -XX:+UseG1GC: Enables the G1 Garbage Collector algorithm.
• -XX:MaxGCPauseMillis=20: The maximum Garbage Collector pause time target of 20 milliseconds (very aggressive for keeping latency low).
• -XX:InitiatingHeapOccupancyPercent=35: Triggers GC cycles earlier when heap data accumulation reaches 35% (preventing full GC memory pileups).
• -XX:G1ReservePercent=15: Provides 15% free memory reserves to avoid memory allocation failures when transferring data from the OS page cache to the JVM.

Production Configuration: Prometheus JMX Exporter #

To extract the metrics above into Prometheus servers, we must install the Prometheus JMX Exporter as a Java Agent on Kafka startup processes.

Here are the contents of the /etc/kafka/jmx_exporter.yaml configuration file optimized to filter thousands of raw Kafka metrics into concise, efficient Prometheus formats:

# ==============================================================================
# jmx_exporter.yaml - PRODUCTION JMX EXPORTER CONFIGURATION FOR KAFKA
# ==============================================================================

lowercaseOutputName: true
lowercaseOutputLabelNames: true

rules:
  # 1. Main Health Metrics: UnderReplicatedPartitions & OfflinePartitionsCount
  - pattern: 'kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value'
    name: kafka_server_replicamanager_underreplicatedpartitions
    type: GAUGE
  - pattern: 'kafka.server<type=KafkaController, name=OfflinePartitionsCount><>Value'
    name: kafka_server_kafkacontroller_offlinepartitionscount
    type: GAUGE
  - pattern: 'kafka.server<type=KafkaController, name=ActiveControllerCount><>Value'
    name: kafka_server_kafkacontroller_activecontrollercount
    type: GAUGE

  # 2. Data Throughput Metrics (Byte Rates & Message Rates)
  - pattern: 'kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec|MessagesInPerSec), topic=(.+)><>Count'
    name: kafka_server_brokertopicmetrics_$1_total
    labels:
      topic: "$2"
    type: COUNTER
  - pattern: 'kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec|MessagesInPerSec)><>Count'
    name: kafka_server_brokertopicmetrics_$1_global_total
    type: COUNTER

  # 3. Thread Pool Utilization Metrics (RequestHandler & NetworkProcessor)
  - pattern: 'kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerIdlePercent><>MeanRate'
    name: kafka_server_kafkarequesthandlerpool_requesthandleridlepercent_meanrate
    type: GAUGE
  - pattern: 'kafka.network<type=Processor, name=NetworkProcessorIdlePercent, networkProcessor=(.+)><>Value'
    name: kafka_network_processor_networkprocessoridlepercent
    labels:
      processor: "$1"
    type: GAUGE

  # 4. Request Wait Time Metrics (Request Latencies)
  - pattern: 'kafka.server<type=RequestMetrics, name=RequestsPerSec, request=(.+)><>Count'
    name: kafka_server_requestmetrics_requests_total
    labels:
      request: "$1"
    type: COUNTER
  - pattern: 'kafka.server<type=RequestMetrics, name=TotalTimeMs, request=(.+)><>99thPercentile'
    name: kafka_server_requestmetrics_totaltimems_p99
    labels:
      request: "$1"
    type: GAUGE

  # 5. Group Coordinator Metrics (Consumer Group Rebalances)
  - pattern: 'kafka.coordinator.group<type=GroupMetadataManager, name=MinActiveMembers><>Value'
    name: kafka_coordinator_group_groupmetadatamanager_minactivemembers
    type: GAUGE

  # 6. JVM Garbage Collection & Memory Metrics
  - pattern: 'java.lang<type=GarbageCollector, name=(.+)><>(CollectionCount|CollectionTime)'
    name: jvm_gc_$2_total
    labels:
      gc_name: "$1"
    type: COUNTER

How to Install the JMX Exporter Java Agent at Kafka Startup: #

To enable this exporter, add the KAFKA_OPTS environment variable before running the Kafka broker. Configure the agent to listen on port 7071 reading the configuration file above:

# Running the Kafka Broker with the JMX Exporter Java Agent
export KAFKA_OPTS="-javaagent:/usr/share/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/etc/kafka/jmx_exporter.yaml"
bin/kafka-server-start.sh -daemon config/server.properties

After the broker runs, we can test metric extraction by curling port 7071:

curl http://localhost:7071/metrics

The curl output must display raw metrics in Prometheus time-series format (e.g., kafka_server_replicamanager_underreplicatedpartitions 0.0).

Metric Monitoring Configuration Audit Checklist #

Make sure our cluster monitoring system defenses meet the following compliance checklist before being declared production ready:

Summary #

• Focus on Golden Signals — Don’t monitor all metrics raw. Focus on the most important health metrics: URP (Under-Replicated Partitions), Offline Partitions Count, and Request Handler Idle Percent.
• Mandate URP Alerting — URP values above 0 are the main replication damage indicator. Integrate URP notifications directly to on-call teams for downtime prevention.
• JVM G1GC Tuning — Long GC pauses are the main cause of brokers mysteriously disappearing from cluster detection. Always optimize JVM G1GC configurations and monitor their memory cleanup times.
• Filter JMX Metrics — Use efficient regex filters in JMX Exporter files to limit the data size scraped by Prometheus, keeping our monitoring server performance light.