Broker Log: Configuring and Analyzing Apache Kafka Server Logs #

When operating Apache Kafka at production scale, we’re often faced with mysterious situations where cluster performance suddenly drops or one broker disconnects from cluster coordination. To diagnose problems (troubleshooting), the first instrument we must inspect is the broker operational server log (Broker Log). Unlike data logs (log segments) holding original message payloads from producers, server logs contain diagnostic messages from internal Kafka engines (like replication status, consumer lifecycles, ACL authorization, and watchdog metadata).

Many beginner DevOps teams confuse these two log terms or leave default server log configurations running in production. As a result, their server disks quickly fill from giant non-rotating server log files, or conversely, they lose valuable information because logging levels are set too high (WARN or ERROR only).

In this guide, we’ll dissect data log vs server log differences, arrange optimal production log4j.properties configurations, learn dynamic logging level changing techniques without broker restarts (Dynamic Log Level Tuning), and decode server log messages to solve common production problems.

Distinguishing Data Logs vs Server Logs #

Before going further, it’s very important to separate the following two log concepts because failures to understand these differences often lead to disk configuration errors:

+---------------------------------------------------------------------------------+
|                         MAIN DIFFERENCES OF TWO LOG TYPES                       |
|                                                                                 |
|  1. DATA LOG (Data Log Segments)                                                |
|     * Property Path: log.dirs=/var/lib/kafka/data                              |
|     * Contents: Original binary message payloads from producers (.log, .index, .timeindex) |
|     * Settings: log.retention.hours, log.retention.bytes                        |
|                                                                                 |
|  2. SERVER LOG (Server Operation Log)                                           |
|     * Configuration: /etc/kafka/log4j.properties                               |
|     * Contents: Java/Scala JVM diagnostic log texts (ReplicaManager, Controller, ZK) |
|     * Settings: FileSize, MaxBackupIndex, DailyRollingFileAppender              |
+---------------------------------------------------------------------------------+

Server logs are written by the Log4j framework wrapped by the Java Virtual Machine (JVM). These logs will be our discussion focus for monitoring broker operational health.

Internal Kafka Broker Logger Flow Architecture #

Kafka divides its logging activities into several special logger categories so we can separate output files by function.

Here’s a diagram of the internal Kafka logger flow to physical log files:

flowchart TD
    subgraph Subsystems["Kafka Broker Subsystems"]
        ReplicaMgr["Replica Manager (ISR)"]
        Controller["Controller Manager (KRaft/ZK)"]
        GroupCoord["Group Coordinator (Rebalance)"]
        RequestLog["Network Request Processor"]
    end

    subgraph Log4jLoggers["Log4j Logger Categories"]
        StateLogger["state.change.logger"]
        ContrLogger["kafka.controller"]
        CoordLogger["kafka.coordinator"]
        ReqLogger["kafka.request.logger"]
    end

    subgraph PhysicalFiles["Physical Log Files (Disk)"]
        StateFile["/var/log/kafka/state-change.log"]
        ServerFile["/var/log/kafka/server.log"]
        RequestFile["/var/log/kafka/kafka-request.log"]
    end

    ReplicaMgr --> StateLogger
    Controller --> ContrLogger
    GroupCoord --> CoordLogger
    RequestLog --> ReqLogger

    StateLogger --> StateFile
    ContrLogger --> ServerFile
    CoordLogger --> ServerFile
    ReqLogger --> RequestFile

This category separation helps us when wanting to raise log verbosity levels for one specific module without flooding other modules with unnecessary messages.

Production Configuration: Safe log4j.properties #

By default, Kafka writes all logs into one single file rotating daily without capacity limits. In production, we must limit maximum file sizes using RollingFileAppender with MaxFileSize and MaxBackupIndex settings to prevent full disks from log leaks.

Here’s an optimized production /etc/kafka/log4j.properties configuration:

# ==============================================================================
# log4j.properties - KAFKA SERVER LOG PRODUCTION CONFIGURATION
# ==============================================================================

# 1. Define the Root Logger Level and Main Appender
log4j.rootLogger=INFO, kafkaAppender

# 2. Main Appender Configuration (server.log)
log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.File=/var/log/kafka/server.log
# Limit the maximum size per file to 100 MB
log4j.appender.kafkaAppender.MaxFileSize=100MB
# Keep a maximum of 10 backup files (max total capacity 1 GB)
log4j.appender.kafkaAppender.MaxBackupIndex=10
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

# 3. Special State Transition Appender Configuration (state-change.log)
log4j.logger.state.change.logger=INFO, stateChangeAppender
log4j.additivity.state.change.logger=false

log4j.appender.stateChangeAppender=org.apache.log4j.RollingFileAppender
log4j.appender.stateChangeAppender.File=/var/log/kafka/state-change.log
log4j.appender.stateChangeAppender.MaxFileSize=100MB
log4j.appender.stateChangeAppender.MaxBackupIndex=5
log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

# 4. Special Controller Appender Configuration (controller.log)
log4j.logger.kafka.controller=INFO, controllerAppender
log4j.additivity.kafka.controller=false

log4j.appender.controllerAppender=org.apache.log4j.RollingFileAppender
log4j.appender.controllerAppender.File=/var/log/kafka/controller.log
log4j.appender.controllerAppender.MaxFileSize=50MB
log4j.appender.controllerAppender.MaxBackupIndex=5
log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

# 5. Limit Third-Party Library Verbosity (Zookeeper / Netty / Reflections)
log4j.logger.org.apache.zookeeper=WARN
log4j.logger.org.I0Itec.zkclient=WARN
log4j.logger.org.reflections=WARN

# 6. Special Security Logger (ACL & Authorization)
log4j.logger.kafka.authorizer.logger=INFO, authorizerAppender
log4j.additivity.kafka.authorizer.logger=false

log4j.appender.authorizerAppender=org.apache.log4j.RollingFileAppender
log4j.appender.authorizerAppender.File=/var/log/kafka/kafka-authorizer.log
log4j.appender.authorizerAppender.MaxFileSize=50MB
log4j.appender.authorizerAppender.MaxBackupIndex=5
log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

System Logrotate Integration #

To make sure rotated log files are safely compressed to save disk space, we can create the /etc/logrotate.d/kafka configuration like this:

/var/log/kafka/*.log {
    daily
    rotate 7
    copytruncate
    compress
    delaycompress
    missingok
    notifempty
}

The copytruncate method is very important because Kafka holds log file descriptors continuously. By using copytruncate, the operating system copies active log contents to new files and empties the current active file without breaking the JVM process links currently writing logs.

Dynamic Log Level Tuning via CLI and Code #

When clusters experience authorization failures or very complex replication problems, INFO logging levels often lack error details. However, we must not restart brokers just to change log levels to DEBUG because restarts trigger partition failovers, leadership migrations, and worsen overall cluster performance degradation.

Apache Kafka provides dynamic APIs for directly changing specific logger logging levels (on-the-fly).

Method 1: Using the kafka-configs.sh CLI #

We can dynamically modify logging levels using Kafka’s built-in CLI. These changes are communicated through the Kafka AdminClient protocol directly to target brokers and applied instantly without JVM restarts.

1. Checking Current Logger Levels: To see the active logger list along with levels on broker ID 1:

   kafka-configs.sh --bootstrap-server localhost:9093 \
     --command-config /etc/kafka/client.properties \
     --describe \
     --entity-type broker-loggers \
     --entity-name 1
   

2. Dynamically Changing Logger Levels: If we want to monitor ACL authorization request details to track client authentication failures, change the kafka.authorizer.logger level to DEBUG:

   kafka-configs.sh --bootstrap-server localhost:9093 \
     --command-config /etc/kafka/client.properties \
     --alter \
     --entity-type broker-loggers \
     --entity-name 1 \
     --add-config kafka.authorizer.logger=DEBUG
   

3. Restoring to Original Levels: After troubleshooting processes finish, we must immediately restore to INFO levels so broker disk I/O performance isn’t burdened:

   kafka-configs.sh --bootstrap-server localhost:9093 \
     --command-config /etc/kafka/client.properties \
     --alter \
     --entity-type broker-loggers \
     --entity-name 1 \
     --add-config kafka.authorizer.logger=INFO
   

Method 2: Programmatically Changing Logger Levels (Java AdminClient) #

For those of us acting as SRE (Site Reliability Engineering) teams wanting to build automatic log tuning tools, we can trigger logger level changes using the Java AdminClient API programmatically:

package com.mycompany.kafka.admin;

import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AlterConfigsResult;
import org.apache.kafka.common.config.ConfigResource;
import org.apache.kafka.clients.admin.ConfigEntry;
import org.apache.kafka.clients.admin.AlterConfigOp;

import java.util.*;

public class DynamicLoggerTuner {

    private final AdminClient adminClient;

    public DynamicLoggerTuner(Properties properties) {
        this.adminClient = AdminClient.create(properties);
    }

    /**
     * Dynamically changes broker logger levels without restarts.
     *
     * @param brokerId   The ID of the targeted Kafka broker
     * @param loggerName The logger name (e.g., "kafka.authorizer.logger")
     * @param level      The new log level (e.g., "DEBUG", "INFO", "WARN")
     */
    public void setBrokerLoggerLevel(int brokerId, String loggerName, String level) throws Exception {
        // 1. Define the resource type as BROKER_LOGGER
        ConfigResource resource = new ConfigResource(
                ConfigResource.Type.BROKER_LOGGER, 
                String.valueOf(brokerId)
        );

        // 2. Create the new logger level configuration entry
        ConfigEntry loggerConfig = new ConfigEntry(loggerName, level);
        
        // 3. Set the change operation (SET)
        AlterConfigOp op = new AlterConfigOp(loggerConfig, AlterConfigOp.OpType.SET);
        
        Map<ConfigResource, Collection<AlterConfigOp>> configs = new HashMap<>();
        configs.put(resource, Collections.singletonList(op));

        // 4. Send the change request to the Broker asynchronously
        AlterConfigsResult result = adminClient.incrementalAlterConfigs(configs);
        
        // Wait until the handshake confirmation succeeds with the broker
        result.all().get(); 
        
        System.out.println("Successfully changed logger [" + loggerName + "] on broker [" + brokerId + "] to " + level);
    }

    public void close() {
        if (adminClient != null) {
            adminClient.close();
        }
    }
}

This programmatic approach is very safe if paired with automatic alert systems. For example, when authorization failure rates exceed certain limits, systems can automatically raise authorizer log levels to DEBUG for 15 minutes, then lower them back after collecting enough log samples.

Analyzing Server Logs for Production Problem Troubleshooting #

When clusters experience operational failures, we must quickly identify root causes by matching error message patterns appearing in log files. Here’s an in-depth discussion of the five most frequently occurring production failure scenarios, complete with original log structure analyses.

1. Replication Problems: Shrinking ISR #

This scenario happens when inter-broker replication experiences physical obstacles, forcing partition leaders to remove slow replicas from ISR groups.

Example Log Message:

[2026-06-08 14:22:15,102] INFO [ReplicaManager broker=1] Shrinking ISR for partition payment.orders-0 to 1,2 (kafka.server.ReplicaManager)

Problem Analysis:

1. Initial Identification: The payment.orders-0 partition loses one of its active replicas. The previous ISR list (e.g., containing brokers 1, 2, and 3) shrinks to only being inhabited by brokers 1 and 2. Broker ID 3 has been declared out of the ISR.
2. Root Causes: Broker 3 likely experiences one of the following problems:
   • Very long Garbage Collection pauses (stop-the-world pauses) so broker 3 fails to send heartbeat signals to controllers within the replica.lag.time.max.ms period.
   • Disk I/O congestion on broker 3’s data log partitions, causing ReplicaFetcherThread replication threads to lag far behind leader Log End Offsets (LEO).
   • Local network problems (network packet loss) between broker 1 and broker 3.
3. Mitigation Steps:
   • Check garbage collection logs on broker 3. If GC pause durations exceed 5-10 seconds, do G1GC optimization.
   • Check disk I/O latency using iostat utilities to monitor whether there are high disk write queues.

2. Consumer Problems: Rebalance Loops and Session Timeouts #

This scenario is often complained about by developers when their consumer applications seem to stop processing data and continuously do rebalance processes.

Example Log Message:

[2026-06-08 14:25:30,415] INFO [GroupCoordinator 1]: Preparing rebalance for group payment-processor in state PreparingRebalance with member client-1-8c4d21e8 (reason: Peer re-joined group) (kafka.coordinator.group.GroupCoordinator)
[2026-06-08 14:25:45,910] INFO [GroupCoordinator 1]: Member client-1-8c4d21e8 in group payment-processor has failed, removing from group (reason: join-group timeout) (kafka.coordinator.group.GroupCoordinator)

Problem Analysis:

1. Initial Trigger: The Group Coordinator on Broker 1 detects that one consumer group member (client-1-8c4d21e8) exited or responded late, triggering transitions to PreparingRebalance phases.
2. Root Causes:
   • The max.poll.interval.ms property on client sides is set too low compared to one batch’s data processing time. If consumers need 60 seconds to process data from one poll() call, but max.poll.interval.ms is set to only 30 seconds, consumers are considered dead by coordinators.
   • Consumers experience unexpected errors (out of memory) or stall from long local GC pauses.
3. Mitigation Steps:
   • Raise max.poll.interval.ms values in client configurations to give longer processing time tolerances.
   • Alternatively, lower max.poll.records values to limit the data amount fetched in one poll so processing finishes faster.
   • Move heavy processing logic into separate thread pools (worker threads) and use the main consumer thread only for polling and committing offsets.

3. Log Compaction Problems: Starved Cleaners or Thread Crashes #

This scenario happens when dirty log cleanup (log compaction) processes on topics with compact retention policies experience failures.

Example Log Message:

[2026-06-08 14:28:10,005] WARN [LogCleaner 1]: Cleaner 1 is starved on log directory /var/lib/kafka/data/tenant-a.logs-0 (kafka.log.LogCleaner)
[2026-06-08 14:28:15,310] ERROR [LogCleaner 1]: Error due to local offset map overflow (kafka.log.LogCleaner)

Problem Analysis:

1. Initial Trigger: The LogCleaner thread warns that log cleanup on the tenant-a.logs-0 directory is starved and can’t process data compaction.
2. Root Causes:
   • The unique key count (unique keys) on those partitions exceeds the memory capacity of the recording offset map (SkimpyOffsetMap). The default deduplication memory allocation (log.cleaner.dedupe.buffer.size valued at 128 MB) isn’t enough to hold all message key hash representations.
   • When buffers fill before all log segments are mapped, cleanup processes stall or LogCleaner threads crash with ERROR messages.
3. Mitigation Steps:
   • Raise cleaner deduplication memory capacities in server.properties files:

     log.cleaner.dedupe.buffer.size=536870912 # Raise to 512 MB
     

   • Add log cleaner thread counts by setting log.cleaner.threads=2 or more to speed up parallel compaction processes.

4. KRaft Metadata Problems: Recovery Latency and Socket Depletion #

For modern clusters using KRaft mode (without ZooKeeper), tracking metadata coordination heavily depends on KRaft controller logs.

Example Log Message:

[2026-06-08 14:31:02,110] INFO [MetadataLoader] Metadata loader has processed up to offset 1045230 in 150 ms (org.apache.kafka.image.loader.MetadataLoader)
[2026-06-08 14:31:05,420] WARN [RaftManager] Raft leader 1 failed to heartbeat to voter 2 within the timeout of 3000 ms (org.apache.kafka.raft.KafkaRaftClient)

Problem Analysis:

1. Initial Trigger: KRaft controllers detect metadata processing delays or heartbeat failures between KRaft quorum voters.
2. Root Causes:
   • Metadata change flows are too massive (e.g., from dynamically creating thousands of topic partitions in short periods), so metadata loaders experience processing queues (recovery latency).
   • Controller nodes lack thread allocations or experience socket connection exhaustion (file descriptor depletion) from bloated external client connection counts.
3. Mitigation Steps:
   • Physically separate KRaft controller nodes from data broker nodes. Don’t combine broker and controller roles on the same server for large-scale clusters.
   • Make sure the OS ulimit -n parameter is set to a minimum of 100,000 to prevent Too many open files errors on internal KRaft quorum sockets.

5. Security Problems: ACL Authorization Failures #

We must strictly monitor security logs to detect unauthorized access attempts by illegal clients or application credential configuration errors.

Example Log Message:

[2026-06-08 14:35:10,218] INFO Principal = User:CN=payment-app,O=MyCorp is Denied Operation = Write for Resource = Topic:payment.orders (kafka.authorizer.logger)

Problem Analysis:

1. Initial Trigger: Users with the SSL certificate User:CN=payment-app,O=MyCorp are Denied when trying to do Write operations to the payment.orders topic.
2. Root Causes:
   • Clients use correct usernames, but Kafka ACL (Access Control List) policies haven’t been updated to allow write operations on those topics.
   • There are topic name spelling errors on client application code sides (e.g., wrongly writing the topic name as payment.orders when it should be payment-orders-v2).
3. Mitigation Steps:
   • Run the following CLI command to verify target topic ACL lists:

     kafka-acls.sh --bootstrap-server localhost:9093 \
       --command-config /etc/kafka/client.properties \
       --list \
       --topic payment.orders
     

   • Give valid write access permissions if client identities prove valid:

     kafka-acls.sh --bootstrap-server localhost:9093 \
       --command-config /etc/kafka/client.properties \
       --add \
       --allow-principal User:CN=payment-app,O=MyCorp \
       --operation Write \
       --topic payment.orders
     

Operational Compliance and Server Log Audit Checklist #

To guarantee monitoring system reliability, do routine Kafka server log audits using the following production compliance checklist:

Summary #

• Separate Log Concepts — Always separate understandings between data segment logs (holding compressed message payloads from clients) and Log4j server logs (holding JVM diagnostic messages).
• Limit File Sizes — Don’t let server logs grow without limits on broker disks. Use RollingFileAppender with maximum capacity limits to secure remaining broker disk space.
• Dynamic Log Tuning — Leverage kafka-configs.sh commands or the Java AdminClient to raise log levels to DEBUG when troubleshooting without needing broker JVM restarts.
• Decode Accurately — Learn key log messages like Shrinking ISR, Preparing rebalance, and Cleaner starved to speed up root cause detection during production incidents.

← Previous: Broker Health Next: Client Log →