Client Log: Logging Configuration and Troubleshooting on the Client Side #

When we build event-driven systems using Apache Kafka, system stability isn’t only determined by broker cluster health, but also by the reliability of producer and consumer applications on client sides. When message loss, data duplication, or connection failures happen, checking broker logs alone is often not enough. We need full visibility into what happens inside the Kafka Client Library running as part of our applications.

Unfortunately, many developer teams ignore client-side logging. They let built-in logging configurations mix internal Kafka debug messages with application business logs, or even completely disable client logs because they’re considered too verbose. As a result, when critical incidents like offset commit failures or consumer rebalances happen, they lose crucial diagnostic traces.

In this guide, we’ll dissect Kafka client library logging architecture based on the SLF4J facade, arrange optimal Logback and Log4j2 configurations for various environments (development vs production), translate critical client error messages, and implement log correlation techniques using Mapped Diagnostic Context (MDC) for end-to-end message tracking.

Logging Architecture on Apache Kafka Clients #

The Apache Kafka Java client library (kafka-clients) is designed using SLF4J (Simple Logging Facade for Java) as its log recording facade. This library doesn’t bind itself to specific logging frameworks, but leaves backend logging choices to our consuming applications.

flowchart TD
    App["Our Application / Business Code"] --> Facade["SLF4J Facade API (Logger/LoggerFactory)"]
    Lib["Kafka Client Library"] --> Facade
    Facade --> Logback["Logback Binding"]
    Facade --> Log4j2["Log4j2 Binding"]
    Facade --> JUL["JUL Binding"]
    Logback --> LogbackConf["logback.xml / logback-test.xml"]
    Log4j2 --> Log4j2Conf["log4j2.xml"]
    JUL --> JULConf["logging.properties"]

With this architecture, we can align Kafka client log formats, rotation, and output destinations with our own application logging systems. The three most popular logging backends used in the Java ecosystem are Logback (the Spring Boot default), Log4j2, and Java Util Logging (JUL).

Client Log Configuration Strategies: Dev vs Production Environments #

The main challenge in managing Kafka client logs is the generated log data volume. At DEBUG or TRACE verbosity levels, Kafka client libraries record every consumer heartbeat, cluster metadata update, and socket polling process. For that, we must distinguish client log configuration strategies based on operational environments.

1. Development Environments (Development/Staging) #

In these environments, we need as much detail as possible to understand connection handshakes, topic partition structures, and rebalance task division behaviors. We’re advised to set the org.apache.kafka category log level to DEBUG.

2. Production Environments #

In production, excessive log writing at DEBUG levels triggers significant CPU and disk I/O overhead on our application servers. Therefore, we must limit default log levels to INFO or WARN, and only raise verbosity levels on specific modules when actively investigating problems.

Here’s a balanced Logback configuration example (logback.xml) for production:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <!-- Console Appender (Standard Output) -->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- File Appender for Isolated Kafka Client Logs -->
    <appender name="KAFKA_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/var/log/app/kafka-client.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>/var/log/app/kafka-client-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <maxHistory>7</maxHistory>
            <totalSizeCap>500MB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Application Root Logging Settings -->
    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
    </root>

    <!-- Special Kafka Client Library Configuration -->
    <!-- By default, set to WARN to avoid noise -->
    <logger name="org.apache.kafka" level="WARN" additivity="false">
        <appender-ref ref="KAFKA_FILE"/>
        <appender-ref ref="CONSOLE"/>
    </logger>

    <!-- Except for the Group Coordinator, use INFO so we can track Rebalances -->
    <logger name="org.apache.kafka.clients.consumer.internals.ConsumerCoordinator" level="INFO" additivity="false">
        <appender-ref ref="KAFKA_FILE"/>
        <appender-ref ref="CONSOLE"/>
    </logger>

    <!-- Monitor the initial connection process (Metadata) at INFO level -->
    <logger name="org.apache.kafka.clients.NetworkClient" level="INFO" additivity="false">
        <appender-ref ref="KAFKA_FILE"/>
    </logger>

</configuration>

Here’s the equivalent if we use Log4j2 (log4j2.xml):

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="ConsoleAppender" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        
        <RollingFile name="KafkaRollingFile" fileName="/var/log/app/kafka-client.log"
                     filePattern="/var/log/app/kafka-client-%d{yyyy-MM-dd}-%i.log.gz">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="50 MB"/>
                <TimeBasedTriggeringPolicy/>
            </Policies>
            <DefaultRolloverStrategy max="7"/>
        </RollingFile>
    </Appenders>
    
    <Loggers>
        <Root level="info">
            <AppenderRef ref="ConsoleAppender"/>
        </Root>
        
        <!-- The Kafka Client Library in general -->
        <Logger name="org.apache.kafka" level="warn" additivity="false">
            <AppenderRef ref="KafkaRollingFile"/>
            <AppenderRef ref="ConsoleAppender"/>
        </Logger>
        
        <!-- Monitor consumer group lifecycles -->
        <Logger name="org.apache.kafka.clients.consumer.internals.ConsumerCoordinator" level="info" additivity="false">
            <AppenderRef ref="KafkaRollingFile"/>
        </Logger>
    </Loggers>
</Configuration>

Error Detection Flow Diagram in Client Libraries #

Before diving into specific error message details, we need to understand how client libraries process transmission failures from network socket levels to exception handling in our application code.

Here’s an internal error detection flow visualization in Java client libraries:

flowchart TD
    AppStart["Application: producer.send() / consumer.poll()"] --> NetworkSend["NetworkClient: Send Request via TCP Socket"]
    NetworkSend --> SocketCheck{"Is the Socket Connection OK?"}
    
    SocketCheck -- "No (RST / Timeout)" --> ErrorSocket["Throw NetworkException / DisconnectException"]
    ErrorSocket --> RetryCheck{"Are Retries Allowed?"}
    
    SocketCheck -- "Yes" --> AwaitResponse["Wait for the Broker Response"]
    AwaitResponse --> RespCheck{"Does the Response Contain an Error Code?"}
    
    RespCheck -- "Yes (Retriable, e.g. NotLeaderForPartition)" --> RetryCheck
    RespCheck -- "Yes (Non-Retriable, e.g. RecordTooLarge)" --> FailImmediate["Wrap as a Fatal Exception"]
    RespCheck -- "No" --> Success["Return Metadata / Successful Record"]
    
    RetryCheck -- "Yes (Retries>0 & Time Not Timed Out)" --> RefreshMetadata["Request Cluster Metadata Refresh"]
    RefreshMetadata --> NetworkSend
    
    RetryCheck -- "No" --> FailImmediate
    
    FailImmediate --> Callback["Trigger Callback / Throw RuntimeException to the Application"]

By understanding this diagram, we know that not all WARN messages in client logs mean our applications fail to send or receive data. Many of them are automatically recoverable errors (retriable errors).

Decoding and Solving Key Client Log Errors #

When fatal problems occur that can’t be handled by automatic client retry mechanisms, client libraries record errors to logs and throw them into our application threads. Let’s dissect the five key errors most often encountered in production.

1. RecordTooLargeException #

This error appears when producers try sending one single message (or message batch) whose size exceeds the configured maximum acceptance limits.

Example Log Message:

org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

Root Cause Analysis: Kafka has several message size gatekeeper parameters that must be consistently aligned from producer ends to brokers and consumers:

• On producers: max.request.size limits the maximum request size sent in one network call (default 1 MB).
• On brokers: message.max.bytes (per-topic or global configurations) limits the maximum record size allowed into disks (default 1 MB).
• On consumers: max.partition.fetch.bytes limits the maximum data amount fetched from each partition in one fetch (default 1 MB).

If producers try sending 2 MB messages without changing these settings, brokers immediately reject those requests and throw RecordTooLargeException to producer callbacks.

Solution Steps: If we truly must send large messages (e.g., 5 MB), we must change parameters on all three lines:

1. On brokers (server.properties or dynamic per-topic):

   message.max.bytes=5242880
   

2. On producers:

   max.request.size=5242880
   

3. On consumers:

   max.partition.fetch.bytes=5242880
   

Note: It’s highly recommended to enable data compression (compression.type=zstd or lz4) on producer sides before raising these size limits.

2. CommitFailedException (Offset Commit Failed) #

This error exclusively occurs on consumer sides when using manual offset commit management schemes.

Example Log Message:

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.

Root Cause Analysis: The log message above actually explains the cause very well. Inside Kafka consumer architectures:

• Consumer threads are obliged to periodically send heartbeat signals declaring themselves healthy.
• Besides heartbeats, consumers must routinely call poll() functions. The maximum time range between poll() calls is limited by the max.poll.interval.ms property (default 5 minutes).
• If our application threads spend too long processing messages returned by one poll() call (e.g., doing slow database I/O operations, external API integrations experiencing timeouts), the pause time between the next poll() calls exceeds the 5-minute limit.
• Coordinators on broker sides consider those consumers dead (hung), remove consumers from groups, and trigger rebalances to shift partitions to other group members.
• When the slow consumer finally finishes processing its data and tries calling commitSync(), the broker rejects that commit because partition ownership has already changed hands.

Solution Steps:

1. Lower Batch Sizes: Limit the record count fetched in one poll by shrinking max.poll.records values (default 500). Set it to e.g., 50 or 10 if each record’s processing takes long.
2. Raise Time Limits: Increase max.poll.interval.ms values in consumer configurations (e.g., to 10 or 15 minutes) if batching processes can’t be avoided.
3. Delegate to Thread Pools: Implement multi-threading architectures where main consumer threads only handle poll() and insert tasks into memory queues (BlockingQueue), while business execution is done by worker thread pools.

3. SSLHandshakeException #

This error occurs on Kafka clusters enabling transit encryption (SSL/TLS) when client libraries fail to build secure connections with brokers.

Example Log Message:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to neighboring target

Root Cause Analysis: The JVM where our client applications run doesn’t trust the Certificate Authority (CA) that signed Kafka broker SSL certificates. This commonly happens with self-signed certificate usage or company internal CAs whose root certificates haven’t been registered into Java’s built-in truststore.

Solution Steps: We must provide a truststore file containing the broker public CA certificate in Kafka client configurations:

security.protocol=SSL
ssl.truststore.location=/var/private/ssl/client.truststore.jks
ssl.truststore.password=OurTruststorePassword123

If we also enable two-way authentication (mTLS), make sure the client keystore file containing client private certificates and public keys is correctly configured:

ssl.keystore.location=/var/private/ssl/client.keystore.jks
ssl.keystore.password=OurKeystorePassword123
ssl.key.password=OurPrivateKeyPassword123

4. DisconnectException and Network Timeouts #

These connection termination messages are often confusing because they usually appear at WARN warning levels yet occur repeatedly.

Example Log Message:

org.apache.kafka.common.errors.DisconnectException: Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

Root Cause Analysis:

1. Dead Brokers: Target broker nodes are truly dead or in restart processes.
2. Idle Connections Terminated: By default, Kafka has the connections.max.idle.ms property (default 9 minutes) on both broker and client sides. If there’s no data transmission between clients and specific brokers during that time (e.g., on topics rarely receiving messages), one party cleanly closes TCP connections. When clients need the connection again, they record DisconnectException before building new sockets. This scenario is normal and harmless.
3. DNS/IP Advertisement Problems: Clients successfully connect to one bootstrap server, but when requesting metadata, brokers return internal host addresses (advertised listeners) unreachable by client application networks (common problems in cross-VPC Docker/Kubernetes deployments).

Solution Steps:

• Check advertised.listeners configurations on brokers. Make sure hosts advertised by brokers can be DNS-resolved and ports accessed by servers running client applications.
• Use the nc -zv [broker-host] [port] command from application servers to make sure there are no firewall blocks.

5. GroupAuthorizationException #

This error occurs on consumer sides when authorization security features (ACLs) are enabled in Kafka clusters.

Example Log Message:

org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: payment-group

Root Cause Analysis: The authentication credentials (SSL Principals or SASL Usernames) used by our consumer applications don’t have access rights (Read) to use the specified Consumer Group name (payment-group). Without this permission, broker coordinators reject group join requests (JoinGroup).

Solution Steps: Ask our Kafka administrators to add the Read ACL for the relevant Group resource:

kafka-acls.sh --bootstrap-server localhost:9093 \
  --command-config /etc/kafka/client.properties \
  --add \
  --allow-principal User:CN=payment-app,O=MyCorp \
  --operation Read \
  --group payment-group

Application Log Correlation Techniques with Kafka Metadata #

One of the biggest challenges in microservices architectures is tracking a message’s journey from start to finish (end-to-end tracing). When there are complaints that a specific payment transaction wasn’t processed, we must be able to trace application logs from API Gateways, into Kafka Producers, entering Brokers, until processed by Consumers.

To do this efficiently, we can leverage MDC (Mapped Diagnostic Context) supported by Logback and Log4j2. MDC allows us to insert contextual metadata (like correlation_id, topic, partition, and offset) into local threads so every written log line automatically contains that information.

Here’s a Java consumer implementation example enriching MDC on every message fetch:

package com.mycompany.kafka.consumer;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class TransactionConsumer {

    private static final Logger log = LoggerFactory.getLogger(TransactionConsumer.class);

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "payment-processor");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("enable.auto.commit", "false");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("payment.orders"));

        try {
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                
                for (ConsumerRecord<String, String> record : records) {
                    // 1. Extract the correlation ID from the record header (if sent by the producer)
                    String correlationId = getCorrelationIdFromHeader(record);
                    
                    // 2. Insert Kafka metadata into the Mapped Diagnostic Context (MDC)
                    MDC.put("kafka.topic", record.topic());
                    MDC.put("kafka.partition", String.valueOf(record.partition()));
                    MDC.put("kafka.offset", String.valueOf(record.offset()));
                    MDC.put("correlation.id", correlationId);

                    try {
                        // 3. Run our application business logic
                        log.info("Starting to process the payment transaction for payload: {}", record.value());
                        processPayment(record.value());
                        log.info("Finished processing the payment transaction successfully.");
                        
                        // Commit manually after success
                        consumer.commitSync();
                    } catch (Exception e) {
                        log.error("Failed to process the transaction at offset {}", record.offset(), e);
                    } finally {
                        // 4. Clear the MDC to prevent context leaks to other threads
                        MDC.clear();
                    }
                }
            }
        } finally {
            consumer.close();
        }
    }

    private static String getCorrelationIdFromHeader(ConsumerRecord<String, String> record) {
        if (record.headers() != null) {
            var header = record.headers().lastHeader("X-Correlation-ID");
            if (header != null) {
                return new String(header.value());
            }
        }
        // Fall back to a random UUID if not provided by the producer
        return java.util.UUID.randomUUID().toString();
    }

    private static void processPayment(String payload) throws Exception {
        // Simulate business processing logic
        Thread.sleep(50); 
    }
}

To make these MDC values printed in our log files, we must update the <pattern> section in our logback.xml files by inserting %X{key} variables:

<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - [CorrID: %X{correlation.id}] [Topic: %X{kafka.topic} Part: %X{kafka.partition} Off: %X{kafka.offset}] - %msg%n</pattern>

When our applications run, log outputs will look very informative like this:

2026-06-08 15:40:12.512 [main] INFO  c.m.k.c.TransactionConsumer - [CorrID: tx-8829-ac91] [Topic: payment.orders Part: 2 Off: 10421] - Starting to process the payment transaction for payload: {"order_id": 9921}
2026-06-08 15:40:12.565 [main] INFO  c.m.k.c.TransactionConsumer - [CorrID: tx-8829-ac91] [Topic: payment.orders Part: 2 Off: 10421] - Finished processing the payment transaction successfully.

With log formats like this, our operations teams can easily search in Elasticsearch/Splunk by filtering correlation IDs or specific partition-offset combinations when analyzing transaction failures.

Operational Compliance and Client Log Audit Checklist #

Do periodic evaluations of our client application logging configurations using the following audit table to guarantee production system compliance:

Summary #

• Use the SLF4J Facade — Leverage the SLF4J facade to bind Kafka client libraries with our application’s main logging frameworks (like Logback or Log4j2).
• Limit Production Log Levels — Set org.apache.kafka log levels to WARN in production environments to minimize disk I/O overhead, but leave consumer coordination modules at INFO levels.
• Decode Errors Quickly — Understand critical error root causes like RecordTooLargeException (size limit alignment) and CommitFailedException (processing times exceeding polling limits).
• Correlate Using MDC — Implement Kafka metadata filling into Mapped Diagnostic Contexts (MDC) to ease end-to-end message tracking across various services.

← Previous: Broker Log Next: Debugging Message Flow →