When Should You Use Kafka (and When Not)? #

In software engineering, the temptation to use the newest, most popular technology is enormous. Apache Kafka is often seen as a silver bullet that can solve all asynchronous data communication problems. Many developer teams rush to adopt Kafka just because of its reputation at giants like Netflix, Uber, or LinkedIn. However, installing and running Kafka for an application that doesn’t actually need it is a very expensive form of over-engineering, both in terms of infrastructure and maintenance time.

In this article, we’ll dive deep into the architecture decision criteria: when you should use Apache Kafka and when you should avoid Kafka and choose a simpler technology instead. We’ll provide an eligibility checklist, real-world scenarios, anti-pattern analysis, and practical guidance to save your project budget and your operations team’s sanity.

When Should You Use Apache Kafka? #

Apache Kafka shines when used for scenarios that match its core strengths. Here are some key indicators that your system needs Kafka:

1. Large-Scale Event Streaming Needs (High Throughput) #

If your application generates millions of events per minute that must be processed immediately, Kafka is the best choice. For example, a real-time GPS location tracker for ride-hailing drivers, a sensor metric collector from thousands of IoT devices in a factory, or a clickstream recorder for millions of users on an e-commerce website. Traditional queue systems quickly collapse under this volume of writes, while Kafka handles it with ease thanks to sequential writes and page cache utilization.

2. One Data Stream Read by Many Independent Services (Fan-out Replay) #

One of Kafka’s strongest scenarios is when a single raw data stream needs to be consumed by several different systems for different purposes, without interfering with each other.

For example, a user shopping transaction data stream needs to be read by:

• The Finance Service: To update balance sheets.
• The Recommendation Service: To update shopping preferences in real-time.
• The Compliance Service: To detect signs of money laundering (fraud detection).
• The Notification Service: To send payment receipts to user emails.

With Kafka, all these consumers can read from the same topic asynchronously from their own offset positions without overloading broker memory.

flowchart LR
    Topic["Transaction Topic"] -->|"Pull Event"| C1["Finance Service (Offset 10)"]
    Topic -->|"Pull Event"| C2["Recommendation Service (Offset 9)"]
    Topic -->|"Pull Event"| C3["Anti-Fraud Service (Offset 10)"]
    Topic -->|"Pull Event"| C4["Notification Service (Offset 8)"]

3. The Need to Reprocess Historical Data (Replayability) #

In software development, bugs are inevitable. What do you do if a logic error in your consumer application code causes transaction data from the last 5 hours to be calculated incorrectly?

If you use a traditional broker, that data is gone forever once consumed. But with Kafka, you simply fix the bug in your application code, redeploy the service, then rewind the consumer’s offset pointer to 5 hours ago. Your application automatically re-reads that historical data from disk and repairs the data state in your main database.

4. Log Aggregation & Security Monitoring (SIEM) #

Large enterprises with thousands of containerized applications (Kubernetes) generate gigabytes per second of operational and security logs. These logs must be channeled into security analysis systems (SIEM) and centralized search engines (like Elasticsearch). Kafka acts as a giant, highly reliable buffer that collects all logs from various servers, protecting them from loss if the Elasticsearch server experiences processing slowdowns.

5. Database Synchronization via Change Data Capture (CDC) #

When you need to replicate data from an operational relational database (OLTP) to analytical databases (OLAP or data warehouse) in real-time without locking the primary database tables, you use the CDC technique.

Kafka is an outstanding CDC backbone. Libraries like Debezium read the database’s transaction log (Write-Ahead Log / WAL) and send every row change as an event to Kafka. From Kafka, the data is instantly synchronized to the data warehouse.

When Should You Avoid Apache Kafka? #

Conversely, there are many scenarios where using Kafka is a mistake. Here are indicators that you should pick an alternative:

1. Simple CRUD and Monolith Applications #

If your application is an internal company CRUD (Create, Read, Update, Delete) system with a small number of daily active users, or a simple monolith without many distributed microservices, you don’t need Kafka. A relational database like PostgreSQL or MySQL is more than enough to handle simple asynchronous transactions using database job queues or local async libraries.

2. Limited Operations Team Expertise (SRE / DevOps) #

Kafka is a highly complex distributed system. Running a production-grade Kafka cluster requires specialized expertise in JVM administration, Linux OS parameter tuning, metadata management (KRaft/Zookeeper), partition strategy, and strict disk I/O and network metric monitoring.

If your operations team is small and lacks experience managing complex distributed systems, running Kafka on your own is a big risk. Your system could experience long downtime just from a small memory or disk configuration error.

3. You Need Strict Global Ordering for All Messages #

Kafka only guarantees consistent message ordering within the same partition, not across the entire topic. If you have a topic with 10 partitions, and messages are sent randomly to various partitions, global message order won’t be preserved.

If your application demands 100% consistent message ordering across the entire system globally without exception, you’re forced to use only 1 partition for that topic. Using just 1 partition in Kafka eliminates all the benefits of parallelism and horizontal scalability, making Kafka run slower than a simple traditional message broker.

4. You Need Dynamic and Complex Message Routing #

If your system needs a smart broker that routes messages dynamically based on message content (for example: “Send this message to queue A only if the location attribute is Jakarta and the transaction value is above 1 million”), Kafka isn’t designed for that. Kafka is a dumb broker that only stores raw log data without caring about its contents, relying on intelligence on the client application side (smart client). For such cases, use RabbitMQ, which has very flexible Topic and Headers exchange types.

5. You Need Delayed / Scheduled Message Features #

Many applications need messages delivered after a certain delay (for example: “Send a reminder email 3 days after a user registers”). Kafka doesn’t support delayed or scheduled messages natively.

Trying to simulate delayed messages in Kafka requires building a complex custom architecture using Kafka Streams with local state stores or writing data to temporary staging topics, which wastes resources. In contrast, traditional brokers like RabbitMQ have a delayed message exchange plugin designed specifically for this use case out-of-the-box.

Kafka Eligibility Checklist #

To help you analyze architecture eligibility instantly, use the checklist below before deciding to use Kafka:

NEED Apache Kafka if:
  ✓ Your data flows continuously (streaming) with very high throughput (thousands of events per second).
  ✓ Data needs to be consumed asynchronously by many independent systems with different purposes.
  ✓ You need historical data replay (replayability) for failure recovery.
  ✓ Your system adopts large-scale Event Sourcing or CQRS architecture patterns.
  ✓ Your team has SRE operational capacity ready to manage distributed infrastructure.

DON'T NEED Apache Kafka if:
  ✗ Your workload is dominated by synchronous REST API or gRPC communication (Request-Response).
  ✗ Your message volume is small (only a few thousand messages per day) and inconsistent.
  ✗ You need very complex broker-level message routing (like RabbitMQ).
  ✗ You want to use Kafka as a relational database for random (ad-hoc) queries.
  ✗ Your team lacks experienced DevOps/SRE specialists in distributed system tuning.

Anti-Patterns in the Industry #

Let’s study some real architectural mistakes in the industry so you can avoid decisions that hurt your system’s performance:

Anti-Pattern 1: Using Kafka as a Primary Database for Random Queries #

An e-commerce startup decides to ditch their MySQL database and use Apache Kafka as the permanent storage for their customer order data. They assume that since Kafka stores data persistently and durably, they can save on database licensing costs.

When building the transaction history feature in their mobile app, developers must display a list of purchases by Customer ID in random access patterns.

Consequences: Because Kafka uses a linear append-only log data structure, there’s no search index like the B-Tree in SQL databases. To randomly find one order belonging to one specific customer, the application is forced to do a full scan from the beginning of the log at offset 0 to the end of the log across all partitions.

This triggers severe page faults in the server’s memory cache, destroys the OS Page Cache efficiency, causes disk I/O congestion on broker servers, and eventually crashes the application from CPU resource exhaustion.

# ANTI-PATTERN: Trying to read random data from Kafka for dynamic queries
# Random reads require scanning the entire log file, which is very slow and destroys disk I/O performance.
def find_customer_order_wrong(client_id):
    # SCANNING THE ENTIRE KAFKA LOG FROM THE START
    log_file = open_kafka_log()
    for record in log_file:
        if record.client_id == client_id:
            return record.transaction_detail
    return None

# The CORRECT solution:
# Use Kafka only as a delivery pipeline for transaction events.
# Use a relational database (PostgreSQL) as the read model that indexes transaction data by Customer ID.
def find_customer_order_right(client_id):
    # Using the relational database index (very fast, O(log N))
    return database.query("SELECT detail FROM orders WHERE client_id = ?", client_id)

Anti-Pattern 2: Using Kafka for Synchronous REST API Request-Response #

A microservices architecture team wants all inter-service communication to be asynchronous for maximum decoupling. For their REST order API service, they build this flow: Client calls the order REST API -> Order Service sends an order event to Kafka -> Inventory Service processes the event -> Inventory Service sends a result event back to Kafka -> Order Service reads the result event from Kafka -> Order Service returns an HTTP response to the client. They try to use Kafka as a mediator for a Request-Response pattern that is actually synchronous from the end user’s perspective.

Consequences: Correlating asynchronous responses (using a Correlation ID) to return a synchronous HTTP response in real-time is very hard to manage at high scale. It demands the Order Service keep HTTP connection state open in memory while monitoring a Kafka topic for matching reply messages. The system suffers very high latency from data traveling back and forth across the network. When REST request volume spikes, the Order Service runs out of HTTP connection threads because it waits too long for reply events from Kafka, crashing the web server from resource exhaustion and triggering timeout errors for end users.

# ANTI-PATTERN: REST API http-request -> Producer -> Kafka -> Consumer -> Kafka -> http-response
# Asynchronous synchronization over a synchronous HTTP protocol is an anti-pattern that causes thread leaks.

# The CORRECT solution:
# Use direct synchronous communication (REST API or gRPC) for interactions that need instant responses.
# Use Kafka only for background asynchronous processing (like invoice generation and analytics updates).

First Steps to Adopting Kafka #

If, after evaluating the criteria above, you decide Kafka is the right choice for your architecture needs, the recommended first step is not to migrate the entire system at once. Start with a low-risk, high-value use case, such as application performance metric collection (metrics ingestion) or a secondary analytics log data pipeline. This gradual approach lets your developer and operations teams build practical expertise in configuring and monitoring Kafka without threatening the smooth operation of the company’s main transaction systems.

Summary #

• Workload Analysis — Eligibility evaluation should be based on data throughput, fan-out integration needs, and the importance of data replayability.
• Avoid Over-Engineering — Don’t use Kafka for monolith applications, simple CRUD systems, or when your team lacks distributed infrastructure operations expertise.
• Data Search Limits — Understand that Kafka is a linear log system; it’s not designed to serve advanced random (ad-hoc) data queries efficiently without a supporting database.
• Proper Integration Pattern — Use Kafka as an event-based asynchronous pipeline, then channel the data into a relational or NoSQL database as the application’s read repository.
• Not a Request-Response Replacement — Kafka is designed for asynchronous event streaming communication, not two-way synchronous interaction. Trying to use it for Request-Response API synchronization only hurts latency performance and triggers thread resource failures.