Kafka Internals — A Deep Dive for the Curious Engineer

A no-fluff guide to the parts of Kafka that actually bite you in production.

I have been working with Kafka for a while now, and here is the thing — most Kafka content out there falls into two buckets. Either it is “Kafka is a distributed messaging queue, here is how to produce and consume” (yeah, I know), or it is a 200-page Confluent whitepaper that puts me to sleep by page 4.

There is a weird gap in the middle: the advanced stuff, explained like a human would explain it to another human over coffee.

So that is what this post is. I am going to walk through the parts of Kafka that actually trip people up in production — partitioning strategy, hot partitions, consumer group rebalancing, offset management traps, and a few performance tricks that are surprisingly underused.

I cannot stress this enough. The single most impactful decision you make when designing a Kafka-based system is not how many brokers to run or what replication factor to set. It is what you choose as your partition key.

Here is why.

Kafka guarantees ordering only within a single partition. Not across partitions. Not across topics. Within. A. Single. Partition. So when you pick a partition key, you are really answering the question: “What is the unit of ordering that matters to my system?”

Under the hood, Kafka does something pretty straightforward:

partition = hash(key) % number_of_partitions

It hashes your key (using murmur2 by default), mods it by the partition count, and that is where your message lands. Simple math, massive implications.

Let me walk you through how I think about this in practice.

Event tracking system — Say you are building a system that tracks user activity on a platform. You want all events from a single user to be processed in order (login then page view then purchase, not purchase then login then page view). Your partition key? user_id. Every event from user #4821 lands on the same partition, processed in sequence.

Payment processing pipeline — You need all transactions for a given merchant to be sequential. Partition key? merchant_id. This way, refunds always come after the charge they are refunding, and settlement always happens in the correct order.

IoT sensor data — Millions of sensors pushing telemetry data. You want temporal ordering per device but do not care about ordering across devices. Partition key? device_id.

The pattern here is simple: partition by the entity whose ordering you care about.

But here is where it gets interesting — and where most people screw up.

Let us say you are building an ad click aggregation pipeline. You choose ad_id as your partition key because you want to count clicks per ad in order. Sounds reasonable, right?

Then Nike drops a Super Bowl ad.

Suddenly, one ad_id is generating 50x the traffic of every other ad combined. That one partition is drowning while the other 99 partitions are practically idle. You have got yourself a hot partition, and your throughput just cratered because Kafka parallelism only works when load is distributed.

This is probably the most common production issue I have seen with Kafka, and it is entirely a design problem, not an infrastructure problem. No amount of adding brokers will fix a hot partition — because all the traffic for that key is pinned to one partition by design.

I have used four strategies in practice, each with its own trade-off.

1. Drop the key entirely (random distribution)

If you genuinely do not need ordering, just do not provide a key. Kafka will round-robin messages across partitions, and you get perfectly even distribution. This is great for use cases like logging, metrics collection, or any fire-and-forget pipeline.

The catch? You lose ordering guarantees completely. For my ad click example, that means click events for the same ad might be processed out of order. If your downstream consumer can handle that (say it is doing batch aggregation every 5 minutes anyway), this is the simplest solution.

2. Salt the key

This is my go-to when I need some distribution but can tolerate a bit of complexity on the consumer side. Instead of using ad_id as the key, I use ad_id + "_" + random(0, N) where N is a small number, say 10.

import random

def salted_key(ad_id, salt_factor=10):

salt = random.randint(0, salt_factor - 1)

return f"{ad_id}_{salt}"


Now that Nike ad traffic is spread across roughly 10 partitions instead of hammering one. The trade-off? Your consumer now needs to aggregate across multiple partitions to get the full picture for a single ad. You are essentially moving complexity from the broker layer to the application layer.

In practice, this works surprisingly well with stream processing frameworks like Flink or Kafka Streams, where you can do a secondary groupBy on the real ad_id after the initial parallel processing.

3. Compound keys

Instead of salting randomly, use a second dimension that is meaningful to your business. For the ad example, use ad_id + region as the key:

key = "nike_superbowl_ad:us-west"key = "nike_superbowl_ad:us-east"key = "nike_superbowl_ad:eu-west"

Now you get natural distribution (because traffic is geographically distributed) AND you preserve ordering within each region. Your downstream consumer can aggregate across regions if needed, but each regional stream is perfectly ordered.

I love this pattern because the distribution is deterministic. Unlike random salting, the same region events always hit the same partition, which makes debugging much easier.

4. Back pressure

Sometimes the right answer is: slow down. If your producer is generating events faster than any single partition can handle, you can implement back pressure — the producer monitors partition lag and throttles itself when a partition is falling behind.

This does not solve the distribution problem, but it prevents cascading failures. I think of it as a safety valve rather than a real solution. Use it alongside one of the other strategies, not instead of them.

Here is a scenario. You have got a consumer group with 6 consumers processing messages from a topic with 12 partitions. Everything is humming along — each consumer handles 2 partitions. Then consumer #3 crashes.

Kafka detects the failure (via missed heartbeats) and triggers a rebalance. During this rebalance, all consumers in the group stop processing. Yes, all of them. Not just the partitions that the dead consumer was handling — everything pauses while Kafka redistributes partition assignments.

In the default “eager” rebalancing protocol, this stop-the-world pause can take anywhere from a few seconds to over a minute depending on your group size. For a high-throughput pipeline, that is potentially millions of messages backing up.

The fix? Use the cooperative sticky assignor (available since Kafka 2.4). Instead of revoking all partitions and reassigning from scratch, it only migrates the partitions that actually need to move. The other consumers keep processing uninterrupted.

In practice, setting this up is just a configuration change:

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

This alone reduced our rebalance-related downtime by over 90% in one system I worked on. If you are not using it already, switch today.

Everyone knows consumers track their position via offsets. But the when and how of committing offsets is where things get interesting.

By default, Kafka consumers auto-commit offsets every 5 seconds. This means if your consumer reads a batch of messages, starts processing them, and then crashes 3 seconds in — those messages were never committed, so Kafka re-delivers them. Fine so far.

But what if your consumer reads a batch, the auto-commit fires, and then the consumer crashes before finishing processing? Now those offsets are committed but the work was never completed. You have just silently lost messages.

For any system where message loss is not acceptable, disable auto-commit and commit manually after you have finished processing:

consumer = KafkaConsumer( 'my_topic', enable_auto_commit=False, group_id='my-group')

for message in consumer: process(message) # do the actual work first consumer.commit() # THEN commit the offset

The trade-off is that if you crash after processing but before committing, you will reprocess that message. This means your consumers need to be idempotent — processing the same message twice should produce the same result as processing it once. Design for at-least-once delivery, enforce idempotency at the application layer.

Let me rapid-fire through a few things that have made real differences in systems

Kafka producers do not send messages one at a time by default — they batch them. But the default batch settings are conservative. Tuning linger.ms (how long to wait before sending a batch) and batch.size (max batch size in bytes) can dramatically improve throughput.

we can start with linger.ms=20 and batch.size=64000, then adjust based on latency requirements. The idea is: a small delay on each individual message can result in massive throughput gains because you are sending fewer, larger network requests.

Enable it. Seriously. GZIP gives the best compression ratio, but LZ4 or Snappy give better throughput because they are faster to compress and decompress. For most use cases, I default to LZ4 — it cuts message sizes by 60 to 70 percent with minimal CPU overhead.

This one is less about configuration and more about understanding why Kafka is fast. Kafka does not manage its own in-memory cache — it relies on the OS page cache. This means Kafka reads are essentially reading from memory (if the data is recent enough to still be in cache) without any JVM garbage collection overhead.

On top of that, Kafka uses the sendfile() system call (zero-copy) to transfer data directly from the page cache to the network socket without copying it through application memory. This is why Kafka can saturate a 10Gbps network link with a single broker — it is barely doing any work at the application level.

The practical takeaway? Do not put Kafka brokers on machines with small amounts of RAM. The more RAM, the more data stays in page cache, and the faster your reads. This is one of those cases where throwing hardware at the problem actually works.

Kafka default retention is 7 days. But retention is not just a storage decision — it affects your system recovery capabilities.

If a consumer goes down for 3 days and your retention is 7 days, no problem — it picks up where it left off. If your retention is 2 days, those messages are gone.

teams use compacted topics in clever ways. Instead of time-based retention, compacted topics keep only the latest message for each key. This is perfect for things like user profile snapshots or configuration state — you always have the latest version without storing the full history.

Think of it this way: regular retention says “keep everything from the last N days.” Compaction says “keep the latest version of everything, forever.”

Kafka is deceptively simple on the surface but has a surprising amount of depth once you start pushing it.

Your partition key is your most important design decision. Get it wrong and no amount of infrastructure will save you. Hot partitions are a design problem, not a scaling problem. Solve them with salting, compound keys, or by rethinking whether you need ordering at all. Consumer group rebalancing can silently wreck your throughput — use cooperative sticky assignment. Commit offsets manually and design idempotent consumers. Do not fight Kafka architecture — lean into batching, compression, and let the OS page cache do its thing.

Kafka is not magic. It is a well-engineered append-only distributed log with some very smart defaults. The more you understand those internals, the better systems you will build on top of it.

Thanks for reading