Apache Kafka

What Kafka actually is

Apache Kafka is a distributed, append-only commit log. That single sentence explains almost everything else. Records are appended to the end of a log, each one receives a monotonically increasing number called an offset, and nothing is ever mutated in place. Readers do not remove records when they consume them; they simply move a cursor forward.

This is the fundamental difference from a classic message queue. In a queue, a consumer takes a message and the message is gone. In Kafka, a consumer reads a record, remembers its offset, and the record stays where it is for as long as the topic’s retention policy allows. Ten different consumers — and ten different applications — can read the same record independently, at their own pace, without coordinating with each other.

Kafka was built at LinkedIn to handle activity streams: page views, clicks, log lines, all at millions of events per second. It was open-sourced in 2011 and became an Apache project in 2012. Today it is the default backbone for event streaming: change data capture, metrics pipelines, event sourcing, log aggregation and stream processing.

If you come from RabbitMQ or Redis, the mental shift is to stop thinking “a message to be delivered” and start thinking “a fact that was recorded”.

Topics, partitions and ordering

A topic is a named, durable log. Producers write to it, consumers read from it. Topics are split into partitions, and the partition is the unit of both parallelism and ordering.

• Records within one partition are strictly ordered by offset.
• There is no ordering guarantee across partitions.
• More partitions allow more parallel consumers, at the cost of more files, more replication and slower rebalances.

That is the single most important guarantee in Kafka: ordering is per partition. If two events must be processed in order, they must land in the same partition. If they do not, Kafka may hand them to different consumers that run concurrently, and order is lost.

Partitions also determine the maximum consumer parallelism in a group: a group can have at most one consumer per partition actively reading it. Six partitions means at most six useful consumers, and a seventh sits idle.

# three partitions, replicated across three brokers
kafka-topics.sh --create \
  --topic orders.created \
  --partitions 6 \
  --replication-factor 3 \
  --bootstrap-server localhost:9092

Adding partitions later is possible, but it changes the mapping from key to partition for existing keys, so events for one entity can end up split across two partitions and lose their relative order. Decide the partition count when the topic is created, with headroom for growth.

Producers, keys and partitioning

A producer serialises a record and decides which partition it belongs to. The default partitioner hashes the record key and maps it onto a partition. The same key always goes to the same partition, which is how you preserve order for one entity while spreading different entities across partitions.

await producer.send({
  topic: "orders.created",
  messages: [
    { key: order.customerId, value: JSON.stringify(order) },
  ],
});

Choosing the key is a design decision, not a detail. Key by customerId and every event for a customer is ordered together. Key by orderId and you get maximum spread but no cross-event ordering. Records with a null key are distributed for balance — Kafka’s sticky partitioner fills one partition before moving to the next — but they carry no ordering promise at all.

The producer also controls durability and throughput through batching and acknowledgements. acks decides how many replicas must confirm a write, linger.ms and batch.size decide how long it waits to fill a batch, and compression.type decides how much CPU to trade for network and disk. More on those below.

Brokers, replication and the leader

A broker is a Kafka server. A cluster is several brokers working together. Every partition has one leader broker and zero or more followers. Producers and consumers talk to the leader; followers replicate the log.

Replication is what makes Kafka durable. If a leader dies, one of the in-sync replicas (ISR) is promoted and the cluster keeps serving. The acks setting decides how much the producer waits for:

• acks=0 — fire and forget; fastest and least safe.
• acks=1 — the leader wrote it; lost if the leader dies before replication.
• acks=all — all in-sync replicas acknowledged; safest.

Pair acks=all with min.insync.replicas=2 and a replication factor of three for a production setup: a write is only acknowledged when at least two replicas have it, so losing one broker does not lose data.

const producer = kafka.producer({
  idempotent: true,
  maxInFlightRequests: 1,
  transactionalId: "orders-producer",
});

The idempotent producer adds a sequence number to every batch so the broker can discard duplicates, which removes the accidental duplicates that retries can otherwise introduce. A cluster also has a controller that manages partition leadership and metadata. Older Kafka used ZooKeeper for this; modern Kafka uses KRaft, where the brokers form their own metadata quorum and ZooKeeper is gone entirely.

Consumer groups and partition assignment

Consumers belong to a consumer group, identified by groupId. Kafka assigns each partition to exactly one consumer within the group. That gives you two things at once: horizontal scaling, because partitions are shared, and load balancing, because no two consumers in a group process the same partition.

const consumer = kafka.consumer({ groupId: "billing" });

await consumer.connect();
await consumer.subscribe({ topic: "orders.created", fromBeginning: true });

await consumer.run({
  eachMessage: async ({ partition, message }) => {
    const order = JSON.parse(message.value!.toString());
    console.log(`p${partition} @ ${message.offset}`, order.id);
  },
});

When consumers join or leave, Kafka rebalances: it revokes assignments and hands out new ones. Rebalances pause consumption, so they are expensive; cooperative rebalancing protocols reduce the disruption by moving only the partitions that need to move. A consumer that stops sending heartbeats within session.timeout.ms is considered dead and its partitions are reassigned.

The group is also how Kafka remembers progress. Each group has its own committed offsets stored in the internal __consumer_offsets topic, so two groups reading the same topic can be at completely different positions without any coordination.

Rebalancing and the poll loop

Under the hood a consumer is a poll loop. It fetches records, hands them to your handler and then fetches again. The broker tracks liveness separately through heartbeats, so a slow handler does not immediately look dead, but there is a hard limit: if a single poll takes longer than max.poll.interval.ms, the broker assumes the consumer is stuck and triggers a rebalance.

const consumer = kafka.consumer({
  groupId: "billing",
  sessionTimeout: 45_000,
  heartbeatInterval: 3_000,
  rebalanceTimeout: 60_000,
});

A rebalance is triggered whenever a consumer joins, leaves or is evicted, whenever partitions or topics change, and whenever a subscription changes. During the rebalance Kafka revokes assignments and pauses consumption, so frequent rebalances are a throughput killer. Three habits keep them rare:

• Bound per-record work. A handler that sometimes runs for minutes will eventually blow past max.poll.interval.ms.
• Use cooperative rebalancing. The incremental cooperative protocol moves only the partitions that must move, instead of stopping the whole group.
• Use static membership. Setting a stable group.instance.id lets a restarted consumer reclaim its old partitions without triggering a rebalance at all.

Kafka also gives you consumer.pause() and consumer.resume() to apply backpressure when a downstream dependency is struggling. Pausing is better than blocking the poll loop: the consumer keeps heartbeating, stays in the group and simply stops fetching until you are ready.

Offsets and delivery guarantees

An offset is a consumer group’s position in a partition. When a consumer processes a record it can commit the offset, which tells Kafka “this group has finished everything up to here”. Commit too early and a crash loses work; commit too late and a crash reprocesses work.

• At-most-once — commit before processing. A crash loses the record.
• At-least-once — process, then commit. A crash reprocesses the record. This is the common default.
• Exactly-once — transactions and idempotent producers make the read-process-write cycle atomic within Kafka.

Auto-commit happens on a timer in the background. It is convenient but dangerous, because it can commit offsets for records that are still being processed. Manual commit after the work is done is the honest default.

await consumer.run({
  autoCommit: false,
  eachMessage: async ({ topic, partition, message }) => {
    await chargeOrder(JSON.parse(message.value!.toString()));
    await consumer.commitOffsets([
      { topic, partition, offset: String(Number(message.offset) + 1) },
    ]);
  },
});

Because at-least-once is the norm, handlers should be idempotent. The Batch Processing guide covers the same contract for job queues: assume the work can run twice and make the second run a no-op. A dedupe key derived from the work — not from the Kafka offset — makes the handler safe even if the same logical event is produced twice.

Exactly-once is real but narrower than it sounds. It works for Kafka-to-Kafka pipelines with transactional producers and isolation.level=read_committed, but the moment you write to an external database you are back to needing idempotent writes there.

Retention, replay and log compaction

Kafka keeps records according to a retention policy, not a delivery acknowledgement. retention.ms (default seven days) and retention.bytes bound each partition; when either is exceeded, old log segments are deleted. Because nothing is removed on read, a consumer can replay history by seeking to an earlier offset or resetting the group.

# rewind a group to the beginning of a topic
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group billing --topic orders.created \
  --reset-offsets --to-earliest --execute

Replay is Kafka’s superpower. A new service can be bootstrapped by reading the entire history of a topic. A bug fix can be deployed and the last day reprocessed. An analytics pipeline can be rebuilt from the raw event log. A queue cannot do any of this, because the data is already gone.

For keyed state, the other retention mode is log compaction. With cleanup.policy=compact, Kafka keeps at least the latest value for every key and discards older values. A null value is a tombstone that deletes the key. The log becomes a changelog that can rebuild a table — exactly what Kafka Streams uses for its state stores and what change-data-capture pipelines rely on.

# keep the latest value per key instead of deleting by age
kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name user.profiles \
  --add-config cleanup.policy=compact,min.cleanable.dirty.ratio=0.1

Schemas and the Schema Registry

Because the log outlives any single application, the shape of a record becomes a contract between teams. A producer and a consumer are coupled by the bytes they exchange, and that coupling survives deploys. A schema registry turns this into a managed compatibility problem instead of a surprise.

The registry stores versioned schemas — usually Avro, Protobuf or JSON Schema — and assigns each a numeric id. Producers register a schema and write the id alongside the payload; consumers fetch the schema by id and deserialise. When a schema changes, the registry enforces a compatibility mode such as backward or forward, rejecting a change that would break existing readers.

import { SchemaRegistry } from "@kafkajs/confluent-schema-registry";

const registry = new SchemaRegistry({ host: "http://localhost:8081" });

const encoded = await registry.encode(schemaId, {
  orderId: order.id,
  totalCents: order.totalCents,
});

await producer.send({
  topic: "orders.created",
  messages: [{ key: order.customerId, value: encoded }],
});

Even if you do not run a registry, treat payloads as an API: add fields rather than renaming them, give new fields a default, and version when the meaning changes. A consumer running the previous release must be able to read a record written by the next one.

Kafka Connect and Kafka Streams

Two parts of the platform save you from writing the same plumbing twice.

Kafka Connect is a framework for moving data in and out of Kafka with configuration rather than code. Source connectors pull from databases, object storage or SaaS APIs; sink connectors push to warehouses, search indexes or another database. Debezium, for instance, turns a Postgres write-ahead log into a stream of change events. Connect runs as a cluster, tracks offsets and retries failures, so it is the standard answer to “get data into Kafka” and “get data out”.

Kafka Streams is a client library for processing data in Kafka. It gives you a stream DSL — map, filter, groupByKey, join, window — over KStream and KTable abstractions, with state stores backed by compacted topics. It runs inside your application, scales by adding instances, and handles fault tolerance through changelog topics. If you prefer SQL, ksqlDB offers a query layer over the same ideas.

const stream = builder.stream("orders.created");

stream
  .filter((key, order) => order.totalCents > 10_000)
  .groupBy((key) => order.customerId)
  .windowedBy(tumblingWindow({ size: 60 * 60 * 1000 }))
  .count()
  .toStream()
  .to("customer.hourly_orders");

Both are worth knowing even if you start with plain producers and consumers, because they define what “the Kafka way” looks like at scale.

A minimal end-to-end example

It helps to see the whole thing in one file: connect a producer and a consumer, subscribe, run, and shut down cleanly.

import { Kafka } from "kafkajs";

const kafka = new Kafka({
  clientId: "orders-app",
  brokers: ["localhost:9092"],
});

const producer = kafka.producer();
const consumer = kafka.consumer({ groupId: "orders-app" });

async function main() {
  await producer.connect();
  await producer.send({
    topic: "orders.created",
    messages: [{ key: "customer_1", value: JSON.stringify({ id: "order_1" }) }],
  });

  await consumer.connect();
  await consumer.subscribe({ topic: "orders.created", fromBeginning: true });
  await consumer.run({
    eachMessage: async ({ partition, message }) => {
      const order = JSON.parse(message.value!.toString());
      console.log(`p${partition} @ ${message.offset}`, order.id);
    },
  });
}

async function shutdown() {
  await consumer.disconnect();
  await producer.disconnect();
  process.exit(0);
}

process.on("SIGTERM", shutdown);
process.on("SIGINT", shutdown);

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Three things in that file matter in production. The producer is connected once and reused, not created per message. The consumer subscribes before it runs. And the process handles SIGTERM, because a hard kill during a rebalance or a commit leaves the group in a worse state than a graceful close.

Event sourcing and the outbox pattern

Kafka is often described as the backbone of event sourcing, where the log is the source of truth and current state is a projection of events. Instead of storing only the latest row, you store the sequence of facts — order.created, order.paid, order.shipped — and rebuild any view by replaying them. Compacted topics make the projection a table; retention makes it auditable.

The hard part is never writing the event; it is writing the event and the database row atomically. A process can crash after committing the row and before publishing, or after publishing and before committing. Publishing inside a database transaction is impossible, so the standard answer is the outbox pattern: write the event to an outbox table in the same transaction as the state change, then let a relay or CDC connector publish those rows to Kafka.

BEGIN;
INSERT INTO orders (id, status, total_cents)
VALUES ($1, 'created', $2);

INSERT INTO outbox (id, topic, payload)
VALUES ($1, 'orders.created', $2);
COMMIT;

Debezium or a small polling relay then tails the outbox and produces to the topic, deleting rows once they are published. Because the outbox write shares the transaction, the event exists exactly when the state does. Consumers must still be idempotent, because the relay can publish a row twice after a crash.

Local development and testing

You do not need a full Kafka cluster to develop against it. A single-node broker in Docker Compose, or Redpanda in compatibility mode, gives you topics, consumer groups and offsets on a laptop.

services:
  kafka:
    image: redpandadata/redpanda:latest
    command: >
      redpanda start --overprovisioned --smp 1
      --kafka-addr PLAINTEXT://0.0.0.0:9092
      --advertise-kafka-addr PLAINTEXT://localhost:9092
    ports:
      - "9092:9092"

For tests, the pattern is the same as any other integration dependency: start the broker in a container, create the topics the test needs, and use a unique group id per test run so committed offsets never leak between runs. Reset offsets explicitly when a test needs to read history, and assert on the records your consumer received rather than on timing.

const groupId = `test-${crypto.randomUUID()}`;
const consumer = kafka.consumer({ groupId });

Keep the consumer handlers pure and thin — parse, validate, call a service — so most of the logic can be unit tested without a broker at all. Reserve the integration test for the wiring: does a produced record reach the right handler, and is the offset committed afterwards?

Throughput, latency and batching

Kafka is fast because it batches and because it writes sequentially. Both are tunable, and the tuning is a latency-versus-throughput dial.

• linger.ms — how long a producer waits to accumulate a batch. Higher means bigger batches and more throughput, at the cost of added latency.
• batch.size — the maximum bytes per partition batch.
• compression.type — snappy, lz4 or zstd. Compression cuts network and disk use; zstd usually wins on ratio.
• fetch.min.bytes and fetch.max.wait.ms — how long consumers wait to fill a fetch response.
• max.poll.records — how many records a consumer processes per poll. Raise it to drain backlogs faster, but keep the processing under max.poll.interval.ms or the consumer is kicked from the group.

const producer = kafka.producer({
  linger: { ms: 20 },
  compression: CompressionTypes.GZIP,
});

const consumer = kafka.consumer({
  groupId: "billing",
  maxBytesPerPartition: 1_048_576,
  maxWaitTimeInMs: 500,
});

A producer optimised for throughput might use linger.ms=20, a one-megabyte batch and zstd. A producer that must publish in single-digit milliseconds uses linger.ms=0. There is no single right answer; there is only the trade-off you choose and the p99 you can live with.

Choosing between Kafka, RabbitMQ and Redis

These three are often compared as if they were interchangeable. They are not.

• Kafka is a replayable log. Choose it when you need high throughput, durable history, fan-out to many independent consumers, event sourcing, stream processing or the ability to reprocess. It is the heaviest to operate.
• RabbitMQ is a message broker. Choose it when routing matters: exchanges, routing keys, per-message acknowledgements, priorities and dead-lettering. It is a natural fit for task distribution and complex routing, and it is lighter than Kafka at modest volumes. See the RabbitMQ guide.
• Redis is an in-memory data structure store that doubles as a job queue. Choose it when the work is simple, the volume is moderate and you already run Redis. BullMQ on Redis gives you retries, scheduling and a UI with almost no operational overhead. See Redis Queues.

A useful rule: if consumers need to replay, fan out to many groups, or read history, use Kafka. If a message must be routed to a specific worker and then forgotten, use RabbitMQ. If it is a background job with a retry policy, use Redis.

Security and access control

Kafka is usually the most valuable data in the building, so lock it down.

• Encryption — enable TLS for broker-to-broker and client-to-broker traffic. Plaintext Kafka inside a VPC is still plaintext.
• Authentication — SASL/SCRAM or mTLS for clients. Avoid unauthenticated listeners in anything but a throwaway local setup.
• Authorisation — use ACLs to grant read, write or create on specific topics and groups. A service should not be able to read every topic just because it can connect.
• Quotas — cap producer and consumer bandwidth per client so one runaway service cannot starve the cluster.
• Secrets — never bake credentials into a producer; inject them from the environment or a secret manager.

Monitoring and consumer lag

Kafka fails quietly if you let it. Four signals cover most of the operational reality.

• Consumer lag — the gap between the latest offset and a group’s committed offset, per partition. Rising lag is the earliest sign consumers cannot keep up.
• Under-replicated partitions — a non-zero count means a broker is down or slow and durability is at risk.
• Request and network latency — broker-side percentiles reveal when disks or the network are the bottleneck.
• Disk usage and segment counts — Kafka is disk-bound, and replication multiplies every byte by the replication factor.

Expose lag on the same dashboard as your services and alert on the trend, not on a single spike. Lag that grows steadily through the day is a capacity problem; lag that spikes and recovers is usually a rebalance or a slow deploy.

Best practices

• Design keys around the ordering you actually need; remember ordering is per partition.
• Use acks=all, min.insync.replicas=2 and a replication factor of three for important topics.
• Enable the idempotent producer and prefer manual offset commits.
• Make consumers idempotent, because at-least-once is the default contract.
• Keep max.poll.records processing under max.poll.interval.ms to avoid rebalance storms.
• Use log compaction for keyed state and tombstones for deletes.
• Track consumer lag per partition, not just the cluster total.
• Treat record schemas as a versioned contract, with a registry when more than one team is involved.
• Separate topics by lifecycle and retention rather than dumping everything into one.
• Set retention from a real requirement; seven days is a default, not a policy.
• Prefer managed Kafka unless running it is genuinely your business.
• Use Connect for standard integrations and Streams for in-cluster processing before writing custom glue.

Common mistakes

• Assuming global ordering when Kafka only orders within a partition.
• Using a random or null key for events that must stay ordered.
• Auto-committing offsets before the work is finished, then losing records on a crash.
• Making handlers non-idempotent and duplicating side effects on rebalance.
• Creating a topic with one partition and wondering why consumers cannot scale.
• Increasing partitions later and silently changing key routing.
• Treating Kafka as a request/response API with per-message replies.
• Setting max.poll.records too high and exceeding max.poll.interval.ms.
• Forgetting that replication multiplies storage and network costs.
• Leaving consumer lag off the dashboard until the backlog is hours deep.
• Running ZooKeeper on a new cluster in 2026.

Where to go next

Kafka is the log at the centre of an event-driven system. The backend roadmap covers event-driven architecture and microservices, where events, producers and consumers fit together and a log becomes the backbone between services. If you are comparing brokers, the RabbitMQ and Redis Queues guides show the routing-first and job-first alternatives. And because consumers are just background workers with offsets, the Batch Processing guide covers the retries, idempotency and observability that apply here too.