Event-Driven Architecture

What event-driven architecture actually means

Event-driven architecture is a style of communication in which services record events — past-tense statements that something happened — and react to them, instead of calling each other directly. An order service does not reach into a billing service and say “invoice this”. It records order.placed and moves on. A billing service that cares about that fact subscribes to it and invoices on its own time.

The important word is fact. An event is immutable and already true. order.placed describes something that happened; no consumer can veto it, and no producer is waiting for a reply. A request is a question with an answer, and the caller is blocked until it arrives. An event is a statement with an audience, and the producer is done the moment the statement is durable.

That difference sounds small and changes everything. Because the producer does not call anyone, it does not need to know who cares. Because consumers do not answer, they can be slow, restarted or added later without the producer noticing. The cost is that the system no longer has a single thread of control you can follow, and correctness now depends on every participant handling duplicates, delays and reordering gracefully.

This guide is about the machinery that makes those guarantees real, and about the cases where the trade is not worth making.

Commands tell, events describe

The most common source of confusion in event-driven systems is calling both kinds of message “events”. Keep them separate.

A command is an instruction: reserve-inventory, charge-card, send-welcome-email. It is imperative, it is addressed to a handler that is expected to act, and it can fail or be refused. There is exactly one owner of a command, and the sender usually cares about the outcome.

An event is a description: inventory-reserved, card-charged, user-registered. It is past tense, it is a fact owned by the producer, and it can have zero, one or a thousand consumers. No consumer can reject it, because it is already true. The producer neither knows nor cares who reads it.

// A command asks for something and expects a handler to decide.
await commands.send("reserve-inventory", { orderId, quantity: 2 });

// An event reports that a decision was made. Nobody may reject it.
await events.publish("inventory.reserved", { orderId, quantity: 2 });

The naming is not pedantry. A channel full of commands is a distributed procedure call, and it has all the coupling of one. A channel full of events is a broadcast, and it can be extended without asking permission. If a message name has no tense — inventory-reservation — nobody reading the code later can tell whether it is a request or a fact.

A useful test: can the sender proceed without knowing the outcome? If yes, it is probably an event. If the sender needs to branch on the result, it is a command, and you should route it to one handler rather than broadcast it.

It is common to need both in one flow. A checkout service sends a charge-card command to the payment service and waits for the answer, because it cannot confirm the order without it. Once payment succeeds, the payment service publishes payment.captured, and every other interested party reacts asynchronously. The command is the synchronous spine; the events are the fan-out. Mixing them deliberately is fine — the mistake is pretending a command is an event and being surprised when nobody answers.

Events, event sourcing and CQRS are three different ideas

These three are related, frequently used together, and entirely separable. Treating them as one thing is the fastest way to build a system that is more complicated than the problem.

Event-driven is a communication style. Services exchange facts through a broker. The database is still the source of truth, and you could remove the broker tomorrow and lose only the decoupling.

Event sourcing is a persistence style. Instead of storing the current row, you store the ordered sequence of events that produced it, and current state is a fold over that sequence. The log is the source of truth. Rebuilding an account balance means replaying its deposits and withdrawals. This gives you a perfect audit trail and the ability to travel back in time, at the cost of more complex reads and a harder migration story.

CQRS (Command Query Responsibility Segregation) is about separating the write model from the read model. Commands go through a model optimised for validation and invariants; queries read from one or more projections optimised for lookup. The two models can share a database or use different stores entirely.

You can adopt any one without the others:

• Event-driven without event sourcing: services publish facts, but each keeps a normal table.
• Event sourcing without a broker: a single service stores its events in its own database.
• CQRS without events: two models over the same data, synchronised synchronously.

Most teams should start with plain event-driven communication and a normal database. Event sourcing is a serious commitment, and reaching for it because the architecture sounds impressive is a reliable way to regret it.

Pub/sub, topics and fan-out

The transport that makes events work is publish/subscribe. Producers publish to a named channel, usually called a topic or exchange, and consumers subscribe to the topics they care about. The broker holds the routing.

The key property is that the producer does not address consumers. It publishes once to orders, and the broker delivers to every subscription: billing, fulfilment, search indexing, analytics, fraud detection. Adding a sixth consumer is a configuration change in that consumer, not a code change in the producer.

// One publish, many independent subscribers.
await broker.publish("orders", {
  type: "order.placed",
  data: { orderId, customerId, totalCents },
});

There are two broad shapes of broker, and the choice affects what you can build:

• Log-based brokers such as Kafka keep every event for a retention window and let each consumer group track its own position. Consumers can replay history, and many groups read the same topic independently.
• Queue or exchange-based brokers such as RabbitMQ route each message to one or more queues, and a message is typically removed once acknowledged. Routing is rich; replay is not the model.

A topic should be named for the fact it carries, not for the consumer that reads it. orders and users age well; billing-inbox does not, because the day a second consumer appears the name is a lie. Keep topics stable and let subscriptions be the thing that changes.

The acknowledgement model is what decides delivery guarantees. A consumer that acknowledges before doing the work risks losing an event on a crash; one that acknowledges after doing the work risks processing it twice. Almost every broker defaults to the second, and that is why idempotency is not optional. Some systems also let one logical subscription receive an event once, even with many competing consumers — a work queue — while others give every subscriber its own copy — a broadcast. Know which one a topic provides before you rely on either.

Eventual consistency and why consumers lag

The moment a producer stops waiting, the system becomes eventually consistent. After order.placed is committed, the order exists immediately in the orders database but not yet in the invoice, the search index or the analytics warehouse. There is a window — milliseconds under normal load, minutes during an incident — where those views disagree.

This is not a defect to hide; it is the defining property of the style. Every read path built on an event has to answer two questions: how stale can this be, and what does the user see in the meantime?

A few habits make it manageable:

• Read your own writes from the source. After a user acts, redirect them to a view served by the write model, not by a projection that has not caught up.
• Show honest status. A 202 Accepted with status: "processing" is better than a page that flickers from empty to populated.
• Measure lag. The gap between the newest published event and a consumer’s position is the single most useful health signal in the system.

A concrete example makes the window tangible. A customer places an order and the confirmation page is served by the orders service, so it is immediately correct. Their account page is served by a projection built from order.placed, so for the next two hundred milliseconds it shows no orders. If the projection is a few seconds behind during a deploy, the customer sees “no orders” and files a support ticket. None of this is a bug in the event flow; it is the flow working as designed, and the UI has to be built for it.

Lag is normal and grows for ordinary reasons: a burst of traffic, a slow downstream API, a consumer restart, a rebalance. Lag that grows without bound is a capacity problem, and it is invisible unless you put it on a dashboard. A good projection tracks its own position and exposes it, so the gap is a number you can alert on rather than a feeling you discover from complaints.

There is one consistency guarantee worth keeping even in an eventually consistent system: monotonic reads. A consumer should never move backwards. If an event arrives with an older timestamp than one already applied, applying it out of order can resurrect deleted data or regress a counter. Version your projections by sequence number or offset, not by wall-clock time, and ignore anything older than what you have already processed.

The dual-write problem and the transactional outbox

Here is the failure that catches almost everyone. A service needs to change its database and publish an event. It writes the row, then calls the broker. What if the publish fails? What if the process is killed between the two?

• Commit first, then publish: the row exists, the event never happened, and downstream services silently miss the change.
• Publish first, then commit: the event announces a state that was never saved, and consumers act on a fact that is not true.

There is no way to make a database write and a network publish atomic. This is the dual-write problem, and it cannot be solved by ordering the two calls more carefully. A broker that supports transactions does not help, because the database is a separate system.

The standard answer is the transactional outbox. Instead of publishing directly, write the event to an outbox table in the same transaction as the state change. Either both rows commit or neither does. A separate relay — a polling worker, or a change-data-capture connector tailing the database log — reads unpublished rows and publishes them to the broker.

BEGIN;
INSERT INTO orders (id, customer_id, status, total_cents)
VALUES ($1, $2, 'placed', $3);

INSERT INTO outbox (id, topic, payload)
VALUES ($1, 'orders', $2);
COMMIT;

The relay then deletes or marks rows once published. If it crashes after publishing but before marking, the event is published twice — which is exactly why consumers must be idempotent. If it crashes before publishing, the row is still there and will be picked up on the next pass. Either way, no event is lost.

Two details matter. The relay should claim rows with FOR UPDATE SKIP LOCKED (or an equivalent) so multiple relay instances do not publish the same row concurrently. And the outbox should be pruned, because a table that only ever grows will eventually be the largest thing in your database.

At-least-once delivery and idempotent consumers

Every broker worth using gives you at-least-once delivery. It does not give you exactly-once, because exactly-once across a network and a crash is effectively impossible. A consumer can process an event, crash before acknowledging it, and receive it again on restart. An outbox relay can publish a row twice. A producer can retry a timeout that actually succeeded.

This is the contract, not a bug. Your consumers must be idempotent: processing the same event twice must leave the same end state as processing it once.

There are three practical patterns:

Natural idempotency. Some operations are already safe to repeat. Setting a status to shipped twice is the same as once. Inserting with ON CONFLICT DO UPDATE converges. Prefer these where you can.

A dedupe table. Record every processed event id with a unique constraint, in the same transaction as the work. If the insert conflicts, the event has been handled and the consumer returns early. This is the general-purpose solution and the one to reach for first.

Provider idempotency keys. Payment gateways and many APIs accept a stable key and return the original result instead of repeating the side effect. Combine it with your own dedupe, because the key protects the call, not the surrounding logic.

const seen = await client.query(
  `INSERT INTO processed_events (event_id, consumer)
   VALUES ($1, 'billing')
   ON CONFLICT DO NOTHING
   RETURNING event_id`,
  [event.id],
);

if (seen.rowCount === 0) return { skipped: true };

Notice that the dedupe key is the event id, not the order id. That makes the consumer safe even when the producer legitimately publishes two different events about the same order — order.placed and order.cancelled are distinct facts and both should be processed.

Idempotency is a property of the effect, not of the transport. A broker can filter duplicate event ids at the edge, which helps, but it cannot know whether your handler already sent an email or charged a card. Only the consumer, inside the same transaction as its side effect, can decide that. This is why dedupe belongs next to the write and not in a middleware layer that runs before it.

Ordering and partitioning

A broker that fans out across many partitions cannot promise a global order. Kafka orders records within a partition; RabbitMQ orders within a queue served by one consumer. Across the system, events arrive in whatever order the network and scheduling allow.

That is fine as long as you choose a partition key that matches the ordering your consumers need. Publish every event for one customer with key = customerId, and all of that customer’s events land in the same partition and are processed in order. Different customers spread across partitions and are processed in parallel.

await producer.publish("orders", {
  key: event.data.customerId, // ordering is per key
  value: event,
});

The trap is picking a key that does not match the invariant. Key by orderId when consumers need per-customer ordering and you get parallelism you did not want and ordering you cannot rely on. Key everything by a single constant and you get perfect order with no parallelism at all.

Two more cautions. Changing the partition count later changes the mapping from key to partition, so events for one entity can end up split across two partitions and lose their relative order. Decide the count with headroom. And if a consumer processes a partition serially, a single slow event blocks everything behind it, so keep per-event work bounded.

Schema evolution and versioning

An event is a contract between a producer and consumers that deploy independently. The producer will be upgraded while old consumers are still running, and a new consumer will read events written months ago. The payload shape has to survive both directions.

The rules are the same as for a public API:

• Add, do not rename or remove. New fields are optional and consumers default them.
• Version when the meaning changes. A version field lets a handler branch explicitly instead of guessing.
• Never reuse a field name for a different concept. That is how a quiet data corruption starts.
• Treat old events as valid forever. Replay means yesterday’s payload must still parse today.

export type OrderPlacedV2 = {
  type: "order.placed";
  version: 2;
  data: {
    orderId: string;
    totalCents: number;
    currency?: string; // added later; v1 events simply lack it
  };
};

At scale, a schema registry turns this into a managed problem. Producers register a schema — Avro, Protobuf or JSON Schema — and the registry assigns it an id, enforces a compatibility mode, and rejects a change that would break existing readers. Even without a registry, keeping a versioned schema file in the repo and reviewing changes to it gives you most of the benefit.

The discipline pays off precisely during incidents. When a deploy breaks a consumer, the first question is whether the producer changed a payload in a way nobody agreed to.

Contract tests are the cheap version of a registry. Keep a fixture file of real events per version, and have every consumer parse all of them in CI. A consumer that fails on a v1 fixture will fail in production the first time an old event is replayed. This costs a few files and catches the class of break that otherwise appears days later as a corrupted projection.

Choreography and orchestration

A multi-step business process built from events can be coordinated in two ways, and the difference is significant.

Choreography means each service listens for events and reacts, with no central coordinator. The order service publishes order.placed; inventory reserves stock and publishes inventory.reserved; payment charges and publishes payment.captured; shipping reacts to that. Each service knows only the events it consumes and produces. It is loose, extensible and easy to add a step to. It is also hard to see the whole process, because the flow exists only as the sum of everyone’s subscriptions.

Orchestration means a central component — a saga orchestrator or process manager — explicitly tells each service what to do and tracks the state of the process. The flow is in one place, which makes it visible, testable and easy to reason about. The cost is a coordinator that every service must depend on and that can become a bottleneck and a single point of failure.

Neither is universally right:

• Choreography suits simple, mostly independent reactions and stable steps.
• Orchestration suits long processes with many conditional branches, timeouts and compensations.

A common pragmatic split is to choreograph the happy path between a few services and introduce an orchestrator only for the process that has grown complicated enough to need one.

The signal that choreography has gone too far is a change that requires editing many services at once, or a process nobody on the team can describe without opening five repositories. When adding one business rule means touching six consumers, the flow has stopped being a set of independent reactions and become a distributed program with no author. That is the moment to pull it into an orchestrator.

Sagas and compensations

In a monolith, a multi-step operation can be wrapped in a database transaction and rolled back on failure. Across services there is no shared transaction, so a distributed process cannot simply abort. If payment succeeded and shipping failed, you cannot un-run the payment with a ROLLBACK.

The saga pattern handles this. A saga is a sequence of local transactions, each publishing an event that triggers the next step. If a step fails, the saga runs compensating actions for the steps that already succeeded: refund the payment, release the reserved inventory, mark the order cancelled. Compensation is not rollback — it is a new business action that undoes the effect, and it is itself an event that must be idempotent.

// Forward path
// order.placed -> inventory.reserved -> payment.captured -> order.confirmed

// If payment fails, compensate the steps that already ran.
await events.publish("payment.failed", { orderId, reason });
// inventory service listens and releases the reservation

Two design rules keep sagas sane. First, every step must be idempotent, because a retry can run it again. Second, every step needs a compensating action defined up front — if a step cannot be undone, the saga cannot safely fail after it, and that step belongs last or needs a different design. Sagas also make intermediate states visible, so the UI should show “reserving stock” and “awaiting payment” rather than pretending the operation is atomic.

A saga needs its own state. Either the orchestrator stores the current step in a table, or each service tracks the events it has seen. That state is what lets a process resume after a crash, time out a step that never responded, and know which compensations are still owed. A saga without persisted state is a sequence of messages that will eventually get stuck in a state nobody can reconstruct.

Timeouts deserve particular attention, because a step that never replies is the most common failure. If payment neither succeeds nor fails within a window, the saga must decide: retry, compensate, or park the order for manual review. Leaving it undecided means the order sits in limbo forever, holding inventory that will never be released.

Dead-letter queues and replay

An event that a consumer cannot process will fail every time it is retried: malformed payload, a bug in the handler, a referenced row that does not exist. Retrying it forever burns a consumer slot and blocks everything behind it in the partition or queue.

The answer is a dead-letter queue (DLQ). After a configured number of attempts, the broker moves the event — payload, headers, attempt count and last error — to a separate queue that no consumer reads. Healthy traffic keeps flowing, and an operator can inspect the failed event, fix the cause and replay it.

Replay is the quieter superpower of event-driven systems. Because events are durable, you can reprocess history after fixing a bug: rebuild a projection that was computed wrong, backfill a service that was added late, or re-run a day of events against new logic. The requirement is that consumers are idempotent, because replay will feed them events they may have already handled.

Treat the DLQ as an operational surface, not a graveyard. Alert on its depth, put it on the dashboard, and build a replay path before you need it at 2 a.m. A dead-letter queue nobody watches is where bugs hide.

Replay also needs a retention story. You can only reprocess events the broker still has, so the retention window defines how far back a fix can reach. A topic that keeps seven days cannot rebuild a projection after a bug that ran for two weeks. Decide retention from the recovery you actually want, and remember that every byte is multiplied by replication and by every consumer’s own storage.

Seeing an event flow

The hardest part of event-driven systems is not building them; it is understanding what happened after something went wrong. A single user action can produce a dozen events across six services, and the failure might be in the third consumer of the fifth event.

Three practices make the flow observable:

• Correlation id. Generate one id at the edge, put it on every event and every log line, and propagate it through every hop. A single grep then reconstructs the whole flow.
• Tracing. OpenTelemetry and similar tools model an event as a span linked to the event that caused it, which turns the flow into a graph you can read.
• Consumer lag and DLQ depth. These two metrics catch most problems before a user does: a consumer falling behind, or a handler that has started failing.

await events.publish("order.placed", {
  ...event,
  correlationId: req.id, // set once, carried everywhere
});

Log the event id and type on both the publish and the consume side. Without that, debugging means correlating timestamps across services and hoping the clocks agree.

A useful dashboard for an event flow has three rows: publish rate per event type, consumer lag per group, and DLQ depth per consumer. A publish rate that drops to zero means a producer stopped; lag that climbs means a consumer cannot keep up; DLQ depth that grows means a handler is broken. Between them, those three signals explain most incidents before anyone opens a log.

Thin events, fat events and the payload contract

A recurring design question is how much data an event should carry. A thin event contains only an identifier and a type — order.placed with an orderId. A fat, or enriched, event carries a full snapshot: line items, totals, the delivery address as it was at the time of the order.

Fat events make consumers simpler and more robust. A search indexer that receives the whole order does not have to call back into the orders service, which removes a runtime dependency and a failure mode. But they also widen the contract: every field is now something a consumer may depend on, so changing the shape becomes harder, and the event may carry data a given consumer is not allowed to see.

Thin events keep the contract minimal and the payload small, but every consumer must fetch current state, which reintroduces coupling and can read data that has since changed. The event stops being a complete fact and becomes a pointer.

A workable default is to include the fields that define the fact and are safe to share, and reference everything else by id. order.placed should carry the order id, the customer id and the total, because those are the fact. It should not embed the customer’s whole profile. The test is simple: can a consumer understand the event for the purpose it announces without becoming a copy of your database?

Whichever you choose, freeze the values that must not change. If a price was quoted at checkout, the event should carry that price even if a thin-event instinct says to look it up later — because later, the price may be different, and the event would then describe a fact that never happened.

Choosing a broker without a framework war

The broker is the least interesting decision and the one teams argue about most. Three families cover almost every case.

• Log-based brokers such as Kafka and NATS JetStream retain events for a window and let each consumer group track its own position. Choose them when replay, high throughput or many independent readers are requirements.
• Exchange-based brokers such as RabbitMQ route messages to queues using routing keys, with per-message acknowledgement, priorities and dead-letter exchanges. Choose them when routing and task distribution matter more than history.
• Cloud queues such as SQS and Pub/Sub are fully managed and effectively unlimited, at the cost of lower-level APIs and less control over ordering and scheduling.

The honest guidance is to start with what your platform already runs and what your team already understands. A correct system on a familiar broker beats a theoretically perfect one on a broker nobody can operate. Migrate when a specific limitation — replay, routing or throughput — actually bites, not because a conference talk preferred a different tool.

Whichever you choose, hide it behind a thin publisher interface in your own code. publish(topic, event) is a stable seam; a vendor SDK is not. That keeps the broker decision reversible and lets tests publish to an in-memory collector instead of a real cluster.

Testing an event flow

Events are asynchronous, which makes tests feel awkward until you separate the parts.

Test the producer by asserting on the outbox, not the broker. After calling the service, the state row and the outbox row must both exist, and the payload must match the schema. No broker needed.

Test the consumer as a pure function of an event. Feed it a payload and assert on the resulting state. Then feed it the same payload twice and assert that the second run is a no-op — this is the idempotency test, and it catches the class of bug that only shows up during a redelivery in production.

Test the wiring with a real broker in a container: publish an event, wait for the consumer to process it, and assert on the end state. Use a unique group id or queue per test run so committed offsets never leak between runs, and assert on records received rather than on timing.

test("reprocessing an event is a no-op", async () => {
  await onOrderPlaced(event);
  await onOrderPlaced(event); // redelivery

  const { rows } = await pool.query(
    "SELECT count(*)::int AS n FROM invoices WHERE order_id = $1",
    [event.data.orderId],
  );
  expect(rows[0].n).toBe(1);
});

Keep the asynchronous assertion deterministic by polling for the expected state with a timeout rather than sleeping for a fixed interval. A test that passes because it waited long enough is a test that will flake on a slower machine.

When event-driven shines and when it hurts

Event-driven architecture is a trade, and it is worth being explicit about which side of it you are on.

It shines when you need decoupling between teams that deploy independently, fan-out to many readers, replay of history, a durable audit trail, or a natural feed for analytics and search. It fits systems where a downstream reaction can be slightly late and where adding a new consumer should not require changing the producer.

It hurts when the domain is small and CRUD-shaped, when an operation must be immediately and strongly consistent, or when the team is too small to operate a broker and reason about eventual consistency. In those cases a well-structured monolith with clear modules and a single database is simpler, faster to build and easier to debug. Events can always be added later, and a modular monolith is a much better starting point than a distributed system nobody can trace.

A reasonable rule: do not introduce a broker until you can name the specific problem it solves. “Microservices use events” is not a problem statement. Write down what you expect to gain — a new consumer without touching the producer, replay after a bug, an audit trail — and check later whether you got it. If the honest answer is “we wanted to look modern”, the broker is a cost with no return.

If you do adopt it, adopt it incrementally. Start with one event that solves a real problem, run it in production for a while, and learn how lag, duplicates and schema changes behave on your team and your infrastructure before you make events the backbone of the system.

Best practices

• Name events in the past tense and own them at the producer; name commands separately.
• Write the event to an outbox in the same transaction as the state change.
• Publish through a relay or CDC connector, never directly from a request handler.
• Make every consumer idempotent with a dedupe key derived from the event id.
• Choose a partition key that matches the ordering each consumer actually needs.
• Version event payloads and evolve them additively; never repurpose a field.
• Keep consumers small and single-purpose; one consumer per projection or reaction.
• Define compensating actions for every step before you build a saga.
• Cap retries, route exhausted events to a dead-letter queue, and build a replay path.
• Propagate a correlation id on every event and log it on both sides.
• Monitor consumer lag and DLQ depth, and alert on the trend.
• Start with a modular monolith and add events when decoupling or replay is the real need.

Common mistakes

• Calling commands events and ending up with a distributed procedure call.
• Conflating event-driven, event sourcing and CQRS and adopting all three at once.
• Publishing directly from a request handler and hitting the dual-write problem.
• Assuming exactly-once delivery and double-charging on the first redelivery.
• Keying events randomly and then expecting per-entity ordering.
• Renaming or removing a payload field and breaking consumers on an old release.
• Building choreography with no way to see the end-to-end process.
• Retrying a poison event forever and blocking the partition behind it.
• Never looking at the dead-letter queue.
• Adding a broker for a CRUD app and paying the complexity tax for nothing.

Where to go next

The transport underneath an event-driven system is usually a log or a broker, so Apache Kafka covers the replayable log model and RabbitMQ covers routing-first messaging. Because events are how services in a distributed system talk, the Microservices guide explains where the service boundaries come from in the first place. If this all sounds heavier than your problem, the Modular Monolith guide is the honest counterpoint: most systems should stay a single deployable unit until a real reason to split appears.